Is it permissible to be late for the Friday sermon or not attend it, and suffice with attending the prayer due to the excuse mentioned in the question?

Hastening to the Friday prayer and attending the sermon from its beginning is among the virtuous deeds, for it allows one to benefit from the admonition and knowledge. Lagging behind and not attending the sermon, contenting oneself with only the prayer, may cause the Muslim to miss out on this virtue and may lead to sin according to some scholars. The aforementioned excuse is not considered a justification for neglecting this good. A Muslim should seek Allah's help to achieve his goals and attach his heart to Him.