Is the phrase "and by Allah's House" considered an oath, and should this be clarified to the recipient of the message?

The aforementioned sentence was an objection to swearing by the Sacred House (Kaaba). Hujjat al-Islam (Al-Ghazali) mentioned it in his advice "O Son" in urging vigilance and night prayer. It is important to alert the reader that swearing by other than Allah the Almighty is impermissible, due to the Prophet, peace and blessings be upon him, saying: "Whoever swears by other than Allah has disbelieved or committed shirk." There is no expiation for one who swears by the Kaaba except repentance, and swearing by it is not an oath that requires expiation if broken.