What is the ruling on saying "for God's sake" (بالله عليك) instead of "by the Prophet" (والنبي) and similar phrases that are considered minor polytheism (shirk asghar)? And what should be said to make someone swear an oath if it is not permissible to use such phrases?

If the speaker intends "I adjure you by God" as an oath, then this is a binding and permissible oath. It is sunnah (definition) to fulfill this oath unless it involves a sin or corruption. Expiation is incumbent upon the swearer if he does not fulfill the oath to the one he swore to.