Am I obligated to adhere to the oath I swore upon the Qur'an—under duress from my brother—not to return to my husband, knowing that I wish to return to him?

If the brother forced his sister to swear an oath, then there is no sin upon her for breaking it. Valid duress is considered to be a threat of death or severe harm. However, if there was no valid duress, then she must offer expiation for breaking an oath.

If the husband neglects prayer out of laziness, without denying its obligation, then the brother has no right to prevent her from returning to him or to compel her to divorce. However, if he neglects prayer out of denial of its obligation, then he is an apostate, and it is impermissible to remain with him.

In any case, the husband must be advised and informed of the danger of neglecting prayer. If he does not respond, then it is preferable to separate from him through divorce or khulʿ.