What is the ruling on one who swore by divorce to abandon a bad habit, then broke his oath, and he does not remember his intention at the time of the oath, and a valid seclusion (khalwa shar'iyya) and touching occurred with the wife without penetration?

The majority of scholars hold the view that swearing an oath by divorce results in the divorce taking effect upon breaking the oath. This is the established ruling. Therefore, your divorce has taken effect due to your breaking the oath. You may reconcile with your wife before her 'iddah (waiting period) expires, provided you had a valid private seclusion with her.

However, Sheikh al-Islam Ibn Taymiyyah is of the opinion that an oath sworn by divorce, when the intention is not to effect a divorce but rather to threaten or emphasize, has the same ruling as an oath sworn by Allah. In this case, an expiation for an oath becomes binding, and no divorce takes place, especially if you doubt your intention to effect the divorce. Certainty dictates the continuation of the marriage.