Is it obligatory to fast for a continuous month for someone who swore to fast it if he committed masturbation, then broke his oath by committing it?

If the formula the questioner intended to use to prevent himself from returning to the forbidden act was, "By Allah, I shall fast for a month if I return [to it]," then this is a binding oath (yamin mun'aqidah) that obligates him. He must fast for a month if he does what he swore to abandon. If he does not fast, then an expiation for an oath (kaffarat yamin) becomes obligatory upon him.

However, if he did not swear by Allah, but rather obligated himself to fast for a month with the intention of prevention, this is a "vow of obstinacy" (nadhr lajaj), and he has the choice between fasting for a month or performing an expiation for an oath. This is the view of Imam Ahmad.

The best way to prevent oneself from sins is to be mindful of Allah and to remember His knowledge of what is secret and what is manifest, and that there are guardians who record everything. Among the best methods of prevention are marriage or fasting.