What is the ruling on someone who vowed to stop masturbating and to pay 100 Riyals for each instance, then proceeded to do it more than 50 times? Is he obligated to fulfill the vow for every single instance?

If the intention behind the vow is to deter oneself, then it is a vow of obstinacy and anger (nadhir lajaj wa ghadab), and its ruling is that one has the choice between fulfilling it or offering expiation. The expiation for an oath is feeding ten needy persons, or clothing them, or freeing a slave. If one cannot find these, then fasting for three days, as stated in verse 89 of Surat Al-Ma'idah. If one chooses to fulfill the vow, then they must fulfill it for every instance in which they broke it.