What is entailed for one who broke multiple, repeated oaths to abandon sins (masturbation, watching pornography, smoking) by paying monetary amounts or fasting, with the renewal of the oath and an increase in the amount, and is he obligated to offer a single expiation for each oath or multiple expiations?

The door to repentance remains open; it will not close until the the sun rises from the west. You must offer sincere repentance for the sins you mentioned. Among the things that will help you repent is contemplating Allah's blessings upon you, and showing gratitude to Allah by using His blessings in obedience to Him. Also, among the things that will help you repent is the fear of disgrace in the Hereafter, and contemplating the Names and Attributes of the Lord, and being mindful of His constant observation of you.

As for your oaths, you must fear Allah and avoid disobeying Him. Your oath to pay a certain amount for each sin committed necessitates one of two things: either fulfilling this oath whenever you commit that sin, or performing an expiation for an oath (kaffarat yamin) each time you break it. Your oath is not dissolved after the first breach because your oath implies repetition. The expiation for an oath is feeding ten poor people, or clothing them, or freeing a slave. If one cannot find [the means], then it is fasting for three days.