What is the expiation for someone who made a covenant with Allah to stop committing a sin, promising to fast for a day if he committed it, then committed it repeatedly and forgot the number of times, and whether it was a vow or an oath?

Your statement "If I commit this sin, I will fast for a day" is considered a vow of lajaj (obstinacy) and anger. You are not obligated to fast by it, even if you intended it as a vow. If you break your vow and commit the sin, you must first repent. Then, you have the choice between fulfilling what you vowed or offering an expiation for breaking an oath (kaffarat yamin). If the sin is repeated, you are obliged to repent each time, but you are not obligated to fast or offer another expiation.