What is required of one who used to masturbate and pray while in a state of major ritual impurity for a year or more, believing his ritual bath to be valid, and then discovered his mistake?

All praise is due to Allah, who has accepted your repentance. It is obligatory for you to make up all the prayers you performed without proper purification, because they are a debt upon you, and the debt owed to Allah is more deserving of being fulfilled. There is no expiation required for entering the mosque while impure; rather, you are obligated to repent for your negligence in learning the rulings of Sharia.