Does a prayer become invalidated if one adds more than half of the prayer out of forgetfulness or doubt, or adds two prostrations out of forgetfulness or doubt? And what should one do if they doubt a pillar of the prayer and then remember it?

We do not know of any jurist who invalidated a prayer due to an accidental addition of more than half of it. What is known among the Malikis is that an accidental addition of four rak'ahs in a four-rak'ah or three-rak'ah prayer, or two rak'ahs in an original two-rak'ah prayer (not shortened), invalidates the prayer if the addition is certain after the prayer. Otherwise, sujood al-sahw (prostration of forgetfulness) is sufficient. As for intentional addition, it invalidates the prayer even if it is less than a rak'ah. Al-Dardir said in Sharh al-Kabir: "It is invalidated (by an addition of four) rak'ahs, if certain and accidental, even in a three-rak'ah prayer (like two rak'ahs in a two-rak'ah prayer) originally, such as Jumu'ah and Fajr prayers, but not a traveler's prayer, which is invalidated by four. Witr prayer is invalidated by an addition of two rak'ahs, not one." As for an accidental addition of prostrations or the like, it does not invalidate the prayer. And whoever doubts having omitted a pillar and then becomes certain that he performed it, he returns to what he was doing and completes his prayer.