What is the ruling on someone who doubted the number of rak'ahs, so he based his assumption on the lesser number, then doubted the number of prostrations in the rak'ah he considered to be the first? He then remembered, while sitting between the two prostrations, that he had performed two prostrations, but he does not know if they were in this rak'ah or in the previous rak'ah. Would his prayer be invalidated if he prostrates again?

What you did was correct, by building upon the minimum, due to the Prophet's (PBUH) saying: "If one of you doubts in his prayer and does not know how many he has prayed, three or four, let him cast away the doubt and build upon what he is certain of, then prostrate twice before giving the salām." Your performing the second prostration is required because you must build upon the minimum. Your prayer is valid, then you perform the prostration of forgetfulness to spite Satan, and it is done after the salām. However, if you experience frequent forgetfulness consistently, do not pay attention to it, and no prostration is required of you.