Is the prayer valid, and is the prostration of forgetfulness sufficient for someone who doubted the number of prostrations, then interrupted the Tashahhud to supplicate, then doubted the number of rak'ahs, then completed his prayer and prostrated out of forgetfulness?

If one doubts having omitted a pillar of prayer, such as a rak'ah or a prostration, he should base his action on certainty and perform that pillar, then prostrate for forgetfulness (sujud (definition) al-sahw). This is what the questioner did, and it was correct, and his prayer is valid. If doubts occur daily, the worshiper should disregard them and is not required to perform sujud al-sahw. The authentic supplication between the two prostrations is not "Rabb ighfir li wa li walidayya" (My Lord, forgive me and my parents) three times.