Does the prayer become invalid by saying the opening Takbir (Takbirat al-Ihram) instead of the transitional Takbir when standing for a missed Rak'ah, with prostration of forgetfulness (sujud al-sahw) after the Taslim?

It was more appropriate for you to enter into ihram (definition) (sacralize the prayer) with takbir (saying "Allahu Akbar") while intending to return to the prayer before standing for the rak'ah. However, the prayer is not invalidated by entering into ihram while standing, nor by omitting ihram in this situation. Rather, you should sit for ihram, according to the more apparent opinion.

You are not liable for omitting the takbirat al-intiqal (transition takbirs), and your prayer is not invalidated, because takbirat al-intiqal are considered sunnah (definition) (recommended) by the majority of scholars.