Is it permissible to replace the word "Amen" after Al-Fatiha with other phrases such as "Allahumma istajib" (O Allah, answer) or "Amen, ya Rabbal-Alamin" (Amen, O Lord of the worlds) or "Ya Allah" (O Allah), and is such a replacement considered to invalidate the prayer?

It is not appropriate to replace the "ameen" mentioned in the Sunnah with anything else, for all goodness lies in following what is stated therein. Saying "ameen" after finishing the recitation of Al-Fatiha is a Sunnah, and its wording is "ameen," as narrated from the Prophet, peace and blessings be upon him. The phrases mentioned in the question—even if they differ from what is narrated—do not invalidate the prayer, nor do they necessitate prostration of forgetfulness (sujud al-sahw), whether done intentionally or unintentionally, because they are a remembrance or supplication not prescribed by the Law for prayer. It was narrated that the Prophet, peace and blessings be upon him, heard a man saying in prayer: "All praise is due to Allah, much praise, good and blessed praise, as our Lord loves and is pleased with," and he did not command him to prostrate.