What is the ruling on one who made a mistake in the invocations of bowing (rukuʿ) or prostration (sujūd), or forgot the transitional takbīrāt, or uttered them in the wrong place, and did not perform the prostration of forgetfulness (sujūd as-sahw)? And is he required to repeat those prayers?

There is no need to repeat the prayer if the glorification of bowing (ruku') was recited during prostration (sujud) or vice versa, because the glorification of bowing and prostration is a Sunnah according to the majority of scholars. And whoever omits it intentionally or unintentionally, or replaces its wording, there is no sin upon him. Likewise, the saying: "Allah hears him who praises Him" (Sami' Allahu liman hamidah) or "Our Lord, and to You be all praise" (Rabbana wa laka al-hamd) is a Sunnah according to the majority. And the takbirat of transition are desirable, not obligatory. Since the questioner suffers from whisperings (waswasah), her prayer is valid, and she can adopt the view of the majority.