Does the continuous movement of the hand scratching the head, along with prolonged remembrance, invalidate the tranquility during the sitting between the two prostrations? And must one repeat a prayer if its invalidity was doubted and its time has expired?

It is evident that you are experiencing intrusive thoughts (waswas), and we advise you to disregard them and not pay them any attention. Tranquility (طمأنينة) is achieved when the limbs are still, even for a short period. Some scholars have defined its minimum duration as the time it takes to say "Subhanallah" (Glory be to Allah). The movement you were performing does not negate tranquility. As for the prayer about which you doubt its validity, consider it valid, and you are not required to repeat it.