Are the prayers performed by the questioner after discovering a trace that he suspects is a remnant of a previous state of ritual impurity valid, or must they be repeated due to the obsessive doubts that have afflicted him?

You are not required to do anything if the matter is merely a doubt that some spots were not reached by the water during ghusl, because doubt after an action has no bearing, and the default is the exoneration of one's responsibility. However, if you were certain that the ghusl was not complete, then what you did by completing the ghusl and repeating the prayer is correct, as continuity is not a condition for ghusl from janabah. We advise you to abandon the forbidden practice of masturbation.