What is the ruling on a congregant saying, "My Lord, forgive me," assuming it is the sitting between the two prostrations, then it becomes clear that it is the sitting for the Tashahhud, and does the ruling differ if he is a latecomer?

The saying "My Lord, forgive me" when uttered out of place, is a prescribed remembrance said in the wrong context. It does not necessitate a prostration of forgetfulness (sujud (definition) sahwi) upon the follower (ma'mum), because the Imam bears this prostration on his behalf. There is no difference in this ruling between a latecomer (masbuq) and others. Ibn Qudamah mentioned that uttering a prescribed remembrance out of its proper place does not invalidate the prayer, and there are two narrations regarding the permissibility of prostrating for it.