Does the ruling differ when a person suffering from obsessive doubts (al-mawsuwis) believes that the "millah" (religious community) refers to a specific group or family? And does regret and performing prayer with the congregation count as repentance in this case?

As long as you do not mean religion by the word "millah," but rather intend a person's family or relatives, then you are excused by this and the rulings concerning insulting religion do not apply to you. However, you must repent to Allah generally for insulting others, because insulting a Muslim is a transgression (fusuq), and you must resolve never to return to such an act again. Regret over a sin is repentance. In your case, the regret and repentance should be for insulting Muslims, because by "millah" you intended the man's tribe and family, and this is a sin, not apostasy.