Are the actions I took against my father, who committed adultery and insulted the Prophet, and against the woman with whom he committed adultery, and against my mother, who allowed him back into the house, considered forbidden (haram), and how should I deal with this matter and behave towards my parents?

If your father is not a Muslim or has apostatized from Islam, it is not permissible for your mother to return to him until he repents and enters Islam, due to the Almighty's saying: (Then if you ascertain that they are believers, do not return them to the disbelievers; they are not lawful [wives] for them, nor are they lawful [husbands] for them). What your father does by insulting the Prophet, peace be upon him, is disbelief and apostasy. Therefore, it is not permissible to enable him to return home.

With your father's apostasy from Islam, the rights of kindness and good treatment towards him are forfeited. However, it is not advisable to engage in arguments with him, especially in front of people.

As for your mother abandoning prayer, it is a great danger and is considered major disbelief by many scholars. You must advise her about the obligation of prayer and the seriousness of abandoning it, with gentleness, especially since she is on her way to commitment.