Is it obligatory to host my mother in my home despite the problems that arise between her and my wife due to her psychological condition, or is it better to provide her with private accommodation while continuing to look after her? And is the occurrence of some uncontrolled behavior towards one's mother due to anger considered disobedience that necessitates expiation?

Perhaps it is best for you to sit with your father and speak to him about your mother and her illness, or seek help from a virtuous person who can speak to him. If you are unable to convince your father, and your wife is not content with her living with you, then your renting a house for her is a good deed, provided that she is not left alone to face danger.

You are not obliged to house her with you, and if you wish to do so, you must seek your wife's permission. What you called "misbehavior," if it constitutes abuse, then it is an act of disobedience to parents ( 'uquq ), and 'uquq is a grave sin and a major offense. Its expiation is sincere repentance. Strive to honor your mother and be good to her.