What should I do about my brother who curses my mother and hits me, given that he is moody due to being spoiled previously, and I do not hit him back for fear of sin?

Disobeying one's parents is one of the gravest sins, and Allah and His Messenger have forbidden it. One form of filial disobedience is being denied entry into Paradise. Your not striking your father when he strikes you is the best course of action and closer to piety, and Allah may forgive you for that. The father's injustice and disobedience do not justify severing ties with him; rather, it is obligatory to maintain ties with him, remind him of his transgression, not despair of his heeding the admonition, and supplicate for him. It is permissible to boycott him if it helps bring him back to the right path.