How do I deal with my older brother who curses my mother and me, while my father ignores the situation? Should I leave him, hit him, or complain to a relative?

Your brother is in great danger because of his cursing of your mother, which is one of the gravest sins. You must advise him by explaining the severity of his action and that it displeases his Lord. If he responds, then all is well; otherwise, report him to someone who can deter him. If you defend yourself to the extent of his transgression, there is no blame, though forgiveness is better. If he is not deterred and persists in his aggression, then it is permissible to ostracize him, hoping he might be deterred. It is also permissible for your mother to ostracize him until he repents.