Why does the brother see his mother as a devil and curse her when she wakes him up from sleep?

It is not possible to definitively state that your brother is afflicted by possession or magic. You can ask some trustworthy ruqyah practitioners to perform ruqyah on him. If he is of sound mind, not bewitched, and speaks during sleep before waking up, there is no sin upon him, because the pen is lifted from the sleeper. However, if he curses his mother while awake and knowing that she is his mother, then he deserves severe disciplinary punishment (ta'zir); for cursing one's mother is one of the gravest major sins, as it combines disobedience ('uquq) and cursing. The Prophet, peace and blessings be upon him, said: "Among the gravest major sins is for a man to curse his parents." So, let him fear Allah and repent to Him if he curses her while awake and conscious.