What is the next step to take with a brother who insults the religion, does not pray, and raises his voice at his mother, after repeatedly advising him?

This man is in great danger, for insulting religion is disbelief (kufr), abandoning prayer is among the greatest destructive sins, and disobeying one's mother is a tremendous evil. Therefore, he must be advised and made aware of the seriousness of his actions. He should either repent, or else it becomes permissible to shun him and cease speaking to him, especially if shunning him is likely to deter him.