How can one deal with a brother who does not pray, mistreats his family, causes problems, works in music, and does not accept advice, which has caused psychological distress to his family?

You have done well by supplicating for your brother's guidance and by continuing to do so, for his condition, as you mentioned, combines a number of reprehensible acts, the most significant of which are abandoning prayer and disobeying parents. Advice is obligatory for him. If he does not accept it, then seek someone who holds a respected position with him to advise him and remind him of the punishment for injustice and cutting off ties of kinship, which is hastened in this world. If he benefits from the advice, then praise be to Allah. Otherwise, you may shun him if you hope it will benefit him. If not, then it is better to conciliate him. If the matter necessitates threatening him with reporting his case to the responsible authorities to stop his harm from reaching you, then do so. There is no harm in actually resorting to this if you hope for a preponderant benefit therein.