Is it permissible to boycott a brother who mistreats his parents and siblings until he becomes dutiful towards them, even if the boycott lasts for years?

What your brother is doing to his parents is a reprehensible act and one of the gravest sins, for the rights of parents are immense and are coupled with the right of Allah in the Holy Quran. Children must honor and be kind to their parents, obey them in what is good, and not mistreat them. Disobeying parents is a major sin. You should remind your brother of Allah and the consequences of disobeying parents, and explain the high status of filial piety in Islam. Various methods of advice should be used, including instilling in him fear of Allah's punishment. Abandoning a disobedient person is legislated to discipline him and warn others. One should consider the شرعية ( Shar'i ) benefit expected from abandonment; if the benefit outweighs the harm, it is permissible; otherwise, it is not.