How can one deal with the problem of severing ties with one's brother because his daughter insulted one's deceased mother, and the brother refused to apologize or take a stance on the matter, knowing that visiting relatives in their home causes deep psychological pain?

The girl's action, if your claim is true, is not acceptable. The adults in the house should be attentive and keen to be good role models for the children. As for your disciplining the girl by hitting her, you should not have done that due to her young age and the presence of her father. You could have alerted him to the necessity of nurturing the morals of his children. It is not permissible for you to sever your ties of kinship with them because of what happened; rather, work on maintaining those ties.