Is shunning a relative who has anal sex with his wife considered a severance of kinship ties, especially since that led to shunning my sister who is married to him, because she refused to leave him?

It is not permissible to accuse a Muslim of disobedience without proof, for the default is to think well of him. If his committing a great evil is proven, then it is not permissible for his wife to obey him in it, and she would be sinful by her consent. As for boycotting relatives because of it, severing ties of kinship is among the major sins, unless it is for a legitimate benefit that deters them from what is forbidden. If boycotting does not work, then it is better to continue the relationship with advice given wisely.