Is a wife's refusal to shake hands with her sister-in-law a reason for the mother to boycott her son, and what is the son's duty towards his mother in this situation?

A Muslim woman is not permitted to shake hands with a man who is not a mahram (definition) to her. What your wife did is absolutely correct, and your mother is not permitted to be angry about your wife's action, as this would mean being angry about obedience to Allah. We advise your mother to be happy with her daughter-in-law's uprightness and to help her obey Allah, rather than calling her to disobey Him. Indeed, calling someone to disobey Allah is among the greatest prohibitions, and she must repent from that. You must support your wife's stance and explain the truth gently to your mother, seeking assistance from people of goodness and righteousness.