Is shunning one's mother and sisters for not adhering to the Islamic dress code and shaking hands with men a correct action or not? And what is the duty towards them?

You must advise your mother and sisters to do good and forbid evil, such as abandoning the hijab and shaking hands with non-mahram men, but without mistreating your mother, or speaking harshly, or boycotting her; for enjoining and forbidding your mother is not like enjoining and forbidding others. Allah has commanded us to accompany our parents with kindness, even if they strive to make us associate partners with Him. So, you must repent, apologize to your mother, be good to her, and promptly advise them with gentleness and wisdom.