How should a person behave with his non-mahram female relatives with whom he was not nursed, when they become angry at him for avoiding them, claiming they are his sisters?

You did well by refusing to shake hands with your non-mahram female relatives. You must clarify to them that this is forbidden, based on the Prophet’s (peace be upon him) saying: "For one of you to be stabbed in the head with an iron needle is better for him than to touch a woman who is not lawful for him." It is permissible to greet them with salaam and inquire about their well-being without shaking hands, provided there is no khalwa (seclusion) and no fear of fitna (temptation).