What is the ruling on shunning relatives due to their non-observance of prayer and hijab and their commission of reprehensible acts, and how should one deal with them and invite them to adhere to the commands of religion?

The feeling of alienation among family members is common, but your steadfastness and patience will lead to great work for the religion of Allah, bearing in mind that what afflicted the early Muslims was far greater. To deal with your family: have sincere intentions, be a good role model, do not compromise on Islamic law, choose the wise among them, boycott negative gatherings, vary your methods of Da'wah (calling to Islam), and adhere to supplication (Du'a). If your attendance at their gatherings leads to the denunciation of evil and a positive influence, then it is obligatory for you to go to maintain kinship ties and denounce evil. Otherwise, stay away. As for their argument using Allah's mercy to commit evil deeds, this is wrong, for Allah's mercy is for the righteous and the pious. Allah is severe in punishment just as He is All-Forgiving, All-Merciful.