What is my duty towards my relatives who do not adhere to the Shar'i hijab, and is my not visiting them for this reason considered severing ties of kinship?

What you did is right, and it does not constitute severing family ties or extremism. The matter of free mixing between foreign men and women is serious, especially among young people.