How can one deal with a mother who opposes the hijab and encourages sisters to dress immodestly, and with a family environment that raises concerns about raising children, especially with a feeling of being lost after repentance?

The statement that hijab (definition) is not beautiful, or that it is a complication and a source of misery, is clear disbelief (kufr), because it is a denial of the Quran and the Sunnah (definition). However, the one who says it is not ruled to be a disbeliever until the conditions are met and impediments are removed. Mixing with non-mahram (definition) men is forbidden and contrary to Allah's commands, and the Prophet, peace and blessings be upon him, forbade it even in places of worship. You must advise your family regarding hijab and abandoning tabarruj (wanton display) and mixing, and guide them towards righteous company, while maintaining dutifulness to your mother. Marriage should not be abandoned; it is the Sunnah of the Prophets. If your family's situation does not change, you can live far from them to protect your children from negative influence.