What is the appropriate way to deal with my sister and mother who reject my sister's commitment to the legitimate hijab and her behavior, despite my repeated attempts to advise, enjoin good, and forbid evil, especially after being expelled from the house and being psychologically affected?

You should continue to advise your sister gently and kindly. If she responds and repents, then praise be to Allah. If she does not repent, then consider whether shunning her would benefit her or increase her stubbornness. As for the mother, if she condones and encourages evil and trivializes entering Hell, then she has done wrong by expelling her daughter. You must honor her, pray for her, and for those who advise her. The father must be firm with her, reform her, and restrain her from causes of corruption, for he is responsible for her before Allah.