Is one obligated to help a sister who has separated from her husband, abandoned prayer, and removed her hijab, while her husband refuses to financially support her and her children until she returns to him or sends him the children?

You are not obligated to spend on your sister and her children, because their maintenance is originally the responsibility of her husband, and she is disobedient (nashiz). It is obligatory for her to return with her children to her husband's country. You must advise her, enjoin good upon her, and forbid evil.