What is the ruling on a father-in-law's abnormal behavior towards his son's wife, such as flirtation, helping her, going out with her alone, and serving her, while she remains immodestly dressed? And is it permissible for them to remain alone in the house under these circumstances?

You should alert your brother to his wife's behavior towards your father, and he should monitor her himself, as spying on her is permissible when suspicion arises. He must be reminded of Allah and that he should be firm with his wife and father. If the wife does not rectify her ways and repent for her negligence in prayer, it is preferable for him to divorce her, as she might tarnish his honor. Divorcing a woman who is negligent in Allah's rights is recommended. Your father's threat to marry another woman over your mother because she enjoins good and forbids evil is a strange matter.