What is the ruling on a mother continuing an illicit relationship with another man, and is the daughter sinful if she knows about it but cannot prevent her, and what can she do?

If your mother does not respond to your advice, threaten her by informing someone she fears knowing about this matter. If the threat does not work, then inform someone who can prevent her from committing this wrong. Do not abandon your dutifulness to her because of her disobedience, but rather be a good companion to her and supplicate frequently for her guidance.