Would a son be considered undutiful to his mother who creates problems with his wife and demands that he treat her harshly, due to his reduced communication with her and his refusal to fulfill all her demands after she affected his wife's mental and nervous health?

If "I separated from my wife" means moving to a separate residence from the mother, then this is not considered disobedience (uquq), because the wife has the right to an independent dwelling, and she is not obligated to live with her husband's mother. However, if the mother asks to speak with you, then obedience to her is obligatory, unless there is harm to you in doing so. Disobeying her without harm might be considered disobedience (uquq) if she is hurt by it, because disobedience (uquq) is any harm caused to parents that is not minor by custom.