Would a son be considered undutiful to his mother if he filed a lawsuit against her to recover his money that she placed in her brother's hands, knowing that the mother treats him poorly, curses him, and has caused him psychological and material harm, and would his prayers be accepted in this case?

If your money is a trust with your mother, then demanding it back and taking the matter to court after her refusal is not a forbidden act of disobedience (uquq). Nevertheless, you must still be dutiful to her, accompany her with kindness, and refrain from supplicating against her, even if she is unjust. The supplication of parents against their children is not answered if it is without right. Finally, no hardship justifies your negligence in prayer; it is immense, the key to all good, and a helper in times of hardship.