What is the ruling on dealing with a mother who frequently invokes evil upon her children for trivial reasons, and is it permissible to invoke evil upon her in this situation, or to supplicate to Allah that her invocations not be answered?

A Muslim is not permitted to supplicate against themselves or their offspring, unless they have been wronged. This mother must repent for what she is doing. You can supplicate that Allah repels from you the evil of what she is supplicating with, but you are not permitted to supplicate against her, as that would be an act of disobedience.