How can one be dutiful to a disobedient mother who invokes evil upon her children and grandchildren and possesses bad characteristics? Is it sufficient to avoid her and not raise one's voice at her to avert the harm of her invocations?

If the mother is engaged in the grave sins mentioned in the question, her right remains great and honoring her is obligatory, no matter how bad she becomes. This is because Allah Almighty commanded honoring parents even if they strive to make one associate partners with Him. Allah Almighty says: "But if they endeavor to make you associate with Me that of which you have no knowledge, do not obey them, but accompany them in [this] world with appropriate kindness."

Part of honoring her is striving to reform her through supplication and advice with wisdom and good counsel. If she persists in her wrongdoing, you are excused. As long as you are compelled to live with her, you must be patient. There is no harm in disliking her reprehensible actions, but beware of any harm towards her, as that would be an act of disobedience. The Prophet, peace and blessings be upon him, forbade supplicating against one's children.