What is the ruling on a mother disposing of her son's money without guarantees, and preferring her grandchildren from relatives over her grandchildren from her son, despite their attachment to her?

You must lighten the matter for yourself and not burden your soul with more than it can bear, especially since your mother has a great right over you. Writing down a debt is commendable, not obligatory, and you were right to write down your brother's debt in obedience to your mother and in compliance with the Sharia. You should interpret her request to write down the debt in a good light. Do not let Satan corrupt your relationship with your mother because of her past mismanagement of your money. As for your children, if she loves them, then seek the reason for her refraining from visiting them; it might be due to her relationship with your wife. If her refraining indicates something she holds in her heart against you, then work on appeasing her and being gentle with her, and seek help from righteous people for reconciliation, while supplicating that Allah rectifies the situation.