Does stopping the payment of my brother’s debts constitute abandoning him, or does continuing to pay squander the right of my wife and children, especially after selling properties to pay off a portion of the debts without that being sufficient to pay them off completely?

There is no religious obligation to repay a brother's debt, and there is no sin in not doing so. A relative is not obligated to repay the debt of another relative, even if it is a father. Your voluntary repayment of it is commendable and is not considered a squandering of the rights of your wife and children, provided that repaying it does not prevent you from spending on them appropriately, for spending on one's wife and children takes precedence.