What is the optimal course of action with siblings who mistreat the questioner and his family, despite his continuous attempts to show them kindness and maintain kinship ties, taking into account that his mother is greatly affected by the severance of this relationship?

What you are doing is correct. A Muslim does not reciprocate estrangement with estrangement. Muslim reported from Abu Hurairah that a man said: "O Messenger of Allah, I have relatives with whom I maintain ties, but they cut me off. I treat them well, but they treat me badly. I am forbearing with them, but they act ignorantly towards me." He said: "If what you say is true, it is as if you are feeding them hot ashes, and you will continue to have a supporter from Allah as long as you remain in that state." This action pleases your Lord and is an act of obedience to your mother. So, continue with this goodness, and advise your siblings, reminding them of the virtue of maintaining kinship ties and good character. Do not cut off your relationship with them so that you are not deprived of the reward for maintaining kinship and for patiently enduring their harm.