Is it permissible to cut ties with the elder brother who did not care for his mother after their father's death, and did not help his younger brother with his studies, despite the father's great sacrifices for him?

It is not permissible to sever ties with a brother, because cutting off kinship is forbidden and one of the major prohibitions. In the Hadith: "The one who severs ties of kinship will not enter Paradise." The brother's shortcomings or harshness do not negate his right to maintain ties, for the one who maintains ties is not he who reciprocates those who maintain ties with him, but rather he who connects with his kin when they sever ties.