What is the religious ruling on dealing with brothers who do not respect their older brother, who respond to kindness with mistreatment, and whose actions escalate to verbal abuse and physical assault, and when the older brother complains to his father, he does not do him justice, but rather gets angry with him if he punishes his brothers? Is it permissible for the older brother to avoid them, and is his oath not to consider them his brothers considered a severing of kinship ties?

You should treat your siblings with gentleness and kindness, and endure their misdeeds because you are the elder brother. It is wrong to hit your sister merely for being insolent, unless she has physically assaulted you. It is not permissible to sever ties with them, as that is an act of disobedience. The best one who maintains ties is he who connects with his kin when they sever ties. If your intention behind the oath was to abandon them and stop speaking to them, then this is an oath to commit an act of disobedience, which is not permissible to fulfill, and requires an expiation for breaking an oath. However, if your intention was that they are not fulfilling their duty towards you as siblings, then there is no blame upon you.