Does my quarreling with my brother, who abuses and threatens me, count as a sin for me? And what is the legal ruling if I am forced to defend myself and get into a physical fight with him?

You must preserve prayer, for it is the greatest pillar of the religion after faith. Whoever neglects the right of Allah will be even more negligent of the rights of others. If communicating with your brother leads to harm in your religion, soul, or worldly life, then it is permissible to abandon him, based on the saying of Ibn Abd al-Barr: "Many a beautiful abandonment is better than a harmful association." It is not permissible to initiate fighting him, but if he fights you, defend yourself with the lightest [means] then the lightest. If he kills you, you are a martyr, and if you kill him, he is in the Fire. You should continue advising him with kind words and seek help from righteous people.