Is it permissible to kill an insane brother who harms neighbors and محارم (unmarriageable kin) and cannot be cured or institutionalized in an asylum?

Deliberately and wrongfully killing a soul is one of the greatest sins, and a severe warning has been issued against its perpetrator in the Qur'an and Sunnah.

The killing of a soul is not permissible except in one of three cases: the married adulterer, a life for a life, and one who abandons his religion and separates from the community. The execution of such a حد (prescribed punishment) is carried out by the ruler or his deputy.

Based on this, your brother, who is afflicted with madness, is not held accountable, so it is not permissible to kill him. The obligation is to admit him to a mental hospital, or hand him over to the authorities, or restrain him at home. It is also appropriate to supplicate for his healing and present him to someone who practices lawful ruqyah (incantations).