Is it permissible to give zakat to one of the deserving brothers who lives in the family home, as part of the debt owed by him to the rest of the heirs?

If the heirs agree to your two brothers residing in the inherited house, then that is permissible. It is also permissible to transfer ownership of the house to the two brothers after its value is determined, provided they pay each heir the value of their share. Regarding your indebted brother, if he is unable to repay his debt, then he is one of the categories eligible for Zakat. You may give him your Zakat for him to return it to you as part of the debt, unless there is a condition or agreement to that effect, otherwise your Zakat will not be valid.