Is it permissible for the Imam to allocate a large portion of the money designated for the poor to his relatives, and is he accountable for that?

If your father and brothers are in need and are eligible for charity, then there is no harm in what you have done. It is also permissible for you to take from this money to pay off your debts that you cannot settle, provided that the owner of the money has given you a general authorization to spend it on the poor, the destitute, and the needy, and you do not need to seek his permission. This is supported by the hadith of Ma'n bin Yazid, who narrated that his father placed dinars for charity with a man, and Ma'n took them for himself. The Prophet, peace and blessings be upon him, said to his father: "You will have what you intended, O Yazid," and to Ma'n: "And you will have what you took, O Ma'n," because Ma'n was among the poor and needy who were eligible for charity. This indicates acting upon general authorizations as they are.