What is the ruling on taking a portion of money directly from the bank without referring to the agent, especially since the agent has been unfair in distributing the due amounts and has placed the funds in his private account?

The agent or guardian's disposition of his principal's or the ward's property must be done with justice and for their benefit. He must give those who have rights their due. So, if someone requests their share of the inheritance, and they are not legally incapacitated, their right must be given to them. If your brother withholds your right from you, you can resort to the judiciary to compel him to divide it. If you cannot prove your right judicially, then you may take only what you are certain is your right, which is known as the issue of al-Zafar (taking one's right without judicial intervention).