Is the mother sinful for giving a portion of the Zakat money to the questioner's sister to buy books and study necessities, knowing that the brother used to support the family and the household expenses were not enough for necessities?

Financial support is obligatory for every inheriting relative. This brother is not an heir to his sister, so her financial support is not obligatory for him. Consequently, it is permissible to give her zakat if she is poor. If your brother appointed his mother to disburse the zakat, there is no harm in her giving a portion of it to his sister if she is eligible.