Is it permissible for my brother to join the prayer as our imam after my mother has already started leading it, and will my brother receive the reward for congregational prayer in this situation?

If your brother joins the prayer after you and your mother have already entered into the state of ihram for prayer, it is not permissible for him to become your imam or your mother's imam, and your prayer becomes invalid by this. This is due to what results from it: for you, a transition from following one imam to following another imam; and for your mother, a transition from being an imam to being a follower. This is not permissible except in the case of the imam's succession (istikhlaf).