Is it permissible for a man to marry his brother's divorced wife from the mother's side, and is her daughter considered a mahram to him?

The girl is your brother's stepdaughter, not one of his mahārim. Therefore, it is permissible for you to marry her. As for your brother, she is mahram to him, and it is not permissible for him to marry her if he has consummated the marriage with her mother, due to the Almighty's saying: "And your stepdaughters under your guardianship, born of your wives to whom you have gone in—but if you have not gone in unto them, there is no sin upon you [to marry their daughters]..."