To whom does the custody of my brother's children go after his wife remarries, and would it be to my mother or hers, given that they have no paternal uncle?

The mother has the primary right to custody of her young children, as long as she does not marry a non-mahram (definition) (stranger) or have any impediment. The majority of scholars hold that custody transfers from the mother to the maternal grandmother, except for the Hanbalis, who prioritize the father over others besides the mother. If the children's grandmother – their mother's mother – is among those eligible for custody, she has the greater right to their custody after their mother. If a dispute arises, the Sharia (definition) court is the authority that will rule on the matter in a way that serves the best interests of the children.