Does the presence of a dog in a separate apartment at the top of the house, which does not enter or pass through the ground-floor apartment and has its own private entrance, prevent the angels from entering the ground-floor apartment?

The presence of a dog in your brother's apartment does not affect the entry of angels into your house, as long as it is separate from it. It is obligatory upon your brother to fear Allah and to stop raising dogs unless there is a legitimate Shariah need.