How can one deal with a teenager who is in a relationship with a non-Muslim girl at home, with suspicions of transgressions occurring, and the non-Muslim mother showing no interest in the matter?

A Muslim's residence in the lands of the disbelievers poses a danger to his religion and morals. What your brother is doing, establishing a forbidden relationship with that girl and secluding himself with her, is a clear abomination that should not be overlooked. You must explain the ruling of the Sharia to him. If he does not desist, then you must inform whoever is capable of stopping him, or do what you can to prevent him from these abominations, provided that it does not result in harm to you or a greater corruption. Your monitoring of them is not forbidden espionage; rather, it is required to prevent evil. You should also invite your mother to Islam and your brother to repentance and righteousness.