Is what the questioner did, by looking at his brother’s messages on his private phone, considered prohibited spying? And should he continue to track his news so that he remains informed of what he does? And should he tell his father what he discovered, despite his fear that the father might act in a way that causes more harm than good?

We hope you are not sinful for spying on your brother's phone due to your suspicions about him. Once you are certain of the matter, you must advise him and explain that what he is doing is among the major sins. If he does not respond, you must escalate the matter to someone who can restrain him, such as your father, while advising him to choose a good approach. You must also advise your brother to stay away from bad company and invite him to Allah with wisdom, trying to attract him to the path of righteousness, and dedicating some of your time to that.