Is insisting on and asking about the condition of a friend who has openly committed a sin considered a transgression, and what is its expiation?

You should have refrained from inspecting what your companion was hiding, because being keen to look into it and ask directly about it is a form of espionage forbidden by Islamic law, as stated in the Almighty's قول (statement): "O you who have believed, avoid much [negative] assumption. Indeed, some assumption is sin. And do not spy or backbite each other. Would one of you like to eat the flesh of his brother when dead? You would detest it. And fear Allah; indeed, Allah is Accepting of repentance and Merciful."

Espionage does not require it to be from behind a person, and this espionage is the fruit of ill-will. Its expiation is by covering for your companion and repenting.