Is it permissible for a Muslim to study in the mosque between Maghrib and Isha if the time is short and it is difficult for him to leave and return, and what should he do if the mosque guard refuses, especially with people talking about worldly matters inside the mosque?

It is better for the worshiper to remain in the mosque, awaiting the 'Isha prayer, due to the immense reward in doing so. This includes the angels praying for him, and him being considered in prayer as long as he is awaiting it. However, if it is difficult to stay in the mosque for some reason, there is no harm in returning home. If the home is so far that the call to prayer cannot be heard without a loudspeaker, then congregational prayer is not obligatory.