Is it obligatory to say "Assalamu Alaikum" upon entering the mosque after the opening Takbir (Takbirat Al-Ihram) while the Imam is reciting the Quran, and do the worshippers return the greeting by gesturing?

It is not legislated for a Muslim to greet those who are praying, as it distracts them and occupies their hearts from the prayer. If someone greets those who are praying, they should respond by gesturing, not by speaking.