Is congregational prayer permissible behind an Imam who does not vocalize the "Takbir" (Allahu Akbar) and "Sami' Allahu liman Hamidah" (Allah hears those who praise Him), and what is the ruling on prayer if the Imam is mute?

The prayer is not invalidated by the imam's failure to vocalize the takbir, as this is one of the Sunnah (definition) acts of prayer. It is not permissible for a mute person to lead those who can speak in prayer. However, your prayer is, in sha' Allah (God willing), valid if you prayed behind an imam whose condition you were unaware of (whether he was mute or not), as it is more likely that he was not mute.