Is trying to beautify one's voice during prayer for those praying behind the Imam considered ostentation (riya') that invalidates the deed, or is it permissible because of the Prophet's (peace be upon him) practice?

Improving one's voice when reciting the Qur'an is legally (religiously) commendable, as indicated by the Prophet's (peace be upon him) saying: "Beautify the Qur'an with your voices." This is not blameworthy affectation or showing off, and it invites hearts to listen and reflect. The Prophet (peace be upon him) approved of vocal beautification in recitation, as in his saying to Abu Musa al-Ash'ari: "You have been given a wind instrument from the wind instruments of the family of David." Improving one's voice softens the heart and increases reward.

If the intention behind improving the voice is to attain humility and be affected by the Qur'an, or for listeners to be affected by it, then it is a legitimate and desirable act, and not showing off. However, if the intention is to display the beauty of one's voice and good recitation to gain praise and commendation, then this is a type of showing off, and a Muslim should strive to make their actions sincere for the sake of Allah.