Is the prayer of one who recites aloud in the first rak'ah of the Sunnah prayer of Maghrib, then recites silently in the second, and performs sujood as-sahw (prostration of forgetfulness) valid?

Vocalizing in a loud prayer and whispering in a silent prayer is a Sunnah, not an obligation. So, whoever contravenes this, their prayer is valid. The majority of scholars are of the view that the Sunnah in the regular supererogatory prayers is to whisper the recitation. There is no harm in a person hearing themselves recite in the regular supererogatory prayers if a legitimate benefit results from it, such as attaining devotion (khushu') and repelling whispers (waswas), provided that it does not disturb anyone. Your whispering in the regular supererogatory prayer of Maghrib and other regular supererogatory prayers is correct, and your past prayers are valid.