What is the ruling on the prayer of someone who vowed not to pray in a specific mosque, then prayed in it out of forgetfulness?

If your statement "I will vow" was merely a promise to vow to stop praying in the mosque, and the vow did not actually occur, then nothing is incumbent upon you. However, if you actually intended it as a vow, and you were able to pray in congregation at another mosque, then this is a vow of contention (nathr lajaj) and you broke it due to forgetfulness. To avoid disagreement regarding the waiver of expiation due to forgetfulness, it is more prudent to offer the expiation for a broken oath (kaffarat yamin).