Is there anything legally binding upon me for not fulfilling a promise to pay a sum of money as a prize for a joke, given the other party's anger and unforgiveness towards me, and in light of my not remembering if I had sworn an oath to that effect?

What occurred from you is a promise of a gift, and it is legislated for you to fulfill it, but it is not obligatory upon you. So, your brother has no right over you. And your doubt that you swore an oath has no consequence, for the basic principle is the acquittal of responsibility and that you did not swear an oath.