What is the ruling regarding a husband's retraction from paying a sum of money to his wife after he challenged her and swore he would give it to her?

What you mentioned allows for two possibilities:

1. Ja'alah (stipulated reward): If a person says to another, "If you do such-and-such for me, then you will have such-and-such," and the action is permissible, then it is lawful. If the addressed person performs the action, he is entitled to what was promised, and the promiser is not allowed to retract.

2. Permissible promise: If a person says to another, "If I do such-and-such, I will give you such-and-such," then this promise is not binding according to the majority of scholars, and fulfilling it is recommended. If he does not fulfill his promise, he is liable for an expiation of an oath.