What is the religious ruling on a husband divorcing his wife by saying, "I have divorced her definitively," and swearing to that, while knowing that the divorce did not occur, then divorcing her to himself, and after she returned home and asked him what happened, he denied the divorce, then took her back to himself? Does the divorce take place in this case?

If you did not divorce your wife in reality, but rather falsely claimed that a divorce occurred and swore to it, then according to the Shafi'i and Hanafi schools, the divorce does not take effect religiously. That is, your wife remains your wife in what is between you and Allah Almighty, but it does take effect judicially. As for the Hanbali school, they hold that it takes effect both judicially and religiously. The first opinion is the stronger one.

As for the divorce and return (ruju') that occurred between you and yourself, if you intended it as a mere thought, then it is as if it never happened. But if you uttered the divorce, then it has taken effect, and you may return to your wife before her waiting period ('iddah) ends, provided you have not divorced her three times.

You must repent and seek Allah's forgiveness, as you have committed a grave sin by falsely swearing that a divorce occurred. This is called an "oath that immerses the swearer in sin" (yamin ghamus). There is no expiation for it according to the majority of scholars.