What is the ruling on an oath if I said to my wife, "If I do not rupture the hymen tonight, you will be حرمت (unlawful) to me like my mother," and the hymen was not ruptured despite attempts?

What you mentioned falls under the category of conditional zihar, according to the most authoritative opinion. You are obligated to perform an expiation for zihar because you intended that she would become forbidden to you if the hymen was not broken, and indeed it was not broken. The expiation for zihar is: freeing a slave. If you cannot find one, then fasting for two consecutive months. If you are unable to do that, then feeding sixty poor people. This expiation must be performed before having intercourse. It is advisable to first ascertain whether the hymen was broken or not, which can be done by consulting a trustworthy female doctor. We also advise presenting the matter to the Sharia courts, as a judge's ruling resolves disagreements.