Is it true that a man's swearing by divorce diminishes one of his Houris in Paradise, and how many Houris in Paradise are there for one man?

Swearing by other than the names and attributes of Allah is contrary to the saying of the Prophet, peace and blessings be upon him: "Whoever swears an oath, let him swear by Allah or remain silent." As for the reduction in the number of houris due to swearing by divorce, there is no text that mentions this. The minimum share for a man of the houris in Paradise is seventy houris, in addition to two wives from the women of this world.