Why is accusing someone of adultery a graver offense than accusing them of apostasy?

Accusation without proof and declaring someone an unbeliever (takfir) without clear evidence are two grave matters and great sins. Allah has threatened those who accuse chaste, unsuspecting, believing women with a curse and a great punishment. The Prophet, peace and blessings be upon him, warned against declaring a Muslim an unbeliever, saying: "If a man declares his brother an unbeliever, then it has returned upon one of them." There is nothing to indicate that accusing a person of adultery is more severe than accusing them of disbelief. Flogging is a worldly punishment, lighter than disbelief, the evil consequence of which returns upon its pronouncer if the accused is not an unbeliever.