What should I do to get rid of my guilt after my friend refused to forgive me and reconcile, on the condition that I lie to the family of the girl I was engaged to?

You made a mistake by divulging your friend's secret, and this is a sin from which you must repent, because divulging secrets is forbidden. The Prophet, peace and blessings be upon him, said: "If a man tells another man a narration and then turns, it is a trust." He also said: "Whoever confides a secret to his brother, it is not permissible for him to disclose it concerning him."

You must repent to Allah for this sin. Allah's right is fulfilled through repentance. As for your friend's right, try to seek his forgiveness. If he does not forgive you, then increase your good deeds and acts of closeness to Allah.