What should I do, as I borrowed for Hajj with interest, guaranteed by my certificates, and I did not know of its prohibition, and I have already paid the money, and now I am working and the amount of the certificates covers the loan?

You have sinned by placing your money in an interest-based bank and by borrowing with interest. You must withdraw your money from the interest-based bank and place it in a non-interest-based bank. You should also hasten to repay the interest-based loan to reduce the interest, as an act of repentance to Allah. Your Hajj is valid.