What should a Muslim do if he doubts his disbelief due to his abundant sins and transgressions, and does not know whether they are major sins or not?

You should not succumb to whispers of disbelief. The fundamental principle is that a Muslim remains within Islam unless they commit an act that expels them from the faith. A Muslim, no matter how numerous their sins, even major ones, can hope for forgiveness as long as they do not deem them permissible. Major sins do not expel their perpetrator from the fold of Islam. Those who commit major sins from the Ummah of Muhammad (peace be upon him), if they die as monotheists and have not repented, are subject to the will of Allah; if He wills, He may forgive them by His grace, and if He wills, He may punish them by His justice, then release them by His mercy and the intercession of intercessors.

It is obligatory for a Muslim to avoid sins, both minor and major, for some minor sins may be major ones without him realizing it. The sinner must hasten to repent, for Allah accepts the repentance of those who repent and pardons evil deeds.