What is the reason for not succeeding in a job despite diligent prayer and worship, and how can one deal with despair and loss of self-confidence afterward?

There is no specific reason for what you mentioned, but a righteous Muslim should not grieve or despair. As long as he adheres to his religion and obeys God's commands, all his affairs are good. A Muslim should stay away from despair and despondency, as Allah warned against them in His Book. If sadness creeps in, he should reassure his heart by remembering Allah and by the reported supplication: "O Allah, I am Your slave and the son of Your male slave..." For whoever says it, Allah will remove his worry. A Muslim's supplication will not be lost. Either Allah will hasten what he wished for him, or He will store its reward for him, or He will avert evil or affliction from him in this world. And the answer will be one of three: hastening the supplication, or storing it for the Hereafter, or averting an equivalent evil. A Muslim must make many supplications and hope for a response, and he must ensure that the conditions for the supplication's acceptance are met and its impediments are absent.