Is losing a job opportunity due to a doubt considered a punishment, and would it have been better not to leave that job?

You have done well by abandoning this work, which involves prohibitions such as a woman traveling without a mahram (definition), recording usurious loans, and selling cigarettes. It is impermissible to work in such a field, except for the one in dire necessity who cannot find permissible work and fears perishing or severe hardship. This abandonment is a divine success, so pay no attention to those who describe it as evil eye, envy, or a missed opportunity. Rather, it is better to leave it to avoid dispute and steer clear of ambiguities. You must disregard the words of these people, seek Allah's help, and supplicate frequently.