How can one balance their desire to change their job and develop themselves to improve their financial situation, with pleasing their father who opposes this change, knowing that the father did not search for an alternative job for his son, which led to the son feeling distressed towards his father?

A father does not have the right to prevent his son from taking a job that benefits him and causes him no harm, because "there is no harm nor reciprocal harm." The son should discuss the matter gently with his father, explaining his need for the new job and its benefit to him and his family. He should also seek the help of those who hold influence with his father to speak with him. If the father remains unpleased, there is no sin upon the son for moving to the new job, provided he apologizes to his father and is keen on honoring him and showing him kindness. Indeed, honoring one's father is among the best acts of obedience and most beloved to Allah, and it is the middle gate to Paradise.