What is the optimal course of action towards a relative who has been openly practicing sorcery on the father for years?

A Muslim should not think of another Muslim except good, and Allah the Almighty has forbidden ill-thinking of a believer. If the father finds definitive evidence of his relatives' sorcery, he has the right to stop them through admonition and threats of complaint. If that does not work, he should refer their matter to the authorities to deter them. Your father should know that what charlatans and sorcerers propagate, convincing a person that he is bewitched by his relatives, is an endeavor by the devils among jinn and humans to sow discord and sever ties of kinship.