Is a husband entitled to take an oath to prove his claim of discord and dispute, given that the main reason is his suspicion that his wife and her mother are practicing magic, and is this oath considered a "Ghamus" oath?

If you swore truthfully about the occurrence of dispute and discord, there is no blame. However, lying in an oath is not permissible and is one of the major prohibitions. Accusing your wife of witchcraft does not justify lying. Rather, you have the right to present evidence for your accusation against her. If you cannot, then give her her rights and divorce her.