Is it permissible for the father to request the mother's oath regarding his relinquishment of custody to her, given that the judge rejected this on the grounds that the matter does not pertain to financial assets and the father has no evidence?

Custody is the mother's right unless she marries. If she marries, custody transfers to the female who is next most entitled to it, according to the jurisprudential order. If the children's grandmother is the most entitled to custody, then it is hers by right according to Islamic law, regardless of the father's waiver or his oath. The judge's statement that there is no oath involved here because it is not related to money is correct, as oaths are administered in matters of wealth or what leads to it.