Is it considered permissible (halal) or forbidden (haram) for a man to divorce his wife in front of a judge and witnesses, while not intending the divorce, and for them to continue living together as husband and wife?

If you pronounced the divorce in front of the court and witnesses, then the divorce takes effect, whether you intended it or not, because the Prophet, peace and blessings be upon him, said: "There are three matters whose seriousness is serious and whose jest is serious: marriage, divorce, and رجعة (returning to one's wife after a revocable divorce)." Scholars have agreed that a clear pronouncement of divorce by a sane, adult person holds him accountable, and his saying that he was joking does not benefit him. If this was the first or second pronouncement, then your cohabitation with your wife with the intention of continuing the marriage is considered a رجعة, and she is now your wife.