Did a divorce occur due to breaking an oath ("You are forbidden to me"/"You will not return here again"), or is expiation for an oath obligatory? And is it permissible to pay the expiation after three years?

Prohibiting one's wife is not permissible, but an oath by Allah Almighty is lawful. Scholars have differed concerning a husband swearing an oath to his wife by prohibition (of her), and our fatwa is that what the husband intended takes effect. If you intended divorce, then it has occurred, and your wife returns to you if you had intercourse with her before her waiting period expired and this was not the final divorce. If you intended ẓihār, then the expiation for ẓihār is incumbent upon you. If you intended an oath, or intended nothing, then the expiation for an oath is incumbent upon you. As for your statement, "If you go to your family, you won't come back here again," if you intended divorce by it, then it takes effect upon her departure. When calamities befall a servant, they should review themselves and repent to Allah from sins.