Is the husband's saying to his wife, "I don't want you," considered a divorce, knowing that he did not intend divorce, and does the ruling differ if it is the wife who intends divorce by saying, "If you don't want me, tell me 'I don't want you,' and I will go to my family"?

Your statement to your wife: "I do not want you; my soul is with your family [meaning: go to your family's house]" is an allusive divorce (kinayah talaq). Divorce takes effect if you intend it with this statement. However, since you did not intend it, divorce does not take effect. The decisive factor is the husband's intention.