Was the oath I swore against my brother a vain or a binding oath, and do I owe expiation for it, especially since I fasted three days and do not remember if I paid it out?

It is clear that you are afflicted with some form of waswasah (obsessive whispering); so strive against these waswasah and turn away from them, and pay no attention to any of them. If you doubt whether this oath was from the category of laghw al-yamin (a thoughtless oath) or a binding oath, then the default is that you are absolved, and no expiation is required of you. If you are certain that the oath was binding, but you doubt whether you have already expiated for it or not, then if you are afflicted with waswasah, cast these whisperings aside and pay them no heed. Otherwise, the default is that this expiation remains due until you are certain that you have been absolved from it.