Does a person's prohibition of another, by making them forbidden with the inviolability of a mother and sister, or a father and brother, render them unlawful for marriage? And is there expiation for that?

What was uttered by both of you is a meaningless utterance and has no effect, because a man's prohibition of a woman and likening her to one of his unmarriageable kin (mahārim) is ẓihār. The ẓihār of a man from a non-mahram woman is not valid according to the majority of scholars, due to Allah's saying: {And those who declare ẓihār from their wives}, and this indicates that ẓihār can only be from a man's wives, and a non-mahram woman is not among them.

The Hanbalis, however, are of the view that this constitutes ẓihār and necessitates expiation before marriage, because it is an oath that ends with expiation, and thus it is valid to be contracted before marriage.