What is the ruling on one who sees his Muslim brother committing an open sin and does not condemn him for it? And is there expiation for that?

Denouncing evil (munkar) is, in principle, a communal obligation (fard kifaya). If some members of the community undertake it, the obligation is lifted from the rest. However, for anyone who witnesses it firsthand, it becomes a personal obligation to denounce it with gentleness and wisdom, regardless of whether they have a relationship with the perpetrator or not; the bond of Islamic brotherhood suffices. If one refrains from denouncing it despite having the ability and no fear of causing greater corruption, then they may incur sin. This is why Ibn Abbas sought forgiveness when he cried out of fear of remaining silent about evil, saying, "How many evils have we seen that we did not forbid!"