What is the Sharia ruling on a brother committing adultery with his brother's wife, and is it forbidden or permissible for the siblings to ostracize their brother because of his deed?

The brother must repent to Allah for committing adultery, as it is one of the most heinous sins. Ostracization was legislated to deter the sinner. If it does not achieve that, but rather increases his persistence, then drawing near to him and advising him is more appropriate. If he repents, then it is not permissible to ostracize him.