Is reporting the misbehavior of one's brother's wife, who was chosen to care for his children, considered a mistake that warrants family ostracism and severing ties of kinship, especially when the one speaking the truth is accused of wrongdoing?

Your duty was to advise the girl and warn her of Allah's punishment. If she persisted in her sin, it would be permissible to threaten her with informing someone whose authority she fears, provided that the intention is to advise and not to expose, and in a gentle manner. You erred in informing your father and brother about her in that way, for wisdom is required in inviting to the way of Allah. It is not permissible for you to boycott your father or your brother, and you must strive for your father's pleasure and obey him unless he commands disobedience. Speak the truth and forbid evil with wisdom and good counsel, and enjoin patience for any harm that may result from it.