Is it a sin to close the door leading to the house's courtyard, which is used by the troublesome sister-in-law?

The aforementioned sister-in-law's actions, such as throwing trash, pouring water in the courtyard, causing problems, and her husband's (your brother's) estrangement from his family, are all wrong and inappropriate behaviors. However, this ill-treatment should be met with good, for Allah Almighty says: "Nor can a good deed and a bad deed be equal. Repel [evil] with that which is better" [Quran 41:34]. Furthermore, being kind to relatives who sever ties and mistreat you ensures that the doer of good has a supporter from Allah. Therefore, closing the aforementioned door is not correct; it should be opened.