What is the correct Islamic course of action towards a brother who attempts to assault his sister and talks to her about sex and suspicious websites, taking into consideration informing the parents or shunning him?

Adultery is one of the gravest sins, and committing it with a relative is even more sinful. If your brother attempts to commit indecency with your sister, he has disobeyed his Lord and severed ties of kinship. The duty is to advise him and remind him to fear Allah. It is preferable that your sister undertake this advice. If he does not desist, there is no harm in threatening to inform his father, while frequently supplicating for his guidance. Your sister must treat her brother as a non-mahram with regard to hijab and being alone with him, as long as he remains in this state.