What is the ruling regarding a brother who openly abandons prayer and treats his parents harshly, and what is the way to deal with him?

You must make clear to your brother the vastness of Allah's mercy and His acceptance of the repentance of those who repent, no matter how great their sins; because his description of himself as lost and his emphasis on it indicates despair, which is pure harm. Abandoning prayer and disobeying parents are among the greatest major sins. You must prevent him from harming his parents, even if it comes to informing the authorities and preventing him from entering the house if he harms them.