Is a brother's feeling of jealousy over his unveiled sister and his violent behavior towards her, under the pretext of their father's death and his right to do so, correct, or is it from the whispers of Satan, and is he considered a "dayyuth" because of it?

You must teach your sister and command her to observe the hijab, and forbid her from displaying her adornments (tabarruj). It is not permissible for you to condone her in this.

You do not have the right to hit her as a punishment or for discipline.

As long as you disapprove of her display of adornments, you are in no way considered a cuckold (dayooth).

Strive to teach your sister the rulings of the Sharia, remind her of Allah, and deal with her with gentleness and forbearance. Be diligent in supplicating for her guidance.