Is it permissible, in Islamic law, for a sister to inform her father about her sister's transgressions, and what is the most righteous course of action in this situation, especially given that prior advice proved futile and was met with denial and insults?

The default is to conceal and advise the sister privately. If she does not accept direct advice, then one should advise her indirectly, such as by gifting her an audiotape or a pamphlet, or by explaining the warning against disobedience without confronting her. If she still does not take heed, and if informing her father would deter her from the evil, then he should be informed, in order to achieve the greater good.