What is the ruling on the cash prize resulting from a competition in which the sister swore not to open the book, but then opened it for some questions? And how can she be advised?

Allah is self-sufficient and has no need for His servant to torment himself. Your sister must repent and seek forgiveness, and expiate her oath if she erred. Regarding the prize, if the organizing body permits the use of books, then there is no problem. Otherwise, she must inform them of what happened and leave the decision to them.