Have you exceeded the limit in advising your sister not to seek out books containing aberrant ideas, and are you sinful or rewarded for that, and what are the limits of mutual advice for the sake of Allah?

If the dialogue and mutual advice between you were amicable, and you were sincere in advising her, then you are hoped to be rewarded. Mutual advice among Muslims is essential, as the Prophet, peace and blessings be upon him, said: "Religion is sincere advice," and as He said in the Hadith (definition) Qudsi: "My love is due to those who love each other for My sake, and My love is due to those who give sincere advice to each other for My sake." The scope of mutual advice includes alerting to the performance of commanded acts and the abandonment of prohibited acts.