What should I do with the money for books I took, thinking they were free, then learned I had to pay for them, and I was unable to contact the library to pay or return the books? And is it sufficient to give the money in charity on their behalf?

As long as you are able to reach the library, your responsibility will not be absolved except by returning the money to it. It is not permissible to give it as charity when you are able to deliver it. You can appoint someone from among the residents of the area or anyone who goes there to pay the money on your behalf.