Is it permissible for the lender to keep the borrower's book as compensation for the debt if it cannot be recovered, or to keep it until the debt is paid?

You have the right to demand from your friend the money you lent him, and you must return to him his book that he lent you. You are not allowed to take his book in exchange for your right, nor are you allowed to withhold the book from him until he pays you your right.