Am I entitled to cancel a book purchase order after reaching an agreement with the publishing house, and do I owe the house anything after their insistence on completing the purchase, especially since they mentioned that they bought the books at the house's expense?

If you appointed the publishing house to buy books for you, and the purchase was made before you revoked the agency, then the contract is binding and cannot be rescinded. However, if what happened was merely a promise to buy when the books become available, then it is not a binding sale, and there is no blame upon you for retracting the purchase for the reason mentioned. Fulfilling a promise is recommended by the majority of scholars, and obligatory according to some scholars if it results in harm to the other party. It would have been more appropriate to complete the contract.