Is it forbidden to return a book that has been marked after cutting out the designated part from it, then exchanging it for another, only to discover that the replacement copy is not good, and then acting on the basis of taking the first book back again?

You have erred and done wrong by tearing a page from the book and returning it to the library without informing its owner and getting another copy. This is deception and harms him or the buyer. You are obligated to repent by informing him of what you did, then asking for his forgiveness and pardon, or compensating him for the damage. The increase in the book's price at the library has no bearing.