Is it permissible in Islamic law for an owner to be demanded to pay 40% of the market value of the property to a tenant in exchange for vacating his apartment, and is this considered usurped money that one is permitted to defend oneself against, even if it violates the law?

If the lease term has expired and the tenant refuses to vacate the property, then he is an aggressor and an oppressor. He must vacate without demanding any money. The owner is permitted to defend his property and reclaim it by force, based on the hadith: "Whoever is killed defending his property is a martyr." However, one must be cautious not to fall afoul of the law. It is better to resort to the mediation of people of virtue to convince the tenant to leave, even if for a reasonable compensation.