Is the questioner entitled to reclaim from his deceased brother's heirs the lands he had given to his brother, or to demand compensation from them, based on the fact that one of the plots of land the brother relinquished to the questioner is disputed and was not owned by the brother before his death?

Issues related to disputes and litigations are resolved through the Sharia courts. If it becomes clear that the land you took from your brother in exchange for your land is not fully owned by your brother, or if it is rightfully owed to someone else, then there is no blame upon you for returning it, along with the rest of the lands that came with it, and taking back the lands you gave him, provided that the transaction cannot be parceled out. However, if it is possible to parcel it out, then you should return the aforementioned land and take back the land you gave in exchange for it. This is because it is considered valid to separate a deal and invalidate part of a contract while keeping the rest of it valid. This is the view of the Shafi'is, the Hanbalis, and Ibn al-Qassar from the Malikis. Accordingly, if it is possible to separate the deal without harm and the heirs agree to it, then there is no blame upon you for returning the disputed land and taking its equivalent, while keeping the rest of the contract as is. If the deal cannot be parceled out and the heirs do not agree to it, then you may return the entire deal and take back the lands you gave. The decisive factor in all of this is the ruling of the judiciary.