Is this ruling correct, or is there an error in not returning the land after the relinquisher's request?

What the questioner is doing is arbitration (tahkeem), which is the appointment by two disputing parties of an arbiter to judge between them. If the appointed arbiter is qualified to judge, then his ruling becomes binding on both parties, just like a judge's ruling, and he is not allowed to retract his initial ruling. This binding nature is limited to the two disputing parties only and does not extend to others. There is no harm in what the questioner ruled regarding obligating the man to bear the effect of the ruling on his own share of the land, and not from the orphans' share.