What should I do about the remaining amount owed to the smuggler, and is it considered a debt like the unforgivable debt for a martyr?

If the work performed by the man was prohibited (haram), then the remaining amount must be given out in charity. This is because the recompense has been received, even if it was for a prohibited act. In a prohibited contract, one cannot combine the recompense with the object for which it was given. If the transgressor has received the recompense, then the money should be spent on the welfare of the Muslims. This is part of perfecting repentance and returning to Allah beautifully.