Must I pay the bank the amount, even though I did not receive this amount and was swindled after handing over my Visa card to a bank representative?

If you were subjected to a scam and did not withdraw the amount, you are not obligated to pay it to the bank. You should explain to the bank the reality of what happened with the representative. If the bank compels you to pay, then you have the right to claim the amount back from the representative.