How does the penitent rid himself of thoughts of lust and fornication that cling to him despite his adherence to acts of obedience, especially in foreign lands where unveiled women are everywhere?

The sins you have fallen into are due to residing in a land of disbelief. If there is no necessity for your presence there, then leave it. You should marry if you are able, or fast frequently, accompany the righteous, make abundant supplication, and guard your thoughts. Remember death, the terrors of the Day of Judgment, and standing before Allah. Reflect upon the Hellfire and what has been prepared therein for its inhabitants, and reflect upon Paradise and the delights that have been prepared therein for its inhabitants.