Can a person's prayers for evil or failure in life befall someone whose conversation was refused because it is forbidden, despite her trying not to harm him with words?

What you did by refusing the young man's call is absolutely correct, because it is impermissible for a woman to converse with a non-mahram man except out of necessity, and on condition that Islamic guidelines are observed. This young man has no right to curse you, for his prayer for a sinful act will not be answered. Our advice to you is not to concern yourself with him, but rather to focus on what benefits you, and to seek the company of righteous girls.