What is my ruling when they say to me the phrase "the person sticks to you to the point of annoyance" and call it that as well?

The answer has already been given: if the phrase has a meaning that contradicts Islamic law, then the person who utters it should be advised to avoid it. As long as it does not have a meaning that contradicts Islamic law, there is no reason for concern, and you are not required to say anything; simply disregard the whisperings (waswas).