How do we distinguish between male followers who are "ghayr uli al-irbah" (those without sexual desire) and other men?

"Followers who are not possessed of desire (ghayr uli al-irbah)" are those who have no inclination towards women, either due to a mental deficiency or a physical inability to feel lust, such as the simple-minded, the imbecile, the impotent, or the effeminate man whose penis does not become erect. There is no harm in these individuals entering upon women.