What is the ruling on not responding to someone who calls out to the the house if circumstances are not suitable for receiving them?

The visitor should not be permitted to enter, especially if the circumstances of the homeowner do not allow for their reception, based on the Almighty's saying: "And if you find no one therein, then do not enter them until permission has been given to you; and if it is said to you, 'Go back,' then go back. It is purer for you. And Allah is Knowing of what you do." For the visitor to go back is purer for them.