Is the worshipper's feeling that someone is looking at him or that he is praying out of shyness of someone, which he denies to himself, considered hypocrisy that invalidates the act or a satanic whisper?

As long as you reject these thoughts within yourself and push them away, then they are not showing off (riya'), but rather whisperings (waswas) that must be ignored. Instead, turn to Allah alone and be sincere to Him in worship. If a thought of showing off occurs during worship, it is obligatory to repel it and not yield to it or be carried away by it.