Is the worshiper's thinking about people seeing his prayer and praising him for it considered showing off, despite his dislike of showing off and his lack of excellence in prayer?

Satan is keen on corrupting a person's worship and faith, so one must strive against him and seek refuge in Allah from him. Since you dislike showing off (riya') and its thoughts during prayer, you are upon goodness. You must improve your prayer and strive to achieve sincerity.