Why didn't Moses (peace be upon him) invite the people who were devoted to idols when he passed by them with his people?

The general rule is that enjoining good and forbidding evil is obligatory. This obligation is waived in cases of inability or fear of a greater harm. The levels of forbidding evil are: by hand, then by tongue, then by heart. The story of Moses (peace be upon him) and the idolaters does not mention Moses forbidding them in the Quran, nor does it negate it. Therefore, it is not permissible to assert definitively whether it happened or not. Moses might have forbidden them, and the Quran did not mention it, or he might not have forbidden them for a reason he deemed valid. The conclusive evidence from the Qur'an and Sunnah is the obligation to change evil by hand for those who are able, then by tongue, then by heart.