Does the presence of inward transgressions (such as ill-thoughts or envy) during worship invalidate it, or does it merely diminish its reward?

Sins do not invalidate deeds, but they negatively affect the worshiper's act of worship. The decrease and increase of reward are in the hands of Allah, and can only be judged by a religious text. The sins that affect a servant's reward are polytheism (shirk (definition)), apostasy (riddah), showing off (riyaa'), and reminding Allah of one's good deeds (mann bil-'ibadah). Polytheism nullifies deeds, as Allah says: "If you associate others with Allah, surely all your deeds will be nullified." Apostasy nullifies deeds in this world and the Hereafter. Showing off invalidates worship, for Allah accepts only deeds done purely for His sake, and showing off is a form of shirk that nullifies deeds. Reminding Allah of one's good deeds invalidates them. One must beware of sins and transgressions because they may nullify good deeds unknowingly, just as some sins may invalidate deeds, such as abandoning the 'Asr prayer and bay' al-'inah (a specific type of sale). Knowing what nullifies deeds is essential to preserve them.