Does a brief conversation with brothers after Fajr prayer invalidate the reward of Hajj and Umrah associated with the remembrance of Allah until sunrise and praying two rak'ahs?

Sitting in the mosque after Fajr prayer to remember Allah until sunrise is a confirmed Sunnah (definition), and it has been authentically reported that the Prophet (peace and blessings of Allah be upon him) used to do so. The fundamental principle during this time is to be occupied with remembrance (dhikr), and speaking that distracts from remembrance is disliked. However, incidental speech or speech that does not distract from remembrance is permissible.