Does slight straying and deviation from the spot of the worshipper standing behind me during prayer constitute the قطع (breaking) of his prayer?

It would have been better for you to remain seated until your brother finished his prayer, if waiting did not cause you any difficulty. This is because it appears he intended to pray behind you so that you could serve as a sutrah (barrier) for him. Using a person's back as a sutrah is permissible, as Ibn Qudamah said: "If one takes a person as a sutrah, there is no harm in it, for he takes the place of other types of sutrah." As for your departure from your place, there is no sin upon you for that, and it does not fall under the prohibited passing in front of a worshiper, because you passed from behind his place of prostration. This is what Sheikh Al-Uthaymeen and Sheikh Ibn Baz preferred.