Do I incur sin if I leave off waking my family for Fajr prayer due to the hardship and annoyance involved?

You must awaken them and inform them of the entry of the prayer time. If they respond, then that is good and excellent. But if they do not respond, then you are not responsible for them. And if your awakening them causes harm, then you are not obligated to do so.