Is looking at women considered an injustice to their guardians (father and brother), or is it a sin only between the individual and their Lord?

There is no authentic evidence or statement from scholars that establishes a right for a woman's mahrams (such as her father and brother) to hold a grievance against someone who looked at their female relative with a forbidden gaze, necessitating their forgiveness as a condition for repentance. The most that has been mentioned is what Imam Al-Ghazali said, and Al-Haytami followed him in "Az-Zawajir," that in adultery (zina) and sodomy (liwat), there is a right for the human being, so repentance depends on seeking forgiveness from the relatives of the woman involved in zina or the person involved in liwat, and the husband of the woman involved in zina, if no fitna (strife/temptation) is feared. Otherwise, one should humble themselves before Allah in seeking to appease them, because such acts bring shame and defile the husband's bed.