Is delaying sleep and staying up until just before Fajr, which resulted in missing the prayer and leaving one's family asleep, considered committing a major sin or a transgression, knowing that the staying up was for reading a beneficial book, and measures were taken such as setting an alarm and reciting the remembrances before sleep?

There is no sin upon you for what occurred. There is no sin upon a sleeping person who was asleep before the prayer time entered, even if they wake up after its time has exited. This is especially true since you took measures to wake up by setting an alarm. There is no harm in staying up late to read books about the righteous predecessors; rather, you will be rewarded for it.