Is the questioner considered sinful for not prompting his dying relative to say the Shahada, does the deceased have a right over him, and how can he atone for his sin?

The verbal instruction (talqin) of a dying person is a sound Sunnah, so that they may conclude with "La ilaha illa Allah" (there is no god but Allah). It is permissible to perform talqin after burial, to remind the deceased of their beliefs, so that they may be firm when questioned by the two angels.

If you did not perform talqin for your relative while they were in a coma, there is no sin upon you. And if they were conscious, then talqin is recommended, not obligatory.

Your brother's performing talqin for his relative before burial is not a known practice, but there is no sin upon him.