How can I absolve myself of responsibility for the rights of my deceased grandmother, who had a share in my father's inheritance, especially since the money was spent on building the house and I currently own nothing but this house, given that we have an inheritance with my paternal uncles?

Your grandmother is entitled to one-sixth of your father's inheritance because she died after him. Her share does not lapse with her death; rather, it passes to her heirs. Your claim that she forgave you her share requires a Shari'ah-compliant evidence (one man and two women, or one man and the claimant's oath), or the heirs' corroboration. If her waiver is proven, then her heirs are not entitled to anything. Otherwise, you must give them their share from all the inheritance assets. If the inherited property cannot be divided, it should be sold, and its price distributed among the heirs. It is permissible to reconcile with the heirs regarding what they are owed by you, in exchange for what you are owed from your grandmother's estate. In case of dispute, the matter must be referred to the Shari'ah court or to an authorized judicial body.