Is it permissible for my deceased sister to claim her father's share (one-sixth) of her brother's inheritance, knowing that her father had renounced his share of the inheritance?

If the grandfather relinquished his share of his father's inheritance during his lifetime and completed that, then this is a permissible gift to you, and his two daughters have no right to claim anything from it. However, if the relinquishment was conditional upon his death, or if he bequeathed his share during his death illness, then this is a bequest to an heir that is not enforceable without the consent of all heirs. If they do not agree to it, then it reverts to the estate and is distributed according to the Islamic legal division.