Is it permissible for the two brothers who received their share of the family home and relinquished their right to it, to demand their share of the same house or of another house built by the father thereafter, after the father's death?

If the father gave money to your two brothers during his lifetime, considering it their share of his estate, this is not considered an inheritance or a gift, and they must return this money to the estate. The solution is for the two brothers to take their share of the house, and then return what they took to the estate, or the amount can be deducted from their share of the house if the heirs agree to that.