Is it permissible to claim one's right to the deceased father's house, knowing that the mother and unmarried sister reside in it, and what is the heir's share, and should it be in money or by dividing the house?

There is no objection to the questioner demanding his share of the house, even if some of the heirs reside in it. To accurately determine his share, the heirs must be fully and clearly identified via the designated link, as the house can be sold and its price divided among the heirs if the property itself cannot be partitioned.