Is it permissible for the questioner to demand that his unmarried brother leave the house after he has purchased all the shares from the siblings, taking into consideration the brother's behavior of smoking, not praying, and not fasting?

If your brother agrees to sell his share of the house, there is no harm in you buying it and expelling him [from co-ownership]. If he does not agree, then if the house can be divided, you have the right to demand its division. However, if division is not possible, you have the right to demand that it be sold, and he cannot refuse; because anything that cannot be divided is sold, and its price is divided if one of the partners requests it, and the one who refuses is compelled to sell. And you must enjoin upon him good and forbid him from evil if he is neglecting prayer and fasting.