Is the sister's husband, who resided in the shared family apartment with the mother's consent and insisted on vacating it when one of the sisters wished to come, infringing upon the sisters' right, and should he compensate for his period of residence therein; and is the husband entitled to boycott his brother-in-law and his wife's mother because of this?

A mature woman is permitted to dispose of her wealth within the bounds of Islamic law without her husband's permission. Therefore, your husband has no right to object to your brother-in-law (your sister's husband) residing in the apartment of which you own a part, nor is your brother-in-law obligated to pay compensation for the period of residence. Your husband's action of ceasing interaction with your brother-in-law and refraining from visiting your mother for this reason is unjustified, and if it reaches the point of abandonment (hajr), he is sinful; because abandoning a Muslim for more than three days is impermissible except for a legitimate Sharia (definition) reason, and this is not one of them. You should advise your husband and explain the error of his action.