What is the ruling of Sharia regarding my continued silence and concession of my money to my brother in order to maintain kinship ties and my mother's رضا (contentment), or must I demand it, and am I sinful if this causes problems? And how do I estimate the Zakat for the house, the shop, and the merchandise?

You are not obligated to pay zakat on the house rented to your mother, nor on its rent, nor on the shop. Zakat is only obligatory on your share of the merchandise if it reaches the nisab (threshold). Your brother should be reminded about zakat and its importance. You must pay the zakat on your share of the merchandise yourself or by appointing your brother as your agent. If your brother pays the zakat on your behalf without your authorization, it is safer to consider that it does not suffice for you, due to the absence of your intention. As for demanding your right, there is no sin upon you for it. Your mother's anger does not make her sinful because it is without a valid reason. If you forgo demanding it to preserve relationships and out of dutifulness to your mother, it is a good deed for which you will be rewarded.