Is imagining someone attacking oneself considered masturbation? And what should be done about a brother who threatens to kill his family and demands inheritance after the father's death?

What you consider a secret habit is not one. It does not fall within the scope of actions for which one is sinful, as the Prophet, peace and blessings be upon him, said: "Indeed, Allah has forgiven for my nation whatever thoughts or suggestions cross their minds, so long as they do not act upon them or speak of them." As for the inheritance, it transfers to all heirs immediately upon your father's death. You have the right to divide it, and your brother's or mother's refusal holds no weight. It is advisable to report the matter to the competent authorities to protect yourselves from your brother's harm and to place him in a psychiatric hospital for treatment.