What should I do about my mentally ill mother who smokes, whose behavior I can no longer tolerate despite my love for her, and I suffer from intense anger towards her to the point of hitting her?

The mother's right is great, and one must honor her and treat her kindly, especially during her illness and need. Her right does not fall, even if she is a disbeliever. What you are doing, which is an act of disobedience, is a greater sin than her smoking cigarettes. Therefore, repent to Allah and regret your action, and beware lest death overtakes you while you are in this state. And supplicate abundantly for your mother for healing and righteousness.