What is the Sharia ruling on dealing with a mother who speaks with a non-mahram man in a way that displeases Allah, given that there is proof of this, and is it permissible to frighten her with a hypothetical reckoning to deter her?

If it is established that the mother is speaking to a non-mahram man in a way displeasing to Allah, then she is committing a sin and an evil act. It is part of faith to have protective jealousy (gheerah) over the sacred limits of Allah. There is no blame on the daughter for disliking this, as long as it does not lead to ill-treatment, because one is not held accountable for matters of the heart.

It is recommended to supplicate frequently for the mother's rectification, as this is among the greatest forms of righteousness (birr). Continue to advise her kindly, and explain to her what establishes her situation. If she does not repent, threaten to inform someone who can restrain her. If she does not respond, then tell them the truth without details. Frightening her with an imaginary accountability is not permissible unless it is the only way to restrain her.