Will the prayer against a mother with cancer be answered if invoked when a certain act is committed, and will it affect her or reflect upon the invoker or their relatives?

The supplication of the one who invoked illness upon his mother is an act of disobedience (ʿuqūq), and we ask Allah to forgive him. This supplication will not be answered, Allah willing. For Allah, in His forbearance and kindness towards His servants, does not answer them if they supplicate against themselves, their families, or their wealth in a state of anger and distress, for He knows that they do not truly intend it. It is mentioned in Qatadah's interpretation that the noble verse: "And if Allah were to hasten for the people the evil as they hasten the good, their term would have been concluded for them" means that if Allah were to answer their supplications for evil and harm as He answers them for good, He would have destroyed them.