Is a mother sinful for serving and caring for her sick ex-husband's father, despite her family's prohibition of it?

The basic principle is that it is not permissible for a woman to serve a non-mahram (definition) man or to see his awrah (private parts) unless there is a necessity. If there were men available to serve your father, then what your mother did by serving him was a transgression, and she must repent and seek forgiveness. However, if there were no men available to serve him, then she is considered righteous and will be rewarded, Allah willing. It is permissible for a woman to look at a man's awrah and serve him out of necessity, such as when there is no mahram available during illness.