Is expiation obligatory upon the mother for preparing food and giving it to her nephew and his non-Muslim wife during the daytime in Ramadan, knowing that they do not fast?

Your mother is mistaken and must repent to Allah for preparing food for her fasting brother and his wife during the daytime in Ramadan. She should also have advised her brother to repent and fast. It is not permissible to feed the wife during the daytime in Ramadan, even if she is not Muslim. Your mother does not owe expiation, but rather repentance and seeking forgiveness.