What is the ruling regarding someone who pledged to God to serve her husband's mother without grumbling, but then was unable to fulfill that due to her psychological state, knowing that her grumbling was not apparent to the husband's mother? Is expiation obligatory upon her?

If someone makes a covenant with Allah concerning something, it is obligatory for them to fulfill it if it is an act of devotion. If they do not fulfill it, repentance is due along with an expiation for a broken oath (kaffarat yamin). What has occurred from you is a violation of the covenant you made with Allah, which was not to show annoyance towards your husband's mother. This is a clear violation of the covenant, even if it was not in her presence or if she did not hear it.