Wasn't the Prophet (peace be upon him) more deserving of being followed than David (peace be upon him), and doesn't his not observing the fast of David, despite his ability to do so, indicate that it is better not to observe it?

It was part of his guidance (peace and blessings be upon him) to sometimes abstain from certain virtuous deeds or not to perform them continuously, out of love to lighten the burden on his Ummah and spare them hardship, and fearing that these deeds might become obligatory upon them, or due to his engagement with matters more important than them. Examples include his not continuously observing the fast of David (peace be upon him) and his not performing Umrah in Ramadan, despite his guidance to it and its virtue. So, whoever desires the best of fasting, let him fast one day and break his fast the next.