How can one differentiate between doubt that is based on the lesser of two possibilities, such as doubt about the number of rak'ahs, and doubt that is based on what first comes to mind, and what is the evidence for the second type?

A non-obsessive person, if they doubt their prayer, builds upon certainty (the lesser) and completes what they doubted, then performs the two prostrations of forgetfulness (sujud al-sahw) after the salam. This is the view of the majority of scholars, including the Malikis.

As for the obsessive person (al-mustankah), who experiences doubt frequently, the relied-upon position in the Maliki madhhab is that they do not act on their initial thought. Instead, they build upon the greater (i.e., they disregard the doubt and assume the prayer is complete), they ignore the doubt, and they perform the prostrations of forgetfulness after the salam as a recommended act to vex Satan. The criterion for obsessive doubt is its occurrence every day, even if only once, whether its nature is consistent or varies.