What is the ruling on someone who tried to expel phlegm, then felt they were about to vomit so they held it back, and they think what they expelled was merely phlegm? And does one break their fast if they vomit unintentionally?

Whoever tried to expel phlegm, then stopped for fear of vomiting, his fast is valid. The same applies if vomit emerged unintentionally due to expelling phlegm, because vomiting only invalidates the fast if it is done intentionally.

It should be noted that a fasting person ought to refrain from intentionally expelling phlegm. If it comes out, he should spit it out and not swallow it. If he swallowed it before it reached his mouth, there is no blame on him. If he swallowed it after it reached his mouth, there is a difference of opinion among scholars, and the preponderant view is that it does not invalidate the fast, due to the lack of clear evidence for invalidation, and the basic principle is the validity of the fast.