Was the questioner's action of not fasting and praying due to the brown fluid mixed with blood correct? And does the discharge of another brown fluid after ghusl invalidate prayer and fasting?

If a woman sees blood or what is considered as such (yellowish or brownish discharge) fifteen days or more after the end of her menstruation, then it is menstruation. Therefore, the brown discharge mixed with some blood is considered menstruation and necessitates abstaining from prayer and fasting.

As for brownish discharge after purification and ghusl: If there are less than fifteen days (the minimum period of purity) between it and the previous menstrual cycle, then it is not considered menstruation and does not prevent prayer or fasting. In this case, only istinja' (washing private parts) and wudu' (ablution) are required. Fasting is not affected by the presence of impurity because purity is not a condition for its validity.