Is it obligatory to perform ghusl and prayer when bleeding stops after only a small amount has been discharged, along with brown discharge, and before the completion of the usual menstrual period? And what is the ruling on prayers in the case of only brown discharge without blood?

These brown discharges mixed with blood are considered menses. If the blood stops, along with any yellowish or brownish discharge associated with it, you must perform ghusl (ritual bath), even if that is before the usual period of menstruation ends. This is based on the saying of Ibn Abbas, may Allah be pleased with them both: "It is not permissible for her, if she sees purity for even an hour, not to perform ghusl." You are obligated to perform ghusl and pray. If the bleeding returns during a time when menstruation is possible, you are considered to be menstruating again.