Was it a sound decision to relinquish child custody and hand them over to their father, in order to please the parents and obey their command, while the mother was aware of the father's ill-treatment, or would it have been better to hold on to custody of the children and protect them? And is the mother sinful for exposing the children to this predicament?

It was not correct to relinquish custody of the children out of obedience to your parents, because obedience to parents is obligatory only in what is right and lawful, and the primary consideration in custody is the best interest of the child. However, there is no sin upon you, as long as you believed it was obligatory to obey them in that matter. If you can revert and demand custody, then do so, bearing in mind that whoever has custody should not prevent the other parent from seeing and visiting the child.