What is the ruling on money earned from a mixed-gender institute, what are the regulations for dealing in such an environment, and is there sin incurred by the owner of the institute due to the mixing that occurs?

The owner of the institute must prevent unlawful mixing within it, and he sins if he does not do so while being able. As for the money earned from the institute, it is lawful because it is in exchange for teaching and not for mixing.