What is the ruling on holding a large, mixed engagement party, and is it better to limit it to close relatives to avoid forbidden acts and extravagance?

There is no harm in holding an engagement party, provided it is free from prohibited intermingling, musical instruments, and unlawful singing. If the party that your fiancée's family wishes to hold is free from these forbidden acts, then you should not prevent them from holding it, for they have the right to celebrate their daughter's engagement without extravagance or boasting. This gathering can be utilized for inviting to Allah and delivering a lecture that benefits people in their religion.