Is the husband entitled to custody of his two children, who are 3 years old, after discovering his wife's communication with another man, and can he compel her to sign away her custody rights in exchange for a divorce?

You must repent of the sin by abandoning it, regretting it, and resolving not to return to it, while concealing it and not publicizing it, and beware of being lenient with prayer. You must also adhere to the rulings of Sharia (definition) in dealing with foreign men. As for the custody of children in the event of divorce, the default is that you have a greater right to it unless there is an impediment, and the Sharia court is the one that decides on this matter, and your husband does not have the right to force you to waive custody in exchange for divorce.