Is the husband's action of shunning his brother - who converted to Bahaism and is trying to invite his family to it - correct after many futile discussions?

Whoever converts to Baha'ism from Islam is considered an apostate. The general rule concerning an apostate is to abandon them and avoid mixing with them, unless there is a hope of a benefit from associating with them, such as inviting them back to Islam. If the husband despairs of his brother returning from Baha'ism after numerous attempts, then cutting ties with him is correct. He must also warn his family about this person and explain to them the true nature of Baha'ism.