What is the ruling on compelling a person to divorce his sister's husband under threat of death, what is the expiation for that, and will the brother be held accountable for this act on the Day of Judgment?

What you did by forcing your sister's husband to divorce her is impermissible. You must repent to Allah and seek forgiveness from your sister's husband before the Day of Resurrection, because this act involves harming a Muslim without a rightful cause, and coercing a husband into divorce does not make it effective if he was forced. Your sister could have sought khul' (divorce at the wife's instance) or divorce due to harm, or referred the matter to the judiciary; these are the legitimate avenues.