Is it permissible to lie to the wife of one's husband's brother, claiming to have taken pictures of her during her infidelity to her husband, with the aim of deterring her, and is such a lie considered a sin?

If the situation is as described, then her guardian must prevent her from forbidden relationships and not be lenient with her in this regard, because that would be dayouthah (lacking protective jealousy over female family members). The questioner should advise the woman and instill in her the fear of Allah, and advise her husband to fulfill his right of guardianship. If she does not desist, then she should inform someone who is able to prevent her from this evil. As for threatening with lies, it is impermissible unless it is the only deterrent for her, and it is better to use allusions and circumlocution.