Is it permissible to use a program to interact with the brother's wife and send her conversations to a personal email, in order to ascertain the reason for her suspicion, if the suspicion is based on actual actions and not mere conjectures?

A Muslim's defense of himself and clearing his name is a religious imperative, and it is achieved through frankness, as the Prophet, peace and blessings be upon him, did. If you are certain or highly suspect your brother's wife has accused you, confront her and defend yourself. Good conduct and avoiding suspicions preserve one's honor and attract defenders. Since the matter is merely a doubt, and your brother's wife denied the accusation, and given that you may have a psychological complex that magnifies the issue, it is advisable to deal with your brother's wife in a religiously prescribed manner, as she is a stranger. Therefore, it is not permissible to enter upon her unless someone is present to prevent khalwa (private seclusion), and it is not permissible for her to remove her hijab in your presence. Deal with your niece normally to overcome your psychological complexes.