Is a wife obligated to sit with her sister's dissolute husband who looks at unlawful things (haram) and takes pictures, and is she permitted to limit their interaction to merely greeting him?

Your action of not sitting with this man is correct, as he is not a mahram to you, and conversation with him should be limited to necessity. The Prophet, peace and blessings be upon him, warned against non-mahram men entering upon women, especially the husband's relatives like his brother and uncle who are not mahrams. He described them as "death" due to their danger and their ability to enter and seclude themselves without objection.

The emphasis on staying away from him is further strengthened by the fact that he gazes at women's awrah (private parts), which indicates his فسق (immorality/wickedness) and lack of trustworthiness. There is no obedience to the wife's mother in mingling with this man, because there is no obedience to a creature in disobedience to the Creator.