What is the ruling on one who says to his wife: "I swear by triple divorce—and he repeated it three times—that you are unlawful to me like my mother and sister" if she performs a certain action?

Your statement to your wife: "May you be unlawful to me like my mother and my sister" is considered an explicit ẓihār. If you then conditioned this ẓihār with a triple divorce upon the occurrence of a specific action, then if this action occurs, a kaffārat al-ẓihār (expiation for ẓihār) becomes binding upon you. This expiation must be performed in its prescribed order and before you resume marital relations with your wife. As for your oath of triple divorce, it has no effect here, because the breaking of the oath is not conceivable.