Is it considered a sin for a father not to see his daughter if he fears going to his ex-wife's house to see her, knowing that he pays her maintenance and is refused the right to see her outside of his ex-wife's house?

If you can see your daughter without harm, then do not refrain from doing so; because that entails severing kinship ties, depriving her of paternal affection, and neglecting her right to upbringing. However, if you fear harm from going to your wife's family home, then demand to see her in a safe place, otherwise, refer your case to the judiciary to regulate that.