Is my quarrel with my nephew considered a sin that angers Allah, and what is the Shariah ruling on his action that affects honor and reputation?

You must shun your nephew who is attempting to commit a forbidden act with you, and not give him any opportunity to do so until he is deterred, because the Sharia has permitted shunning those who commit sins, especially if there is a fear of religious or worldly harm from associating with them. If he is not deterred, then it is obligatory to inform his father of what he is doing to prevent him before a scandal occurs.