Is the questioner considered to be in error for his decision to boycott two people who habitually backbite, slander, and mock the rituals of Islam, after he advised them repeatedly to no avail?

What you mentioned regarding the situation of your two friends—backbiting, slandering women, mocking your beard and religious people, their aversion to advice, and how this affected your religious commitment—all of this necessitates that you distance yourself from them and cut off interaction with them. Therefore, your decision to boycott them and maintain only a greeting of peace between you is correct. Although the general rule is the prohibition of shunning a Muslim for more than three days, scholars have permitted shunning beyond three days for anyone whose association brings harm to one's religion, self, or worldly life. Scholars have affirmed that among the reasons for permissible religious shunning beyond three days is shunning those who openly commit sins. The Prophet, peace and blessings be upon him, shunned those who lagged behind from the Battle of Tabuk. What your father tells you about appeasing people should not be at the expense of condemning evil. You should also remind your father that this does not contradict the obligation for you to distance yourself from what brings you harm and to boycott bad company.