Will I be sinful if I break off my engagement due to my family's objection and my mother's distress, knowing that I intended good and felt at ease?

There is no harm in breaking off the engagement due to the family's objection, and there is no injustice to the girl in that. This is because withdrawing from an engagement is not forbidden; rather, it is disliked if done without a valid reason. However, there is no disliking if there is a valid reason, as in this case.