What should I do, as I invoked hellfire upon my brother in response to his invoking it upon me, while qualifying my invocation by stating that he had wronged me, and now I am very afraid for myself and my brother?

Neither you nor your brother should supplicate against the other, especially with an imprecation of entering Hellfire. Although your brother's supplication against you is an injustice, the wronged person has the right to seek retribution; however, patience and forgiveness are better. Verses from Surah (definition) Ash-Shura (40-43) emphasize the virtue of forgiveness and reconciliation, and that whoever is patient and forgives, verily that is among the matters requiring strong resolve. You both should now forgive each other, reconcile, and supplicate for each other with goodness, Paradise, and distance from Hellfire; for supplication is beneficial and avails against what has descended and what has not descended, and nothing averts destiny except supplication.