Is what happened to the brothers—their abandoning prayer and neglecting good deeds, and the eldest brother's refusal to go to Mecca and abandoning prayer, despite being severely beaten—considered envy, magic, or evil eye? And what is the solution?

The reason for the brothers' distance from acts of obedience does not necessarily have to be one of the things mentioned. Ruqyah (exorcism/spiritual healing) is a good and beneficial thing, by the will of Allah. So, let them perform ruqyah on themselves, or have others perform it on them—those known for righteousness. They should be advised and invited to Allah with wisdom and good admonition, explaining the danger of disobedience and reminding them of Allah, death, and what comes after it. Strive in supplicating for their guidance, for hearts are in the hand of Allah; He turns them as He wills.