What is the ruling on masturbating during the day in Ramadan while trying to quit but being unable to do so, and how can one repent from it?

You must repent from masturbation. If you broke your oath to abandon it, an expiation for breaking an oath is incumbent upon you. No matter how many times the sin is repeated, hasten to repent and do not despair. Your feeling of fear of Allah's displeasure is a good motivator to abandon the sin.