Is it permissible for a Muslim to ask God to take his soul during Hajj?

What you did involved supplicating for death upon yourself, which is forbidden. The Prophet, peace and blessings be upon him, said: "None of you should wish for death. If he is a doer of good, he may increase his good deeds; and if he is a doer of evil, he may repent."

As for your statement: "If my matter concerns You," we fear that it may contain a lack of proper reverence towards Allah Almighty. Even if you intended by it, "if You are pleased with me and care about my matter, unlike the people of evil whom You do not care about," it should not have been used. And there are servants whom Allah does not regard due to their insignificance in His sight.