Must I return the money to my brother, to whom I vowed to return it, but he refused to take it and considered it a gift to my children?

This vow is considered a permissible (mubah) vow, and thus it is not binding. Therefore, it is not obligatory to fulfill it, nor is an expiation for breaking an oath (kaffarat yameen) required for it, according to the majority of scholars. Consequently, there is nothing upon you.