Am I liable regarding the money I gave to my friend for his younger brother, having sworn not to take it back, and then my friend forgot it on the table in their gathering place?

The money you gave to your friend's brother is a gift (hibah), and a gift is established by what people customarily recognize as offer and acceptance, or by mutual exchange (al-mu'atah) accompanied by what indicates it. Your friend's silence when you placed the money on the table is evidence of his acceptance of the gift for his brother. Therefore, your oath is valid, and you have not broken it.