Must I pay zakat on the money I deposited with my brother, knowing that I withdrew a portion of it after the zakat due date and before paying it? And if zakat is due, should I include the amount I withdrew in the zakat calculation?

If the wealth reaches the nisab (minimum threshold) and a hawl (full lunar year) passes over it, zakat becomes immediately due, and one's obligation is not discharged except by paying it, even if the wealth has been spent. You are excused for the delay due to forgetfulness, but you must hasten to pay thekat immediately upon remembering it for all the wealth that has had a hawl pass over it, whether it has been spent or remains.