Is my brother considered an infidel if I doubt his sometimes missing the Asr prayer, even though he prays the other prayers in the mosque and swears that he performs them, but he sometimes lies?

The basic principle is to think well of a Muslim and to believe what he says without an oath. So if your brother claims that he prayed, believe him and do not think ill of him. If you become certain of his negligence in some prayers, then forbid him from the evil and explain to him the danger of abandoning prayer, and advise him.