What is the ruling on praying with a brother who prays only in the presence of the questioner, and is it obligatory to boycott and abandon him?

If you are the Imam and he is following you, then the prayer is permissible. However, if he is the Imam for you, he is a fasiq (transgressor) for abandoning prayer, and jurists have differed regarding the validity of following a fasiq:

Hanafis: The imamate of a fasiq is permissible, though disliked (makruh). Malikis: Prayer behind a fasiq who commits a major sin is valid, though disliked. However, it is not valid if his transgression is related to prayer. The relied-upon opinion is that it is valid behind an innovator (mubtadi'). Shafi'is: Prayer behind a fasiq Imam is permissible, though disliked. Hanbalis: The imamate of a fasiq is absolutely invalid, and whoever prays behind him must repeat the prayer.

It is preferable and safer to avoid following him. As for shunning him, you must first advise him. If he insists on abandoning prayer and you believe that shunning him might encourage him to maintain it, then shun him.