Is it permissible to build a mosque by taking a portion of a public road, thereby causing harm to its users and to the neighborhood?

The mosque and the road are public facilities, and neither should be expanded at the expense of the other. If four meters are sufficient for the road, then the mosque has more right to the remaining space. However, if narrowing the road will harm the neighbors, then it is not permissible. You can sit with the committee and explain the harm caused to you. If they insist on harming you, you can refer the matter to the Sharia courts.