Does this merit also include visiting a non-believer?

The hadith does not discuss visiting a disbeliever (kafir) because it explicitly mentions visiting a Muslim. Therefore, scholars have distinguished between the ruling on visiting a Muslim and visiting a disbeliever.

Visiting a disbeliever is permissible according to some scholars, and permissible according to others if it is accompanied by a type of sanctity, such as neighborhood or kinship. In fact, it is recommended to visit him if his conversion to Islam is hoped for. One may even be rewarded for visiting a sick disbeliever for the purpose of winning over his heart to Islam.