Is one who playfully calls his male cousin on his mother's side a "faggot" (لوطي), without knowing the Sharia ruling on that, sinful?

Slandering and the like are absolutely forbidden, even if done in jest, because it is foul language that involves slander or cursing, and a Muslim has a sanctity that must not be violated. For joking to be permissible, it must not contain lies or cause fear to a Muslim; otherwise, it becomes forbidden.

As for not knowing the ruling, ignorance of a Sharia ruling is generally one of the impediments to accountability that lifts the sin.