Is killing a vicious guard dog that has changed its behavior and begun to pose a threat to my family and others, after consulting experts and attempting to treat it, a sin, or is it a matter of self-defense and protecting others?

There is no sin in killing a harmful dog, for the Shari'a has granted permission to kill the rabid dog. Some scholars have explicitly stated that killing it is obligatory to avert its harm. The Prophet, peace and blessings be upon him, said: "Five animals are mischievous and may be killed in the lawful and the sacred territory: the snake, the speckled crow, the mouse, the rabid dog, and the kite." A dog is either jet-black, in which case it is recommended to kill it, or rabid, in which case it is obligatory to kill it. It is permissible to keep a dog for guarding, while being careful not to let it enter the house.