Is money earned from modifying a system used to send spam messages and steal from people unlawful (haram)? Does this count as participation in sin? And what should be done with the money if it is unlawful?

As long as the website is originally prepared for permissible purposes, your work in developing it is permissible and your earnings from it are lawful. Your earnings from developing the website are not forbidden if some people use it for impermissible purposes.