What is the ruling on an agreement between two people that one of them will pay the other a large sum of money if he commits an act that displeases Allah and His Messenger against the other, and is this money lawful or unlawful?

If two friends make a vow (nadhir al-lajaj) to abandon what displeases Allah, they have the option either to fulfill what they committed to or to pay an expiation for breaking an oath (kaffarat yamin). This type of vow is disliked, because a Muslim is required to abandon what displeases Allah from the outset, without needing to make a vow. If they commit to the vow, then whatever one takes from the other is permissible.