What is the ruling of Islamic law on agreeing to pay a financial fine of 5 dirhams for every person who fails to attend Fajr prayer in the mosque, and then collecting these funds to purchase books or tapes?

There is no blame upon you for what you have done in using this method to pressure yourselves into performing Fajr prayer in congregation, and it contains benefit for others and the spread of good. There is no impediment to that, provided that it does not weaken resolve and cause one of you to become lax about the five dirhams and stay away from prayer out of laziness.