How to politely reject a request for a letter of recommendation for a professor I am a student. 

I received a request for a letter of recommendation to a professor for some award. I know for a fact that my letter will be directly used in the application for the award. [I have since graduated from this institution.] I haven't interacted with this particular person in about 3-4 years, and feel that I couldn't write the strongest one, since my interactions with this person were not much outside of classwork (two classes). 

It looks like someone else is requesting these letters; how do I politely reject the request? I am particularly not interested in burning any bridges, since I may consider getting a career in academia in the future, but I haven't asked this particular professor for any recommendations (nor do I plan to do so in the future).

Update: I have decided to write the letter. Appreciate all of the feedback here.
If you want to decline to write, then it would be reasonable to reply to the person who requested the letter, saying something like: "My interactions with Professor X were three to four years ago and involved very little interaction with him beyond attendance at two of his classes.  So I do not consider myself to be in a position to provide useful information for award Y."