Applying to work where I said "no" in the past? I'm currently applying for lecturing jobs at a couple of universities (in the US), and one of these is a place that had accepted me in its Ph.D. program a few years back, an offer which I ended up turning down to go somewhere else (this was done in a very courteous way and I do not believe I left a bad impression).


Do you think my history with the school could have an adverse effect on my current application? 
If not, is there any reason to  mention somehow that I applied to their graduate program (I'm worried that if I avoid the "elephant in the room" they might become suspicious)?
There are many reasons why people choose to go to one program over the other. While there might be some thin-skinned people, I don't think that should lead you to rule out applying. Since applying to a job at a school is minimal cost to you, you should go ahead and do it.

Much more likely is that the only person who remembers is the person who would have been your POI.  I wouldn't bring it up unless someone asks.