Reapply to a postdoc position that I rejected before? In my field (math), usually one needs to do more than one post doc. 
I'm in the middle of one right now and to get to this position I had to reject another offer.
I'll be at the position of applying for jobs again (most likely another post doc) and I'd like to apply to the one that I was given an offer before. However, the last time around, I had to ask the potential host for a letter and I feel a bit bad for not going there. Of course, this time round, I think there may be possibilities that they don't want me at all.
Are there some cautionary tales/wisdom that can be shared regarding situations like this? Am I over thinking?
What do you have to lose? Worst case scenario is that you are rejected. I don't see any possible downside here.

Anecdote: This was not for a postdoc position, but for a research internship: I contacted the researcher who I wanted to work for not just once, but three years in a row. All three times they expressed interest. The first time, it was too late so they said to apply next year. The second time, they were going to give an offer, which I turned down. The third time, I felt mildly awkward, but I knew that this was a position I still really wanted -- so I simply explained as much, and thanked them a lot for their interest in previous years. It worked and I got the position again.