Recent publication versus old, crappy letters of recommendation? I honestly didn't do to well in university. The classes that interested me I did well in, everything else not so much. Needless to say my GPA isn't the best.

Recently however I've found a field that really interests me. I've been trying to learn as much as I can, and I've been contributing to open source projects as well. These projects have led to at least one publication (a second is up in the air) as well as an online portfolio.

Since these are open source, remote projects I haven't had a real chance to form close relationships with anyone. Since I'm now considering going back to school, this puts me in a bit of a bind when it comes to letters of recommendation since I doubt in going to get really great letters from anyone from university.

It's been about five years since I graduated, so I'm not sure the impact of poor letters versus more recent work. I'm also not sure how to fix it. Any advice would be nice
If you have done valuable work on open source projects, a recommendation from one of the lead developers would certainly count highly in my book, if you are applying to a related field (or software related in general). That you have never even met face to face should make no difference.

You say you did well at some classes you liked, presumably some of the people who taught you (preferably later courses) will write you "face to face based" letters, I wouldn't forego such totally.

For the record, I had some students around here (OK, still have) that do extremely well in non-directly study related stuff, and do so-so or directly terrible in their regular classes. I would recommend some of them highly, even if their grades aren't stellar. But probably with some warning about "careful, if work isn't interesting to them they might just drop it".