<commit_before>---
author: jeffatwood
comments: true
date: 2011-01-28 07:59:21+00:00
layout: post
redirect_from: /2011/01/stack-overflow-search-now-81-less-crappy
hero: http://blog.stackoverflow.com/wp-content/uploads/lucene-logo.png
slug: stack-overflow-search-now-81-less-crappy
title: Stack Overflow Search -- Now 81% Less Crappy
wordpress_id: 7041
tags:
- company
- serverfault.com
- stackexchange
- superuser.com
---

Google has ruined search for everyone. 

By that I mean they have done it so long, so fast, and so well -- despite the [recent speed bump](http://googleblog.blogspot.com/2011/01/google-search-and-search-engine-spam.html) -- that **users simply expect _everyone's_ search to be as good as Google's**. And that is ... challenging. Particularly considering Google is an enormous company now, with server farms roughly the size of the state of Pennsylvania.

How's a little startup supposed to compete with that? Or should we even try to, really? From the beginning, Joel and I said that [the _de facto_ Stack Overflow home page was a web search](http://blog.stackoverflow.com/2010/10/search-all-stack-exchange-sites/). So why, exactly, do we need to dump tons of engineering resources into creating a super-uber-mega excellent search facility, again?

That's why we relied on SQL Server to provide our internal full-text search for the last two years, and it's been mostly adequate. We did refine it over time to focus on its strengths -- namely, custom searches with specific metadata attributes that search engines can't see:





  * [Stack Overflow Search – Now 51% Less Crappy](http://blog.stackoverflow.com/2008/10/stack-overflow-search-now-51-less-crappy/)

  * [Stack Overflow Search – Now 61% Less Crappy](http://blog.stackoverflow.com/2009/07/stack-overflow-search-now-61-less-crappy/)


Although our de-crapifying efforts have been noble and heroic (well, in my mind, anyway), we've clearly begun to exceed the scope and scale of what SQL Server search can do for us. 

That's why I'm pleased to announce that as of today, we have switched all of our search functionality network wide from SQL Server to [Lucene](http://lucene.apache.org/java/docs/index.html), or more specifically, the [Lucene.NET](http://lucene.apache.org/lucene.net/) flavor.

[![](http://blog.stackoverflow.com/wp-content/uploads/lucene-logo.png)](http://lucene.apache.org/java/docs/index.html)

(We are, however, a little concerned that [Lucene.NET was dropped by the Apache Incubator](http://search-lucene.com/m/nMTuR1pPJbq). We'd like to see what we can do to help the project stay vital and in sync with the master Lucene project. Let us know how we can best do that!)
<commit_msg>Updated tags<commit_after>---
author: jeffatwood
comments: true
date: 2011-01-28 07:59:21+00:00
layout: post
redirect_from: /2011/01/stack-overflow-search-now-81-less-crappy
hero: 
slug: stack-overflow-search-now-81-less-crappy
title: Stack Overflow Search -- Now 81% Less Crappy
wordpress_id: 7041
tags:
- company
- serverfault.com
- stackexchange
- superuser.com
- community
---

Google has ruined search for everyone. 

By that I mean they have done it so long, so fast, and so well -- despite the [recent speed bump](http://googleblog.blogspot.com/2011/01/google-search-and-search-engine-spam.html) -- that **users simply expect _everyone's_ search to be as good as Google's**. And that is ... challenging. Particularly considering Google is an enormous company now, with server farms roughly the size of the state of Pennsylvania.

How's a little startup supposed to compete with that? Or should we even try to, really? From the beginning, Joel and I said that [the _de facto_ Stack Overflow home page was a web search](http://blog.stackoverflow.com/2010/10/search-all-stack-exchange-sites/). So why, exactly, do we need to dump tons of engineering resources into creating a super-uber-mega excellent search facility, again?

That's why we relied on SQL Server to provide our internal full-text search for the last two years, and it's been mostly adequate. We did refine it over time to focus on its strengths -- namely, custom searches with specific metadata attributes that search engines can't see:





  * [Stack Overflow Search – Now 51% Less Crappy](http://blog.stackoverflow.com/2008/10/stack-overflow-search-now-51-less-crappy/)

  * [Stack Overflow Search – Now 61% Less Crappy](http://blog.stackoverflow.com/2009/07/stack-overflow-search-now-61-less-crappy/)


Although our de-crapifying efforts have been noble and heroic (well, in my mind, anyway), we've clearly begun to exceed the scope and scale of what SQL Server search can do for us. 

That's why I'm pleased to announce that as of today, we have switched all of our search functionality network wide from SQL Server to [Lucene](http://lucene.apache.org/java/docs/index.html), or more specifically, the [Lucene.NET](http://lucene.apache.org/lucene.net/) flavor.

[![](http://blog.stackoverflow.com/wp-content/uploads/lucene-logo.png)](http://lucene.apache.org/java/docs/index.html)

(We are, however, a little concerned that [Lucene.NET was dropped by the Apache Incubator](http://search-lucene.com/m/nMTuR1pPJbq). We'd like to see what we can do to help the project stay vital and in sync with the master Lucene project. Let us know how we can best do that!)

{'fraction_non_alphanumeric': 0.09960533734260478, 'fraction_numerical': 0.02518323623379064, 'mean_word_length': 4.253701875616979, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5107595', 'n_tokens_mistral': 1681, 'n_tokens_neox': 1504, 'n_words': 637}