<commit_before>---
layout: post
title: What "viable search engine competition" really looks like.
comments: true
permalink: building-search-engines.html
analysis: ■
---

Hacker News is up in arms again today about the RapGenius fiasco. See [RapGenius statement](http://news.rapgenius.com/Rap-genius-founders-rap-genius-is-back-on-google-lyrics) and [HN comments](https://news.ycombinator.com/item?id=7010997). One response article argues that we need [more "viable search engine competition"](http://peebs.org/2014/01/04/we-need-viable-search-engine-competition-now/) and the HN community largely seems to [agree](https://news.ycombinator.com/item?id=7011472).

In this response article, and in much of the discussion, there is a picaresque notion that the "search engine problem" is really just a product problem, and that if we try really hard to think of good features, we can defeat the giant.

I work at Microsoft. Competing with Google is hard work. I'm going to point out some of the lessons I've learned along the way, to help all you spry young entrepreneurs who might want to enter the market.


### Lesson 1: The problem is not only hiring smart people, but hiring *enough* smart people.

The Bing search relevance staff is probably ~20% the size of Google's. And Google's engineers are amazing. It might be possible to make up for a small difference in size, but competing with a brilliant workforce that is *5 times as large as yours* is very hard.

What's worse is that the problem is not even that we need to buy such a team. MS is rich enough that it could do that if it wanted to. The problem is *finding enough people* to build such a team. There are a limited number of search relevance engineers available, and many of them work for Google.

Thus, to be a viable threat to Google, you will need to take this into account and compensate somehow.

<commit_msg>Update numbers in search engines post<commit_after>---
layout: post
title: What "viable search engine competition" really looks like.
comments: true
permalink: building-search-engines.html
analysis: ■
---

Hacker News is up in arms again today about the RapGenius fiasco. See [RapGenius statement](http://news.rapgenius.com/Rap-genius-founders-rap-genius-is-back-on-google-lyrics) and [HN comments](https://news.ycombinator.com/item?id=7010997). One response article argues that we need [more "viable search engine competition"](http://peebs.org/2014/01/04/we-need-viable-search-engine-competition-now/) and the HN community largely seems to [agree](https://news.ycombinator.com/item?id=7011472).

In this response article, and in much of the discussion, there is a picaresque notion that the "search engine problem" is really just a product problem, and that if we try really hard to think of good features, we can defeat the giant.

I work at Microsoft. Competing with Google is hard work. I'm going to point out some of the lessons I've learned along the way, to help all you spry young entrepreneurs who might want to enter the market.


### Lesson 1: The problem is not only hiring smart people, but hiring *enough* smart people.

The Bing search relevance staff is probably ~20% the size of Google's (or something like that &mdash; I forget). And Google's engineers are amazing. It might be possible to make up for a small difference in size, but competing with a brilliant workforce that is *5 times as large as yours* is very hard.

What's worse is that the problem is not even that we need to buy such a team. MS is rich enough that it could do that if it wanted to. The problem is *finding enough people* to build such a team. There are a limited number of search relevance engineers available, and many of them work for Google.

Thus, to be a viable threat to Google, you will need to take this into account and compensate somehow.


{'fraction_non_alphanumeric': 0.06907378335949764, 'fraction_numerical': 0.013605442176870748, 'mean_word_length': 4.346853146853147, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3215500', 'n_tokens_mistral': 1075, 'n_tokens_neox': 997, 'n_words': 564}