text:
By Mona Charen
When it comes to foreign policy, I sleep better at night knowing that Republicans control the White House and the Congress. But on health care, it has lately become difficult to imagine how things could be much worse with the Democrats in charge.
When the Clinton administration attempted to move us down the road toward socialized medicine, Republicans responded that the answer to health-care problems was more choice and more market incentives, not state control. Democrats have never learned this lesson, and it appears that Republicans have now forgotten it.
Take the prescription drug benefit bills that recently passed the House and Senate. Reasonable people might be moved to pass a bill granting special drug benefits to the elderly poor or to those experiencing particularly crushing medical expenses. But that's not the way the politicians like to do things. Instead, all seniors — rich and poor — will be eligible for this subsidy.
The cost? A recent estimate put the price to taxpayers at $400 billion per year. But don't forget, when Medicare was introduced in 1965, it was estimated to cost $8.8 billion by 1990. The true cost was about $66 billion.
Well, the elderly are the poorest segment of society and thus require more subsidies from the rest of us, right? No. The Census Bureau reports that poverty among the elderly was 10.2 percent in 2000, whereas the child poverty rate hovered around 16.2 percent. Besides, some 76 percent of the elderly already have prescription drug coverage through private plans, Medicaid or pensions. Of the 24 percent who lack coverage, some are not sick. But they'll surely get Viagra and more if they're subsidized.
In addition, the presence of a government entitlement will cause those employers who do provide a drug benefit to their senior employees to drop it, thus increasing the pool of those dependent on the state. Why not target assistance only to those who need it?
As Deroy Murdoch suggests in National Review magazine, a conservative approach to the drug benefit issue would permit those who like Medicare to keep it. The elderly poor would receive subsidies to purchase private health coverage that included prescription drugs. As incomes rose, the subsidies would fall. Instead, Congress proposes to tax all of us, including strapped families with young children, to pay for Lipitor for Ted Turner — now and forever, entitlements without end, amen.
Another gem the Republicans are not opposing — in fact, many are supporting — is the drug reimportation proposal. This is the brainchild of some congressmen who looked at lower drug prices in Canada and elsewhere, and declared, "Hey, no fair." If this bill becomes law, Americans will be able to import American-made drugs from Canada and other nations, and thus enjoy lower prices.
But there's a worm in this beautiful apple. Canada and Europe enjoy low drug prices because their governments put price caps on drugs.
Those consumers are, as the American Enterprise Institute's John Calfee explains in The Weekly Standard, "free riders" on our system. The United States is the only market in which drug manufacturers make a profit. That's why European pharmaceutical firms are flocking to America. In 2000, U.S.-based firms spent 24 billion euros on research and development, compared with 17 billion euros for European firms. (Europe has a larger population and greater total wealth than the United States.) Fifteen of the top 20 best-selling drugs worldwide are made in the United States.
It costs money, lots of it, to create new medicines. Of every 10,000 substances considered as new drugs, only 250 make it to animal testing. Only five are tested on humans. And only one gets FDA approval.
People gripe about the cost of drugs, but the medical costs they save are hardly taken into account. A family member had a stomach ulcer a number of years ago. Before the drug Prilosec was available, people lost countless days of work, had costly and painful surgeries, and even died from stomach ulcers. Now, they take a pill for a few weeks and that's it. Cured. That's one example among millions. It is estimated that half of the extension in life expectancy experienced between 1971 and 1991 was due to new drugs.
Reimportation would result in one of two unintended outcomes: Either prices would not drop because drug companies would decline to ship huge quantities of drugs to places like Canada, or worse, prices would drop and companies would stop research and development.
Time was, Republicans understood these things.
Mona Charen writes for Creators Syndicate.


text:
Standard Colors is a leading supplier of color chemistry for a broad spectrum of industries including:
With our on-site standardization lab, we can precisely control the strength, shade and physical properties of our colors to guarantee you get consistent, repeatable results every time. Our global supply chain ensures we use the highest quality raw materials and intermediates for all our products.
We conduct professional laboratory analysis and offer expert technical advice for selecting colors, auxiliaries and application procedures to meet your specific needs. We offer fast, accurate custom shade matching using the latest color management systems.
We pay close attention to our customers. We listen carefully when you discuss the demands and challenges you face in business. We form close bonds so we can respond to your needs with timely and time-proven solutions that are tailor-made for your requirements.
For more than 35 years, Standard Colors and its forerunner companies have offered customized color chemistry, high-quality products and expert technical support. Our business has been successful only because we help our customers succeed in theirs.

subset: commoncrawl


text:
UPDATE: TextFileHarvester for SQL 2008 can now be found here.
I think the first thing to answer is, What is Solr? I'll let Solr explain itself:
Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.
Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Tomcat. Solr uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming language. Solr's powerful external configuration allows it to be tailored to almost any type of application without Java coding, and it has an extensive plugin architecture when more advanced customization is required.
Simply put, Lucene is an open-source Java library that does full text searching. In other words, if you want to provide searching beyond the simple "sql like query" you'll need full text search. Now Lucene does just that, but firstly, it's written in Java (there is a .Net version called Lucene.net but frankly they're way behind the original Java project, and I don't know how active that community is.) Secondly, it's just a library, so you'd need to create your own search engine around it. That's where Solr comes in. Solr is a Java web based application that functions as a Search engine (using Lucene under the covers). The nice thing about it is that it has a simple Web Service API that's XML based which means, it can be used from C# (or any other language for that matter).
The point of this tutorial is to provide a simple walkthrough to getting Solr up and running to index and search some documents using C#. I work in Computer Forensics / E-Discovery where being able to search and filter documents is vital. I learned alot on the job, and felt that there wasn't really one good end to end tutorial aimed at C# developers, so I hope someone will find at least some of this information useful.
Here's how this will work. First, I'll walk you through getting Tomcat up and running. Then, I'll demonstrate how to install Solr. Next, I'll have you download a little utility I wrote which basically creates a database, and downloads a whole bunch of text files off the net. Finally, we'll write some C# code to index and search those text files.
Part 1: Installing Tomcat:
From their website:
Apache Tomcat is an open source software implementation of the Java Servlet and JavaServer Pages technologies. The Java Servlet and JavaServer Pages specifications are developed under the Java Community Process.
Since Solr is a Java web app, we need a server to host the Java Servlet (that's basically a Java web app). First, you'll need to make sure Java is installed on your machine. You can go to this site which will tell you whether or not it's installed: http://www.java.com/en/download/help/testvm.xml If you see this:
then you have Java installed. Otherwise, head on over to this link, and grab the latest version of the JRE (Java Runtime Environment.)
The next step is to install Tomcat. You can grab it here: http://apache.cyberuse.com/tomcat/tomcat-6/v6.0.24/bin/apache-tomcat-6.0.24.exe That's the windows Installer version. Run the install and click Next until you get to this page:
You can put any port number into the Port text box (the default is 8080). I put 8983 as that's the default you'll see in all Solr examples on the net. You can choose any port you'd like it doesn't really matter. Just remember which port you choose as that'll always be part of the URL when connecting to Solr. In all my examples here, I'll be using 8983.
On the next screen, it'll try to find the location of the JRE on your system:
It should find it automatically, but if it doesn't make sure to point it to directory where Java was installed on your system. Click next and Tomcat will install.
Once you finish the installation, tomcat will start and you'll see this:
(If you don't see that, either look at your Windows Taskbar in the bottom right for the Tomcat icon, or go to Start-> Programs -> Appache Tomcat 6.0 -> Configure Tomcat)
Click on the Start button and watch Tomcat startup. Above the Start button you should now see "Serive Status: Started. If it does not, here's what you need to do. Go to Program Files\Java\jre6\bin and locate the file called msvcr71.dll. Copy that file and paste it into C:\Windows\system. Now try starting Tomcat up again, and now it should work. Don't ask me why, but after breaking my head a while back on one of my test machines, I found the answer on Google.
Once Tomcat is started, open a web browser and navigate to: http://localhost:8983 (remember, if you used a different port during installation, use that one instead). You should be greeted with this beautiful page:
If you don't see that page, please refer to Tomcat's FAQ.
Step 2: Installing Solr
The next step in this process is to install the Solr web application. First, let's shut down Tomcat. (Click Start-> Programs -> Appache Tomcat 6.0 -> Configure Tomcat and click on the Stop button.) Next, we'll need to download Solr. As of the time of this blog post, the latest version of Solr is 1.4 so let's download and install that one. Head on over to this page and choose the mirror of your choice: http://www.apache.org/dyn/closer.cgi/lucene/solr/1.4.0 Once you click on one of the mirror links, you'll be taken to a page where you can choose the different formats to download. Download this one:
Once that's downloaded, unzip the file and locate the folder "$\apache-solr-1.4.0\dist". In there you'll see a file "apache-solr-1.4.0.war"; war files are basically zipped up files that contain all of the necessary binaries for the servlet container (Tomcat) to run the webapp. Copy that file and past it to: "$\Program Files\Apache Software Foundation\Tomcat 6.0\webapps" Rename the file to solr.war because that will be the name of the URL to access Solr.
Next, we need to create the "Solr Home". The Solr home is basically a folder where Tomcat will look for all the relevant Solr configurations, as well as where the actual index files will be stored. Go back to the unzipped solr package, and locate the folder: "$\apache-solr-1.4.0\example\solr". Copy the contents of that directory (should be a bin folder, a conf folder and a readme.txt) and place it under your root directory, under a folder named Solr (e.g. C:\Solr). Note: You can place it anywhere on your hard drive, doesn't have to be under the root, but wherever you place it, make sure to remember the full path. For now, don't worry about what exactly you're doing, I'll explain more later.
Next next step is to tell Tomcat where the Solr Home folder is. Open Tomcat's configuration window again, and go to the Java tab. Locate the Java Options text box, and enter this line at the end of all the other stuff that's already there:
(If you chose a different Home Folder, place that path instead.)
Now go back to the General tab, and click Start. Once Tomcat starts up, open your browser and navigate to:
You should be greeted with this page:
Click on the Solr Admin link and you'll see this page:
In the Query String box, replace the word solr with this: *:* (this is the lucene syntax for "select all") and click Search. You should now see a page of results in XML format. At this point we haven't indexed anything, so you won't see any results, but Congratulations, you've set up Solr!
Step 2.1: Configuring Solr
In order to move on to Solr Configurations, it's important to take a minute and explore how Lucene works. I'm no expert, and if you want a deeper understanding, use Google, but I will mention a few key points.
The first step to getting documents search-able is to index them. I'm sure you've heard that term before, but what exactly does that mean? At the heart of indexing are two concepts known as Tokenizing, and Filtering. The idea is, you use these tokenizers (that have custom analyzers) and filters to analyze the data and store them in a custom file structure known as an index. I think this page does a great job of explaining it:
The analyzer's job is to take apart a string of text and give you back a stream of tokens. The tokens are presumably usually words from the text content of the string, and that's what gets stored (along with the location and other details) in the index.
Each analyzer includes one or more tokenizers and may include filters. The tokenizers take care of the actual rules for where to break the text up into words (typically whitespace). The filters do any post-tokenizing work on the tokens (typically dropping out punctuation and commonly occurring words like "the", "an", "a", etc).
So you feed Lucene the data, it analyzes it and tokenizes it, at which point it is now search-able. Then, when you submit a query, it uses the same tokenizers and filters to parse the query, and return the results. Why is this relevant? Well, because different data will be tokenized differently. For most text you'll use a WhitespaceTokenizer which splits on white spaces, and tokenizes the words and generally ignoring special characters. For integers or dates for example, you obviously don't want that kind of tokenizer. You want to be able to specify that this is an int field, and be sorted as an int for example.
That's where the Solr config file comes in. You specify all the fields that you want to index, as well as which tokenizers and filters to use for those fields. If you've been following along with this tutorial, you'll find the config file(s) under $/Solr/Conf. There will be many files there, but we won't concern ourselves with all of them. For now we'll focus on schema.xml. The other one I'd like to mention is the solrconfig.xml file which basically are settings for solr itself (how often to commit the index, how much ram to use while indexing etc...). The great thing about the example files is that there's very very detailed documentation in the XML files.
For this tutorial, I'll be using text files that can be downloaded for free from http://www.textfiles.com. There you'll find thousands of text files from the early days of the net. I'll provide a link further down to a simple application that I wrote that scrapes the site, downloads as many files as you specify, and creates a database for you with the relevant information. For now though, I'd like to show you the config file needed to index these documents. We'll be indexing these fields:
- fileid
- doctext
- title
- datecreated
Here's the schema.xml file needed:
<?xml version="1.0" encoding="UTF-8" ?>
<schema name="example" version="1.2">
<fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true" />
<fieldType name="int" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0" />
<fieldType name="date" class="solr.TrieDateField" omitNorms="true" precisionStep="0" positionIncrementGap="0" />
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory" />
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1" />
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt" />
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true" />
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1" />
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt" />
<field name="fileid" type="int" indexed="true" stored="true" required="true" />
<field name="doctext" type="text" indexed="true" stored="false" required="false" />
<field name="title" type="text" indexed="true" stored="false" required="false" />
<field name="datecreated" type="date" indexed="true" stored="false" />
<solrQueryParser defaultOperator="OR" />
As you can see, the XML is pretty straight forward. I'd like to touch on two of the attributes in the field tag:
The first one is "stored". A lot of people make this mistake the first time they mess with Lucene, and they stored everything in the index. I'd like to stress this point really really strongly. An index is NOT a data storeage mechanism. It's not intended for that, it's not optimized for that. That's what Databases are for. What I mean by that is, the text you send to get indexed, gets tokenized and totally bastardized. There's no way to re-construct the original document. So what Lucene allows you to do, is store the text as is in the index as well for later retrieval. That's OK for small index's or small fields. However when you get to larger indexes or larger fields, your performance will suffer noticeably. What's most commonly done, is you index everything you need, however you only STORE the primary key (e.g. fileid). When you retrieve the items from the index during search, you get the primary key field, and then you query your database which IS meant for data storage, and get the rest of the fields you need. That's the approach I'll be taking here.
The other field I'd like to point out, is the "type" field. This tells Lucene, which kind of field this one will be, and based on the fieldtypes specified earlier, it uses the correct tokenizer.
There's also one change I think is worth making in the solconfig.xml. Not too far down in the solrconfig.xml file you'll see this:
<!-- Used to specify an alternate directory to hold all index data
other than the default ./data under the Solr home.
If replication is in use, this should match the replication configuration.-->
For some reason the example defaults to putting the index file in the same directory as the Tomcat server. Just comment out that line:
<!-- Used to specify an alternate directory to hold all index data
other than the default ./data under the Solr home.
If replication is in use, this should match the replication configuration.
we'll use the default "Solr Home" folder ($\solr\data\index) to store the index files.
At this point, in your /solr/conf folder, you should have these files: (delete all the other ones)
- schema.xml (copy and paste the xml from above)
- elevate.xml
- protwords.txt
- solrconfig.xml
- stopwords.txt
- synonyms.txt
Part 3: Downloading Some Text Files
OK, now we have most of our environment set up, it's time to get some files! I created a simple application, that connects to http://www.textfiles.com and downloads some files. It then creates a database and adds a record for each file downloaded. You can download it here:
For SQL 2005: http://www.box.net/shared/du4zac9vaa
For SQL 2008: http://www.box.net/shared/vtbod3rub6ibvof91xut
Here's how you use it:
TextFileHarvester.exe "ALEX\SQLEXPRESS" "TextFilesDatabase" "C:\MyTextFiles" 1000
First parameter is the Sql Server name. The second one is the Database name to create. The third one is the location where to store the text files on disk, and the last parameter is how many files to download.
Once that's done, you should have a database that looks something like this:
Just as a side point, the Dates are made up. They're just random dates between 1995 - 2005.
You should also have a directory (in the location you specified to the TextFileHarvester app) with many text files in them.
Part 4: Writing Some Code!!
Now comes the fun part. Earlier I mentioned that Solr was basically a Webservice that you can interact with using http and sending XML. You can do all of the work manually and create the XML by hand, but there's an awesome library out there that already does this. The library is open source and called SolrNet. Head on over there and download the dll's. (There's actually another library called SolrSharp but that's much older, and not as up to date as SolrNet. Also, in my opinion, SolrNet is much easier to use.)
Ok, now that we have the SolrNet dll's, we can create a simple application to Index them. We'll keep it simple, and do this: Connect to the database, get all of the files in one shot, connect to Solr and send them all to be indexed.
The way you send stuff to Solr using SolrNet is by having a class that holds your data, and then you add an attribute to each property that maps to a property in the index. Let me explain with code:
public class TextFile
#region Members
private string documentText;
public int FileID { get; internal set; }
public string FileLocation { get; internal set; }
public string DocumentText
if (this.documentText == null)
this.documentText = File.ReadAllText(FileLocation);
return this.documentText;
public string Title { get; internal set; }
public DateTime? DateCreated { get; internal set; }
You add the SolrField attribute (or the SolrUniqueKey for the index primary key field) and provide the name for that field in the index. I made the DocumentText property be lazy loaded so we don't hit the file system for all file's text in one shot (there should probably be more error handling there, but hey, this is just a demo...).
SolrNet will require a collection of these objects, and it will then send that off to the index. The next step therefore, is to write some code to populate these objects. We can use Linq to Sql, or whatever you want to connect to the database and populate these objects. I wrote some (very) basic ADO.NET code to do this:
internal class TextFileRepository
#region Members
private string connectionString;
#region Constructors
public TextFileRepository(string connectionString)
this.connectionString = connectionString;
#region Methods
public IEnumerable<TextFile> GetTextFiles()
return ExecuteSql("SELECT * FROM FILES");
public IEnumerable<TextFile> GetTextFiles(IEnumerable<int> fileIds)
if (!fileIds.Any()) { yield break; }
string sql = String.Format("SELECT * FROM FILES WHERE FILEID IN({0})", fileIds.ToDelimetedString());
foreach (var item in ExecuteSql(sql))
yield return item;
private IEnumerable<TextFile> ExecuteSql(string sql)
using (SqlConnection connection = new SqlConnection(this.connectionString))
using (SqlCommand command = connection.CreateCommand())
command.CommandText = sql;
var reader = command.ExecuteReader();
while (reader.Read())
yield return FromReader(reader);
private TextFile FromReader(SqlDataReader reader)
var result = new TextFile();
result.FileID = (int)reader["FileID"];
result.Title = reader["Title"] as string;
result.FileLocation = reader["FileLocation"] as string;
var date = reader["DateCreated"];
result.DateCreated = date == DBNull.Value ? (DateTime?)null : date as DateTime?;
return result;
Nothing fancy. If you look at the GetTextFiles method it's about as straight forward as they come. Just do a SELECT * and convert to the TextFile object. (You'll notice a GetTextFiles with a parameter, but that's used for Searching as I'll explain shortly.)
Part 4.1: Indexing the Files
Now, we can write some code to Index these files:
public class BasicIndexer
private string connectionString;
private string solrUrl;
public BasicIndexer(string connectionString, string solrUrl)
this.connectionString = connectionString;
this.solrUrl = solrUrl;
public void IndexFiles()
var solrWorker = ServiceLocator.Current.GetInstance<ISolrOperations<TextFile>>();
var files = new TextFileRepository(this.connectionString).GetTextFiles();
Pretty simple no? First, we need to call the Init method. Internally SolrNet uses IoC to handle the instantiation of the classes. Therefore, we can just use Microsoft.Practices.ServiceLocation from the Enterprise Library. (I'm not a huge fan of the Enterprise Library, but it's quick and easy for this example. Refer to the SolrNet documentation for better approaches using Castle Windsor.) Once we have the ISolrOperations, we can just call the Database to get all the files, and then call Add on the ISolrOperations. Once that's done, we just call Commit() and that's it!!
Let's give it a whirl. Create a quick ConsoleApplication and call the BasicIndexer class:
public class Program
public static void Main(string[] args)
string connectionString = "";
string solrUrl = "";
BasicIndexer indexer = new BasicIndexer(connectionString, solrUrl);
Fill in the blanks for your specific connection string, and your solr url (e.g. http://localhost:8983/solr). MAKE SURE YOU RESTART TOMCAT!! (Start -> Programs -> Configure Tomact -> Start)! Run the program and viola, you just indexed all of your files! (In a real world scenario, you'd obviously batch this operation, getting only x amount from the database, possibly event multithreading it to maximize performance, but again this is just a demo :) ). Now let's see if in fact our indexing worked. Open your browser to http://localhost:8983/solr (or whatever the Solr Url is) and click on Solr Admin. In the Search box enter "*:*" (the Lucene equivalent of SELECT *) and click the Search button. You SHOULD see this:
(depending on how many files you decided to download using the Harvester application you'll see different totals). Congrats! You've successfully indexed the files!
Part 4.2: Searching the Files
Now we want to search using C#. That too is VERY easy using SolrNet. The only thing to remember here, is like I mentioned earlier, we didn't STORE the fields in Solr. All we stored was the FileID. Therefore, we need to first retrieve those file id's, and then hit the database to get the rest of the information. It may seem like double work, but TRUST me!!! when dealing with larger data sets, it's MUCH faster. So, first let's create a class to hold the FileID's:
internal class FileIDResult
public int FileID { get; set; }
Same concept like Indexing. We use the SolrField attributes for the properties that will map to the Index fields.
Now, we can write some code to execute the Search. There are a few parameters I'd like to discuss first. Aside from the search query itself, you need to specify the amount per page, and which number to start from. Solr has paging built right into it, so the way it works is, you specify how many items you want per page, and then how many items to skip over. So for example, if you have 100 results, and you have 10 items per page, and you want Page 3, you'd start at item 30 (it's zero based). With that in mind, here's the searching code:
internal static class SolrOperationsCache<T>
where T: new()
private static ISolrOperations<T> solrOperations;
public static ISolrOperations<T> GetSolrOperations(string solrUrl)
if (solrOperations == null)
solrOperations = ServiceLocator.Current.GetInstance<ISolrOperations<T>>();
return solrOperations;
public class BasicSearcher
private string connectionString;
private string solrUrl;
public BasicSearcher(string connectionString, string solrUrl)
this.connectionString = connectionString;
this.solrUrl = solrUrl;
public SearchResults Search(string query, int resultsPerPage, int pageNumber)
var solrWorker = SolrOperationsCache<FileIDResult>.GetSolrOperations(this.solrUrl);
QueryOptions options = new QueryOptions
Rows = resultsPerPage,
Start = (pageNumber - 1) * resultsPerPage,
ISolrQueryResults<FileIDResult> results = solrWorker.Query(query, options);
var textFiles = new TextFileRepository(this.connectionString)
.GetTextFiles(results.Select(r => r.FileID));
var searchResults = new SearchResults
Results = textFiles,
QueryTime = results.Header.QTime,
TotalResults = results.NumFound
return searchResults;
public class SearchResults
public IEnumerable<TextFile> Results { get; set; }
public int QueryTime { get; set; }
public int TotalResults { get; set; }
Same idea as before. Get an instance to the ISolrOperations, and call the Query method. I used the overload that takes a QueryOptions object so that I can specify the page number and items per page. The result that comes back from that method call is an ISolrQueryResults which in addition to the search results itself, has some metrics. I wrapped all that up in the SearchResults class. Once we have the file id's we can hit the database, and get the rest of the data.
Let's give it a whirl:
public class Program
public static void Main(string[] args)
string connectionString = "";
string solrUrl = "";
BasicSearcher searcher = new BasicSearcher(connectionString, solrUrl);
var results = searcher.Search("*:*", 10, 1);
foreach (TextFile file in results.Results)
Console.WriteLine("FileID: {0}, Title: {1}",file.FileID,file.Title);
If you run this, you should see the file id's and titles for the first 10 documents! Congratulations, you've just executed a search!!
OK, this has gotten WAAAAAY longer than I ever imagined. There's ALOT going on here, I won't deny that. There's a few closing random points I'd like to make:
- There are many various Tokenizers and Analyzers for various data. If you need to tweak one and create your own, you're SOL to do it in C#, you'll have to blow the dust off that old Java book and do it there.
- The actual index is stored under $\solr\data\index. Sometimes it's useful to actually look into the index files and read them. For that you can use Luke (a standalone Java application.)
- In the real world, you'd possibly have your Indexing Server somewhere other than where the data is. Keep that in mind, as the XML sent across can get quite big, so see if you can optimize that.
There's so much more to Lucene and Solr than what I covered here. Here are some links for more reading:
Lucene Home Page
Lucene tutorial. (While this is an actual Lucene tutorial of how to use the Lucene Java libraries, it has some great insight in to what's happening behind the scenes.)
Solr Home Page
Solr Wiki
The various Tokenizers an Analyzers available from Solr
That's all I can think of for now. If there's anything else, I'll add it in the future.
Lastly, here's a zip file containing everything I covered in this post. You'll find the TextFileHarvester app. You'll find the solr configuration files I've used for this demo. You'll also find a simple library I created to wrap the indexing and searching functionality of these text files, as well as a basic windows application to test both. Good luck, and Happy Coding!
P.S. This tutorial took a lot of time and effort. If you find it useful, just drop me a a line in the comments.


text:
Mar 8, 2016
Seeing Through Photographs has given me a whole new way of looking at my own and other peoples' work. So many aspects, details and points of view, the world of photography now seems even more amazing.
May 28, 2020
Coursera gives me a platform to explore new skills. I was able to experience education from world-renowned universities without losing out on career opportunities.
- Thaneshwary kirupanantharasa -
By Usama B Y•
May 26, 2019
It was well organized course and learned essential principles to interpret photos. Thanks Coursera, MoMA and other sponsors for such an opportunity and platform for learning exploration. With Love!
By Yannis L•
Mar 5, 2018
This has been an excellent Course by MoMA on the Art and the medium of Photography! Mind you, it does not provide you with any technical skills or techniques at all. However, it handholds you through photography, from its inception up until the current era. During this process, it provides you with short yet excellent crafted excerpts from several photographers and their body of work, through brief subtitled videos. Within seven Modules, the course explains in depth the meaning of photography, the importance of context, and how you should learn to "read" photos, not only as a passive spectator but more actively. During the course you will learn to appreciate photography not only in terms of light, composition or the synthesis and the subject, but by also considering the background of the creator, his/ her prejudices, the era when each photo was taken, the social context that led to the capture of each distinct photo etc. The course also successfully criticizes the use of photography, helping you identify what is the ambivalent meaning of each photo and how it is frequently manipulated by the media. To make things better, the Course encompasses rich online reading material (some directly from the MoMA library) and also several informative slideshows. In brief, this a magnificent, well-designed Course if you are passionate about understanding photography. It will provide you with a restart point towards photography and it will create an immense appetite to delve again into photography and to try to understand it, this time by having a richer background. All from the comfort of your home, through self-paced yet timed Modules' sessions. An excellent course! Kudos to MoMA and Sarah Meister for crafting it.
By Vincent A D C•
Aug 21, 2016
I did enjoy these 6 weeks. I found the course very informative.
There are a lot of rich content , text, images, video that make the class the more interesting.
I really recommend this course.
By Alice D•
Dec 26, 2019
This is a really great way to get a beginners academic insight into photography, really interesting video and written content. I learned a lot and enjoyed studying again, thank you MoMA and Coursera!
By Nika P•
Mar 27, 2016
I was a student of Art History and I graduated with a theme focusing on Documentary Photography and Photojournalism, therefore I have read many books about photography and I thought I knew a lot about it. But this course showed me new perspectives and ideas, presented new photographers I have not yet know, and gave me new motivation to work in the field of photography. I also appreciated so much that there were scans of books and articles for us to read in combination with images and video - I think the mix of all three is a perfect way to study and should be also practised at Faculties (and from my experience, it is not always like that). Great work, I was really lucky and honoured to be part of the course.
By Alexis Q•
Mar 9, 2016
Seeing Through Photographs has given me a whole new way of looking at my own and other peoples' work. So many aspects, details and points of view, the world of photography now seems even more amazing.
By Emmanuel N Q E•
Jun 3, 2019
It was very insightful, and a good thing is they offered quizzes and other full content, which some other free courses do not, which really helps learning. Hope to go for other courses under them. :)
By David D•
Jan 27, 2017
This was my first course done through Coursera. The MOMA resources were great and I found the course really well curated allowing me to develop my understanding of photography that much better. The modules moved me from one interesting aspect to another. Reading resources were excellent as well as all the other clips embedded in the program. Opportunity to provide and receive feedback was good and the online platform worked really well.
By EdusoundsNg•
Mar 11, 2016
The course was educative and highly engaging. The short video clips with the quiz questions at the end of each module helped me to further understand the topic of discussion.
Furthermore, the submission at the end of the course and review of colleagues work added to my better understanding of the course, as I was able to share in the understanding of my other coursemates.
I hope MOMA will come up with independent courses that will treat each of the six modules as a separate course.
By Thaneshwary k•
May 29, 2020
Coursera gives me a platform to explore new skills. I was able to experience education from world-renowned universities without losing out on career opportunities.
- Thaneshwary kirupanantharasa -
By Garry R•
Jun 3, 2016
I would have, if possible, not rated this course. I was not interested in the work of the more recent photographers/photo manipulators. This is a course for the art crowd, not for those interested in the history of photography.
By Chean K•
Oct 30, 2019
It was a pleasure and valuable learning experience to study this excellent, very well presented Coursera course, and it felt a privilege also to have the course led by such a world-renowned respected organisation as MOMA! I liked the structure of the course, with its good balance of written material, excellent video lessons, and assessment exercises. The list of reading material for each week was highly relevant to that week's theme - with the required reading providing a sound basis for understanding photographic ideas, theory as well as practice, and also provided a direction for further investigation. I really appreciated the videos where we get to hear photographers talk about their works, which gave insight into the photographers' mindset and what they were trying to show through their work. As well as all that, I enjoyed seeing the large range of photographs that was presented as part of the learning. These covered works by accomplished, talented, acclaimed, and some historically important photographers - and for me, this was the icing on he cake! Thank you Coursera and MOMA for this wonderful course!
By Alina T•
Jun 25, 2016
Seeing Through Photographs by MoMa is an extremely well organised course. Sarah Meister from the Department of Photography has done a fabulous job of curating this course. The modules of the course follow a chronological process which is easy to understand. As a person who has just finished the entire course, I would have to recommend it to fellow photographers who are looking for a way to be involved in the various discussions regarding photography in the art world. It encourages you to grapple with some of the questions raised and challenges you to be consistent. It is a great course for anyone interested in the history of photography and evolving contemporary practices in the field. What strikes me the most is that it is not an outdated lesson in history but ties up the course together by making it relevant to our time and age. Overall a great course by MoMA and I had an amazing time learning.
By Beata W•
Apr 2, 2018
The course presents many important topics in a very attractive and engaging way. The course changed my perception of photography very much. Many of my photographs stopped to please me .. I also appreciated a few previously rejected .. I got to know some of the works of many great artists and intend to deepen the knowledge about these artists. The most valuable value that I personally drew from the course is the enlargement of the frame .. thinking about how important is everything that is what is important and what so far I neglected and disregarded in the frame. How to change the focal length from 100mm to 35mm Participation in this course has put a lot of new questions ahead of me ... and gave me inspiration to look for answers at the end of the course. Thank you for preparing this wonderful course.
By Valeria B G•
Apr 10, 2016
Bastante bien todo, tuve algunos problemas as y nadie los podía resolver al ser online pero después pude seguir adelante y logran con éxito los objetivos. Sugiero que sea realmente en castellano cua do a uno lo dejan optar x su idioma. Ya que aparentemente quiere ser un curso internacional y al alcance de todos al poder ser gratuito. Uno si o si debe tener un altísimo entendimiento de inglés para finalizarlo, o mayor tiempo para hacerlo de igual modo con tanta información. El material es de primerísima calidad y los felicito por eso! Espero sigan mejorando y hacer más cursos con ustedes ya que son realmente desafiantes! Muy interesante todo el curso y las obras de artes que facilitan junto al material de aprendizaje!!!!! Muchas gracias!
By Olivia P•
May 5, 2020
This course is a magnificent introduction to the various modes of seeing photography - it offers an expansion on just seeing the subject matter of a photograph by bringing into focus different strategies and techniques which could alter the visual discourse within an image altogether. It also offers a great introduction to a few contemporary names (photographers which have been exhibited within MoMa). All in all, it gives the learner a great stepping stone into becoming more comfortable with their knowledge of photography as a medium , their ability to consume and understand the photographic output they are being exposed to on a daily basis and their ability to speak about them with confidence.
By Gerhard C•
Jun 26, 2017
This course exposes you to a variety of approaches to creating and viewing photograph-based art. The student is responsible for integrating the material in a personal way, and can always do more than the minimum required. Instructions are clear, references are plentiful, and the quiz items are mostly straight-forward, on two or three occasions a bit too esoteric. Keeps you on your toes, though, and you can repeat the quizzes. Overall, a very valuable experience, to experience these topics from the perspective of a first-rate museum!
By Chris H•
Aug 17, 2016
Excellent course! One of the best I have ever taken on photography. It opens up a large variety of topics and plots them along the evolution of photography from its very beginning to today. All of the content was hugely interesting, the materials and required readings were well-chosen and, most important of all, the course opened my eyes to many new possibilities of creative expression. This course is a must for anyone seriously interested in the subject of photography.
By Jim W•
May 30, 2016
If what you want is the sort of doctrinaire overview of fine-art photography that you might have learned in a university in the 1970s, this is exactly the course for you!
It's thoroughly saturated with the Museum of Modern Art mindset: the assumptions and interpretations are those touted by MoMA curators, and the photographers are chosen from the MoMA-approved pantheon. (The quiz questions tend to focus on course-content minutiae -- "Which commentator said X about Y?" -- to make sure you've drunk your art-establishment Kool-Aid.)
On the other hand, if you want to examine the diversity of photography through a variety of critical approaches -- Marxist, feminist, queer-theory, post-structural, whatever -- well, they don't allow none of that there here. This is strictly the Photography Gospel According to MoMA, and you either like it or lump it.
By Felipe S C•
May 20, 2016
No actual class, just for selling books.
By Deborah A H•
Nov 12, 2018
Seeing Through Photographs was such a wonderfully complete and inspiring course. The design and content provided a straight forward yet challenging set of well documented resources to learn absolutely everything about the history, development and current components of photography, primarily American.
I absolutely gained so much from the manner of presentation. The details concerning the exhibitions and literature produced by Department of Photography Museum of Modern Art as well as access to its resources was phenomenal. I loved the component breakdowns.
I cannot believe how much I did not know as I have been photographing for over 20 years. The course filled in so many blanks in my mind in terms of history, focus and expansion of the full use of this most valuable tool to see the world. Thank you all for your hard work in creating such an awesome course.
By Giulia R•
Oct 5, 2017
Very interesting, intensive course about the history of American photography and its pioneers, the captivating details about the creative processes and the impact of the medium on the mass and art culture. Interviews, led by curator Sarah Meister, were very appealing, thorough and well-finished. I enjoyed getting to know the stories behind iconic pieces and the artistic lives of international photographers, it spurred my own creativity. Readings were also a relevant support in overcoming the tests at the end of every module and writing the final essay for the completion of the course. Fair grading given by the instructor and other partecipants.
By Anabel S d G•
Jul 23, 2017
Amazing course! Enjoyable from the first day. Sara Meister and her team did a great job with the video selections, interviews, reading material and visual input. Not to mention the invaluable sharing of her knowledge as curator of the MOMA photography department.
What a wonderful experience! I highly recommend it!
This course has given me the opportunity to enhance my knowledge on photography and strengthen my confidence towards the way I see photographs.
My best wishes to every one that decides to take this course! Regards
By Ana L•
Mar 7, 2017
It was such a pleasant journey, thank you so much because I learned a lot and specially I discover a variety of talented photographers that have inspire me even more, This course has aroused my curiosity in to all the different types and possibilities what we can work and explore within. I really like this course and the way it is done. Sorry if I did not participate on the discussions but apart the lack of time as my english is not very good I do not feel very comfortable exposing my ideas on another language.
By Michela B•
Mar 30, 2016
A really interesting course on the history of Photography throughout its main categories, on its relationship with the world, with society, with people, with media and technology, from early years to now. The course is organized in exhaustive and complete lessons and modules which include interviews, final texts, confrontation exercises with the community, free online sources and educational materials.