Yesterday, I came across Nextag which is a website for comparing prices of various products from different retailers. Is this site automated? Can the process of data pulling from various sites can be automated for comparison with each other. I tried understanding a hell lot, but could not realize this fact and now think that they ought to be manually managed because on every other retailer site, the product name was different and it was very hard to recognize a similarity for a specific product.I have made such a system in which one can add a product and then add the respective retailers and their pricess. Every thing is manual in here. Today, I will be entering the data into the system, but thought it would be better to know if there is a way to automate this thing. Or I am going the right way as there can't be an automation for such system, because the data has no similarities.  
How to : Nextag and such other dynamic data pulling website
web applications;product management;dynamic data
Such applications face two important challenges: Gathering the data, andMaking sense of the data.Gathering the dataA variety of techniques may be employed, depending on what's supported by each source site:Simple web crawlerAnything from a simple curl script to a full fledged custom web crawler.   Web servicesThe optimal way, if source site supports them. Any kind of web service, from RSS to SOAP, that's their point, to provide a method of communication between two sites over the web.Black magic stuffDepending on the relation of the source and target sites, this can be anything really. I've seen one site that literally send SQL dumps of it's database to another site, via ftp. That's definitely not recommended, but it's a good example of a weird counter-intuitive approach you never thought existed, until you saw it with your own eyes.By handObviously not recommended. But under very specific circumstances and for very small sets of data it might prove to be the optimal solution.The automated techniques commonly require some time-based background process, and for websites the obvious example is a cron script. Scheduling plays an important part in the process as you need to schedule the gathering process(es) according to:How often products are modified (created / deleted)How often product information is modifiedBandwidth restrictions (on your site and each source site)I've separated 2 from 1, as I've seen them treated as separate processes, with different workflows and scheduling. For example I've worked on a similar site that had a vendor that added new products only on Mondays. So we only checked on Tuesdays, at 00:00. Making sense of the dataThis is where the fun begins! Heterogeneous data is always a blast to work with! Every bit of data you gather will most probably follow a unique structure. If you are lucky enough to gather the data via a web service, chances are that their structures will make sense and you can build a combined index relatively easy.But if there's at least one site you have to crawl, or its web service structure doesn't really makes sense, then your index is not going to be that easy to build. The most common problem is the one you've identified:For example, in case of iPhone, some websites name Apple iPhone, while some name it iPhone 4S. How can one make sure that the sync is maintained. There's no single answer to that, it depends on data size, volume, complexity etc. It can be as simple as building and maintaining an associative index by hand: tokenA1 == tokenB1 == token1tokenA2 == tokenB5 == token2tokenA5 == tokenB2 == token3or as complex as building a custom algorithmic solution to discover the associations automatically. Some relevant buzzwords are: Pattern matching:In computer science, pattern matching is the act of checking some sequence of tokens for the presence of the constituents of some pattern. In contrast to pattern recognition, the match usually has to be exact. The patterns generally have the form of either sequences or tree structures.Pattern recognition:In machine learning, pattern recognition is the assignment of some sort of output value (or label) to a given input value (or instance). An example of pattern recognition is classification, which attempts to assign each input value to one of a given set of classes (for example, determine whether a given email is spam or non-spam)Information extraction:Information extraction (IE) is a type of information retrieval whose goal is to automatically extract structured information from unstructured and/or semi-structured machine-readable documents. In most of the cases this activity concerns processing human language texts by means of natural language processing (NLP).Document retrieval:Document retrieval is defined as the matching of some stated user query against a set of free-text records. These records could be any type of mainly unstructured text, such as newspaper articles, real estate records or paragraphs in a manual. Statistical classification:In machine learning, statistical classification is the problem of identifying the sub-population to which new observations belong, where the identity of the sub-population is unknown, on the basis of a training set of data containing observations whose sub-population is known. Therefore these classifications will show a variable behaviour which can be studied by statistics.Data mining:Data mining (the analysis step of the knowledge discovery in databases process,1 or KDD), a relatively young and interdisciplinary field of computer science2 is the process of discovering new patterns from large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics and database systemsOne algorithm (or variation of1) I've seen used in a somewhat comparable scenario is the k-nearest neighbor algorithm:In pattern recognition, the k-nearest neighbor algorithm (k-NN) is a method for classifying objects based on closest training examples in the feature space.Normally, it won't be that hard, my intention is more to illustrate the spectrum of possible solutions than pinpointing an optimal solution. There isn't a single one, the optimal solution will be different for each target site and almost in every scenario you'll need to mix and match.A possible solution of average difficulty that would work for a wide range of source sites could be build around a text search engine / information retrieval library, like Apache Lucene, or even just a few well written regular expressions. And of course, to make matters worse, there isn't a single workflow. The simplest I can think of are:Gather the data first, then analyse,Analyse the data while gathering. Which highly depends on your data storage, and it might even depend on your hardware. It will also depend on how you gather the data, multiple requests for lots of data pose the risk of hurting the target site, so the workflow may even be different per site.It's a grand mess! Make sure you ask for a lot of money if ever involved in such a project...1 To be perfectly honest it might have been something different entirely, at the time I was told it was a naive nearest neighbor search, but was too young and impressionable to deeply investigate. It worked, though, and it looked like NNS.