FIELD

Embodiments of the invention relate to finding content on computer networks. With embodiments, the content cannot be found by following known web links, so the content may be described as “hidden” or “uncrawlable”.

BACKGROUND

Oftentimes, a search engine “crawls” (searches) for content (e.g., web sites) on a computer network, such as the Internet, and indexes the content for use in responding to search requests (queries). A sitemap may be described as a list of pages of a web site accessible to the search engine.

However, some content may only be found behind proprietary search boxes on individual websites and such content is not provided in the sitemap. In some cases, such content may be found by crawling links across websites (i.e., by going to a second website that links to the first website's uncrawlable content and following the link to access the uncrawlable content. Yet, there is some content that is still not crawled, since it has never been linked to from any web site, or at least any publicly crawlable web site.

For instance, content on a company's support portals is not always well indexed. New content (e.g., technotes, ifix documents, etc.) may take a while to be indexed in search engines. Some pages are dynamically generated. In this case, any number of pages may be generated based on a large number of queries, which are not enumerated ahead of time in any sitemap. It is in the web site's interest to be well indexed (so that they are located for search requests and displayed to a user), so web sites have a motivation to provide complete sitemaps of content that may be hidden behind search fields. Still, some web sites do not do this. It is also in the interest of the search engine to have a complete index, despite web sites not providing a complete sitemap.

SUMMARY

Provided is a method for finding content on computer networks. The method comprises: receiving, with a processor of a computer, a search request having search terms; executing the search request against an index to identify first search results, wherein the index comprises mappings of search terms to web sites and mappings of search terms to patterns; and, in response to determining that the search request failed based on first search results, identifying an unknown search term in the search terms that does not exist in the index, matching the unknown search term to a pattern of a similar search term using the mappings of search terms to patterns, identifying a candidate web site based on the matching, requesting additional content from the candidate web site for the unknown search term to identify second search results comprising previously hidden content, and indexing the additional content results.

Provided is a computer program product for finding content on computer networks. The computer program product comprises a computer readable storage medium having program code embodied therewith, the program code executable by at least one processor to perform: receiving, with a processor of a computer, a search request having search terms; executing the first request against an index to identify first search results, wherein the index comprises mappings of search terms to web sites and mappings of search terms to patterns; and, in response to determining that the search request failed based on first search results, identifying an unknown search term in the search terms that does not exist in the index, matching the unknown search term to a pattern of a similar search term using the mappings of search terms to patterns, identifying a candidate web site based on the matching, requesting additional content from the candidate web site for the unknown search term to identify second search results comprising previously hidden content, and indexing the additional content results.

Provided is a computer system for finding content on computer networks. The computer system comprises one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices and program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to perform operations comprising: receiving, with a processor of a computer, a search request having search terms; executing the search request against an index to identify first search results, wherein the index comprises mappings of search terms to web sites and mappings of search terms to patterns; and, in response to determining that the search request failed based on first search results, identifying an unknown search term in the search terms that does not exist in the index, matching the unknown search term to a pattern of a similar search term using the mappings of search terms to patterns, identifying a candidate web site based on the matching, requesting additional content from the candidate web site for the unknown search term to identify second search results comprising previously hidden content, and indexing the additional content results.

