FIELD OF THE PRESENT TECHNOLOGY

The present technology relates generally to data processing, and more particularly, to systems and methods for enriching documents for indexing.

BACKGROUND

Search platforms can perform indexing of documents to optimize storage of the document and facilitate fast document retrieval. Prior to indexing, search platforms may pre-process documents. The pre-processing may include, for example, normalization of a text by removing extra spaces and converting the text to lowercase. In some cases, when pre-processing documents, it is desirable to enrich the documents with extra information to make the documents more informative upon retrieval. The extra information can also make indexing of documents more accurate. It is also important to ensure that the process of searching, retrieving, and adding the extra information does not add substantially to the time required for pre-processing and indexing the documents.

SUMMARY

This section is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description section. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

According to one embodiment of the disclosure, a method for enriching documents for indexing is provided. The method may include receiving a plurality of documents and generating a plurality of enriched documents. The generation of the plurality of enriched documents can include determining reference data based on a document of the plurality of documents. The generation of the plurality of enriched documents can further include determining, based on the reference data and an enrichment policy, additional data, and adding the additional data to the document. The method may further include indexing the plurality of enriched documents to generate a target index.

Prior to the generation of the plurality of enriched documents, the method may index the reference data of plurality of documents to obtain a source index. Thereafter, an enrichment index can be generated based on the enrichment policy and the source index. The additional data can be retrieved from the enrichment index. The additional data can be retrieved by performing a remote search in a batch, wherein the remote search includes a query of the additional data for at least two documents of the plurality of the documents.

The enrichment policy can define at least one data field to match the document to the reference data and at least one additional data field. Determining the additional data may include determining, based on reference data, a value for the data field and determining, based the value for the data field, a value for the additional data field. The additional data field and the value for the additional data field can be added to the additional data. The value of the additional field can be determined based on metadata associated with the document. The value of the additional field can be obtained from a look up table or a remote source. The enrichment policy may define a query for searching a value of the additional field in the look up table or the remote source.

The additional data may include geographical data associated with the document, an Internet protocol address associated with the document, a domain address associated with the document, user data associated with the document, and organization data associated with the document.

According to one example embodiment of the disclosure, a system for enriching documents for indexing is provided. The system may include a processor and a memory communicatively coupled to the processor. The memory may store instructions executable by the processor, wherein the processor can be configured to implement operations of the above-mentioned method for enriching documents for indexing upon execution of the instructions.

According to yet another aspect of the disclosure, there is provided a non-transitory processor-readable medium, which stores processor-readable instructions. When the processor-readable instructions are executed by a processor, they cause the processor to implement the above-mentioned method for enriching documents for indexing.

Additional objects, advantages, and novel features of the examples will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and the accompanying drawings or may be learned by production or operation of the examples. The objects and advantages of the concepts may be realized and attained by means of the methodologies, instrumentalities and combinations particularly pointed out in the appended claims.

