Embodiments relate generally to computer data systems, and more particularly, to methods, systems and computer readable media for data partitioning and ordering.

Data sources within a computer data system may include static sources and dynamic sources. Data from these sources may be stored into one or more tables that may not preserve an ordering of the data when stored on a physical data storage. The tables may not be partitioned and/or grouped so as to maintain an order to the stored data. Thus, when data is retrieved from the tables and a need to order the data exists, there may additional computation cycles required to order the retrieved data. Also, a need may exist to provide an efficient method of data storage and retrieval for large data sets in computer data systems. Embodiments were conceived in light of the above mentioned needs, problems and/or limitations, among other things.

Some implementations can include a method for storing ordered data in a computer data system. The method can include receiving an electronic message containing one or more data items, and storing the received data items to a first set of one or more first partitions as first stored data in a first electronic data storage device. The method can also include remapping the data items stored in the first set of partitions to a second set of one or more second partitions in a second electronic data storage device, each of the second partitions having a respective grouping, and storing the remapped first stored data as second stored data in the second set of partitions in a second electronic data storage device according to the respective grouping.

The method can further include sorting each second partition of the second set of partitions according to a strict ordering to generate ordered second partitions, and storing the ordered second partitions in the second electronic data storage device so as to maintain the strict ordering. The method can also include organizing each ordered second partition into one or more groups having one or more grouping levels based on one or more column values, wherein the organizing includes generating grouping metadata associated with the respective ordered second partition.

The second partition of the second set of partitions corresponds to a directory in a file system. The first electronic data storage device and the second electronic data storage device can be different from each other. The second partitions are arranged according to an overall schema and the ordered partitions are arranged according to a partition schema associated with each respective ordered partition.

Some implementations can include a method for retrieving ordered data in a computer data system. The method can include receiving an electronic message including a computer data system query, and performing first processing of the computer data system query to identify one or more data objects within a set of partitions stored on an electronic data storage device, the one or more data objects identified as containing data responsive to the query. The method can also include generating an intermediate result data object based on the first processing, and performing second processing of the computer data system query to generate an ordered collection of index values corresponding to one or more data items from the identified data objects having data responsive to the query, wherein the ordered collection contains index values into one or more data objects maintained according to a strict ordering associated with the data objects. The method can further include generating a subsequent result data object based on the second processing, and providing a reference to the subsequent result data object containing the ordered collection as a query result, the subsequent result data object including references to data and corresponding index values for that data, wherein the subsequent result data object maintains the strict ordering associated with the data objects.

The intermediate result data object may include only locations and location keys of data responsive to the query. The subsequent result data object can include locations of data responsive to the query and one or more references to data responsive to the query. Accessing and movement of data is delayed until after the first processing of the computer data system query. The set of partitions can include a hierarchical arrangement of partitions.

Some implementations can include a method for computer data system data object schema modification. The method can include receiving a request to modify a schema associated with a data object, and retrieving schema information from metadata associated with the data object. The method can also include modifying only the schema information according to the received request, and storing the modified schema information in the metadata associated with the data object, wherein the data object is accessible according to the modified schema information.

The method can also include providing the modified schema from the metadata associated with the data object in response to a request for information about the data object. The request to modify the schema can include a request to add a column to the data object, and wherein the column can be added by schema modification without accessing any of the data in the data object. The request to modify the schema can include a request to remove a column from the data object, and wherein the column can be removed by schema modification without accessing any of the data in the data object.

The request to modify the schema can include a request to modify a data type of a column within a data object, and the modified schema information can include a modified data type for that column based on the request.

The request to modify the schema can include a request to add a formula column to the data object, wherein a definition of the formula column can reside in the schema information and the formula column may not contain stored data, and wherein when data is requested from the formula column, the computer data system can compute a response result to the request for data from the formula column based on one or more data columns referenced by the formula column and returns that result.

The request to modify the schema can include a request to add a formula column to the data object, wherein a definition of the formula column can reside in the schema information and only a portion of rows in the formula column contain stored data, and wherein when data is requested from the formula column, the computer data system can compute a response result to the request for data from the formula column based on stored data in the formula column for rows where data is present and, for rows where data is not present in the formula column, can compute a response result to the request for data from the formula column based on one or more data columns referenced by the formula column and returns that result.

