Q:

Aggregating heterogeneous documents in MongoDB

I am writing a tool to inspect a MongoDB session store, in which the structure is namespaces containing variables containing information on the variables, roughly:
{
    "some namespace" : {
        "some variable" : {
            "t" : "serialization type",
            "b" : "data value",
            "c" : "java class type"
        },
        "some other variable in this namespace" : {
            "t" : "serialization type",
            "b" : "data value",
            "c" : "java class type"
        },
        ...
    },
    ... // more namespaces
}

The names of the namespaces and variables are all different. I wrote a map-reduce job to loop through the namespaces and loop through the variables in each namespace and emit for each variable in each namespace.
However, I want to try to switch to Mongo's aggregation framework to avoid potential write lock problems. Based on the docs, it seems I have to specify the names of the variables to $group on, which doesn't seem compatible with heterogeneous data like mine. Is there any way to involve iteration in an aggregation invocation?

A:

Currently, to use the aggregation framework, you have to know the field names to write the query. There are some proposals to remove that limitation.

Right now, given your design, to use the aggregation framework, you probably have to consolidate at regular intervals your data using map-reduce to something more or less like that:
{
    namespaces: [
      { name: "some namespace",
        variables: [
          { name: "some variable",
            t : "serialization type",
            b : "data value",
            c : "java class type"
          },
          { name: "some other variable in this namespace",
            t : "serialization type",
            b : "data value",
            c : "java class type"
          },
          ... // more variables
        ]
      },
      ... // more namespaces
    ]
}

Now it would be relatively easy to query using the aggregation framework. Admitting you may work on stale data...

