Q:

python map_async, where is the overhead coming from?

I am using map_async as intended - to map an iterable over multiple processing cores using:
cores = mp.cpu_count()
pool = mp.Pool()

r = pool.map_async(func, offsets,callback=mycallback)
r.wait()

func returns a dict, so the callback 'merges' the dicts using:
ddict = defaultdict(set)
def mycallback(w):
    for l in w:
        for key, value in l.items():
            for v in value:
                ddict[key].add(v)  

Offsets is an iterable that I have tested with 1,000 - 50,000 elements.  
If I remove r.wait() it is not possible to return all of the output from the map_async call.  
Using r.wait(), I am seeing processing times that are both inferior to a serial implementation and do not scale, i.e. the parallel implementation increases in time exponentially, while the serial version increases linearly.
I know that func is sufficiently expensive as in serial and parallel it pegs my processing cores.
Where have I introduced overhead using map_async?  It is not in the callback function, as removing and replacing with result.append does not impact time. 
Edit for Comments:

I am moving large dicts around, anywhere from 1,000 - 100,000 elements.  The value are sets that are generally 3-5 elements.  So, pickling could definitely be an issue.  What alternative data structures would one suggest without moving to something in shared memory?
apply_async with a similar callback, save the for l in w line, returns roughly the same results.  Speed is slightly better than map_async for some problem sets and slightly worse for others.  Using a managed dict and a joinable queue are significantly worse. 
Some time tests.  This is using 2 cores.  As I add additional cores I see the exponential increase, so I can only assume that that increase is caused by process spawning or pickling to return data.

func takes a data point and looks for neighbors.  It is an identical function for all cases, except for the need to pass offsets telling the parallel code which data points to search.  This is essentially a KDTree search function.
Homogeneously Distributed
1,000 data points: Serial 0.098659992218 | apply_async  0.120759010315 | map_async 0.080078125
10,000 data points <====== ONLY IMPROVEMENT WITH PARALLEL | Serial 0.507845163345 | apply_async  0.446543931961 | map_async 0.477811098099
Randomly Distributed
10,000 data points: Serial 0.584854841232 | apply_async 1.03224301338 | map_async 0.948460817337
50,000 data points: Serial 3.66075992584 | apply_async 4.95467185974 | map_async 5.37306404114

A:

Can you change func() to return dictionaries of sets instead of dictionaries of lists?  Then your callback function could be rewritten like this:
def mycallback(w):
    for l in w:
        for key, value in l.items():
            ddict[key].update(value)

That should help with both serial and parallel processing times.
Unfortunately, I think @Dougal is right about pickling/unpickling all that data when passing it between threads.  It might be faster to write binary data to disk and read it back again, instead of passing it around in memory because of the overhead of pickling.  You could use a format like:
key value1 value2 value3 ...
key2 valueA valueB valueC ...
...

which should be easy to both write and read.

