Q:

How to assemble and solve a matrix system in parallel from values generated in different processors?

I am solving a multiscale problem using the Heterogeneous Multiscale Method (HMM).  Essentially, my particular procedure uses the following iterative process:  

Solve many local matrix systems.
Compute a value of interest from the solutions of the local systems.
Assemble a global matrix system from the local "values of interest"
Solve the global matrix system
Use the solution of the global matrix system to form new local matrix systems.  

Repeat until some convergence criteria is met.  
Since there are many local (independent) linear systems of equations and multiple systems can fit into local RAM memory, I figure it is best to load multiple "local" systems into each processor and solve each system sequentially (see this posted question).  
My question regards the best strategy to assemble and solve the global matrix system.  In my particular case, the global matrix system is small enough that it could fit entirely on any processor's RAM memory.  Furthermore, the local & global matrices do not change size between iterations.  So, I foresee one of three possible strategies:  

Gather the "values of interest" onto a single processor, and assemble/solve the global matrix system sequentially on one processor.  
Copy the values of interest onto every processor, and assemble/solve the same global matrix system sequentially on every processor.
Assuming that each processor possesses the "values of interest" necessary to produce contiguous blocks of the global matrix, then we can assemble partitions of the global matrix locally, then solve them together in parallel.

I can see some advantages/disadvantages to each method.  In Method 1, no communication is necessary in the solving phase, but communication to and from the root processor may become a bottleneck (especially at scale).  Method 2 may require more interprocessor communications to assemble the global matrix than the first method, but no communication is needed in the solving phase or in the local matrix assembly stage that follows. Method 3 requires no interprocessor communication for assembly of the local or global matrices, but requires it in the solving phase.
Suppose that each local system is on the order of $10^3$x$10^3$ and there are $10^3$x$10^3$ local matrix systems. Let's further suppose that the global matrix system has size $10^3$x$10^3$.  Under these assumptions, which one the three aforementioned strategies will likely lead to a faster solution of the global system?  Are there other mapping strategies for the global matrix that might work faster per iteration?

A:

I do not think there is any case where you want to solve on rank 0. Redundant solve is almost always better since, for small things, allreduce is as efficient as reduce, and redundant computation only has one instead of two.
However, whether to compute redundantly on all nodes, or on a subset, or redundant subsets depends on the hardware and system size. Thus, you should have a system that can do any of them. The PCREDUNDANT in PETSc can solve redundantly on all processes, some processes, or subsets of processes in parallel.
But if the global problem is of size $10^6$ as you claim in the comments, it is large enough to benefit significantly from a parallel solve. Parallel assembly is very much the standard and recommended scenario.

