The major benefit of GIST arises from its communication-efficient methodology for disjointly partitioning the weights of a single, large GCN model across multiple machines for distributed training.
A simple version of local SGD [1]} could also be implemented for distributed training of GCNs by training the full model on each separate worker for a certain number of local iterations and intermittently averaging local updates.
Similarly to GIST, this method would allow GCN training to be distributed across multitple GPUs, thus enabling some larger-scale experiments to be explored.
However, GIST has superior communication-efficiency in comparison to local SGD, as only a fraction of model parameters are communicated to each machine.
Furthermore, because each sub-GCN is smaller than the global model, GIST achieves an additional speedup during the local training of each sub-GCN in comparison to locally training the full model.
To verify the efficiency benefits of GIST, we perform a direct comparison to local SGD over the Reddit dataset using a 2-layer GCN model with a hidden dimension of 256.
<TABLE>