Q:

Python multiprocessing hive hang in Linux

The code below worked in Windows but in Linux is hanging:
from impala.dbapi import connect
from multiprocessing import Pool
conn = connect(host='172.16.12.12', port=10000, user='hive', password='hive', database='test',auth_mechanism='PLAIN')

cur = conn.cursor()
def test_hive(a):
    cur.execute('select {}'.format(a))
    tab_cc = cur.fetchall()
    tab_cc =tab_cc[0][0]
    print(a,tab_cc)

if __name__ == '__main__':
    pool = Pool(processes=8)
    alist=[1,2,3]
    for i in range(len(alist)):
        pool.apply_async(test_hive,str(i))
    pool.close()
    pool.join()

When I change alist=[1,2,3] to alist=[1] it works in Linux.

A:

I see two possible causes for this behavior: 

an exception raised in test_hive in the context of a forked subprocess
a deadlock caused by the fact that fork does not copy threads from the parent and/or the fact that mutexes are copied in the state they have when the fork call is executed

To check for exceptions, add return tab_cc to the end of your test_hive function and gather the results returned by the pool:
if __name__ == '__main__':
    pool = Pool(processes=8)
    alist = [1,2,3]
    results = []
    for i in range(len(alist)):
        results.append(pool.apply_async(test_hive, str(i)))
    pool.close()
    pool.join()
    for result in results:
        try:
            print(result.get())
        except Exception as e:
            print("{}: {}".format(type(e).__name__, e))

As for the threads, I did a quick search through the impala repo and it seems like they somehow play a role around the usage of thrift. I'm not sure if Python's threading module can actually see them, when originating from that library. You might try with print(multiprocessing.current_process(), threading.enumerate()), both at the module level (e.g. after cur = conn.cursor()) and at the beginning of the test_hive function and see if the _MainProcess(MainProcess, started) shows a longer list of active threads than all of the ForkProcess(ForkPoolWorker-<worker#>, started daemon).
As for a potential solution: I somewhat suspect the fact that you create conn and cur at the module level to be the culprit; all childs use a copy of those two.
Try and move these two lines to the beginning of test_hive, so that each process creates a connection and a cursor if its own:
conn = connect(host='172.16.12.12', port=10000, user='hive', password='hive', database='test',auth_mechanism='PLAIN')
cur = conn.cursor()

