Q:

how to add new categorical column in pandas

I have a dataframe like this, with 10M rows:
                     probe
time                      
2016-01-01 00:05:00    3
2016-01-01 00:05:00    1
2016-01-01 00:05:00    5
2016-01-01 00:05:00    5
2016-01-01 00:05:00    4
2016-01-01 00:05:00    2
2016-01-01 00:05:00    5
2016-01-01 00:05:00    6
2016-01-01 00:05:00    3
2016-01-01 00:05:00    4
2016-01-01 00:05:00    5
2016-01-01 00:05:00    2
2016-01-01 00:05:00    3
2016-01-01 00:05:00    3
2016-01-01 00:05:00    5
Name: probe, dtype: uint8

I want to add a categoricat column based on the value of probe
def categorize_R(x):
    return "inner" if x['probe'] in (1, 4) else "outer"

data['category_R'] = pandas.Categorical(data.apply(categorize_R, axis=1))

this is terribly slow. Actually computing a mask like this:
mask_inner = (x['probe'] == 1) | (x['probe'] == 4)

is quite fast, but then I don't know how to add a column of type categorical.

A:

I think you need numpy.where with mask created by between:
mask = data.probe.between(1,4)
data['category_R']  = pd.Categorical(np.where(mask, 'inner', 'outer'))
print (data)
                     probe category_R
time                                 
2016-01-01 00:05:00      3      inner
2016-01-01 00:05:00      1      inner
2016-01-01 00:05:00      5      outer
2016-01-01 00:05:00      5      outer
2016-01-01 00:05:00      4      inner
2016-01-01 00:05:00      2      inner
2016-01-01 00:05:00      5      outer
2016-01-01 00:05:00      6      outer
2016-01-01 00:05:00      3      inner
2016-01-01 00:05:00      4      inner
2016-01-01 00:05:00      5      outer
2016-01-01 00:05:00      2      inner
2016-01-01 00:05:00      3      inner
2016-01-01 00:05:00      3      inner
2016-01-01 00:05:00      5      outer

Another solution is use Categorical.from_codes, check also object creation - In [28]::
mask = (data['probe']==1) | (data['probe']==3) | (data['probe']==4)

mask = (data['probe']==1) | (data['probe']==3) | (data['probe']==4)
data['category_R']  = pd.Categorical(np.where(mask, 'inner', 'outer'))
data['category_R1']  = pd.Categorical.from_codes(mask, ['outer','inner'])
print (data)
                     probe category_R category_R1
time                                             
2016-01-01 00:05:00      3      inner       inner
2016-01-01 00:05:00      1      inner       inner
2016-01-01 00:05:00      5      outer       outer
2016-01-01 00:05:00      5      outer       outer
2016-01-01 00:05:00      4      inner       inner
2016-01-01 00:05:00      2      outer       outer
2016-01-01 00:05:00      5      outer       outer
2016-01-01 00:05:00      6      outer       outer
2016-01-01 00:05:00      3      inner       inner
2016-01-01 00:05:00      4      inner       inner
2016-01-01 00:05:00      5      outer       outer
2016-01-01 00:05:00      2      outer       outer
2016-01-01 00:05:00      3      inner       inner
2016-01-01 00:05:00      3      inner       inner
2016-01-01 00:05:00      5      outer       outer

Timings:
In [181]: %timeit pd.Categorical(np.where(mask, 'inner', 'outer'))
1000 loops, best of 3: 196 µs per loop

In [182]: %timeit pd.Categorical.from_codes(mask, ['outer','inner'])
10000 loops, best of 3: 139 µs per loop

