<jupyter_start><jupyter_text>Categorical dataThis is an introduction to pandas categorical data type, including a short comparisonwith R’s `factor`.`Categoricals` are a pandas data type corresponding to categorical variables instatistics. A categorical variable takes on a limited, and usually fixed,number of possible values (`categories`; `levels` in R). Examples are gender,social class, blood type, country affiliation, observation time or rating viaLikert scales.In contrast to statistical categorical variables, categorical data might have an order (e.g.‘strongly agree’ vs ‘agree’ or ‘first observation’ vs. ‘second observation’), but numericaloperations (additions, divisions, …) are not possible.All values of categorical data are either in `categories` or `np.nan`. Order is defined bythe order of `categories`, not lexical order of the values. Internally, the data structureconsists of a `categories` array and an integer array of `codes` which point to the real value inthe `categories` array.The categorical data type is useful in the following cases:- A string variable consisting of only a few different values. Converting such a string  variable to a categorical variable will save some memory, see here.  - The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”).  By converting to a categorical and specifying an order on the categories, sorting and  min/max will use the logical order instead of the lexical order, see here.  - As a signal to other Python libraries that this column should be treated as a categorical  variable (e.g. to use suitable statistical methods or plot types).  See also the [API docs on categoricals](../05_reference/14_Pandas_Arrays.ipynbapi-arrays-categorical).  Object creation  Series creationCategorical `Series` or columns in a `DataFrame` can be created in several ways:By specifying `dtype="category"` when constructing a `Series`:<jupyter_code>s = pd.Series(["a", "b", "c", "a"], dtype="category")
s<jupyter_output><empty_output><jupyter_text>By converting an existing `Series` or column to a `category` dtype:<jupyter_code>df = pd.DataFrame({"A": ["a", "b", "c", "a"]})
df["B"] = df["A"].astype("category")
df<jupyter_output><empty_output><jupyter_text>By using special functions, such as `cut()`, which groups data intodiscrete bins. See the example on tiling in the docs.<jupyter_code>df = pd.DataFrame({"value": np.random.randint(0, 100, 20)})
labels = ["{0} - {1}".format(i, i + 9) for i in range(0, 100, 10)]

df["group"] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)
df.head(10)<jupyter_output><empty_output><jupyter_text>By passing a `pandas.Categorical` object to a `Series` or assigning it to a `DataFrame`.<jupyter_code>raw_cat = pd.Categorical(
    ["a", "b", "c", "a"], categories=["b", "c", "d"], ordered=False
)
s = pd.Series(raw_cat)
s
df = pd.DataFrame({"A": ["a", "b", "c", "a"]})
df["B"] = raw_cat
df<jupyter_output><empty_output><jupyter_text>Categorical data has a specific `category` dtype:<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>DataFrame creationSimilar to the previous section where a single column was converted to categorical, all columns in a`DataFrame` can be batch converted to categorical either during or after construction.This can be done during construction by specifying `dtype="category"` in the `DataFrame` constructor:<jupyter_code>df = pd.DataFrame({"A": list("abca"), "B": list("bccd")}, dtype="category")
df.dtypes<jupyter_output><empty_output><jupyter_text>Note that the categories present in each column differ; the conversion is done column by column, soonly labels present in a given column are categories:<jupyter_code>df["A"]
df["B"]<jupyter_output><empty_output><jupyter_text>Analogously, all columns in an existing `DataFrame` can be batch converted using `DataFrame.astype()`:<jupyter_code>df = pd.DataFrame({"A": list("abca"), "B": list("bccd")})
df_cat = df.astype("category")
df_cat.dtypes<jupyter_output><empty_output><jupyter_text>This conversion is likewise done column by column:<jupyter_code>df_cat["A"]
df_cat["B"]<jupyter_output><empty_output><jupyter_text>Controlling behaviorIn the examples above where we passed `dtype='category'`, we used the defaultbehavior:1. Categories are inferred from the data.  1. Categories are unordered.  To control those behaviors, instead of passing `'category'`, use an instanceof `CategoricalDtype`.<jupyter_code>from pandas.api.types import CategoricalDtype

s = pd.Series(["a", "b", "c", "a"])
cat_type = CategoricalDtype(categories=["b", "c", "d"], ordered=True)
s_cat = s.astype(cat_type)
s_cat<jupyter_output><empty_output><jupyter_text>Similarly, a `CategoricalDtype` can be used with a `DataFrame` to ensure that categoriesare consistent among all columns.<jupyter_code>from pandas.api.types import CategoricalDtype

df = pd.DataFrame({"A": list("abca"), "B": list("bccd")})
cat_type = CategoricalDtype(categories=list("abcd"), ordered=True)
df_cat = df.astype(cat_type)
df_cat["A"]
df_cat["B"]<jupyter_output><empty_output><jupyter_text>>**Note**>>To perform table-wise conversion, where all labels in the entire `DataFrame` are used ascategories for each column, the `categories` parameter can be determined programmatically by`categories = pd.unique(df.to_numpy().ravel())`.If you already have `codes` and `categories`, you can use the`from_codes()` constructor to save the factorize stepduring normal constructor mode:<jupyter_code>splitter = np.random.choice([0, 1], 5, p=[0.5, 0.5])
s = pd.Series(pd.Categorical.from_codes(splitter, categories=["train", "test"]))<jupyter_output><empty_output><jupyter_text>Regaining original dataTo get back to the original `Series` or NumPy array, use`Series.astype(original_dtype)` or `np.asarray(categorical)`:<jupyter_code>s = pd.Series(["a", "b", "c", "a"])
s
s2 = s.astype("category")
s2
s2.astype(str)
np.asarray(s2)<jupyter_output><empty_output><jupyter_text>>**Note**>>In contrast to R’s `factor` function, categorical data is not converting input values tostrings; categories will end up the same data type as the original values.>**Note**>>In contrast to R’s `factor` function, there is currently no way to assign/change labels atcreation time. Use `categories` to change the categories after creation time.  CategoricalDtypeA categorical’s type is fully described by1. `categories`: a sequence of unique values and no missing values  1. `ordered`: a boolean  This information can be stored in a `CategoricalDtype`.The `categories` argument is optional, which implies that the actual categoriesshould be inferred from whatever is present in the data when the`pandas.Categorical` is created. The categories are assumed to be unorderedby default.<jupyter_code>from pandas.api.types import CategoricalDtype

CategoricalDtype(["a", "b", "c"])
CategoricalDtype(["a", "b", "c"], ordered=True)
CategoricalDtype()<jupyter_output><empty_output><jupyter_text>A `CategoricalDtype` can be used in any place pandasexpects a `dtype`. For example `pandas.read_csv()`,`pandas.DataFrame.astype()`, or in the `Series` constructor.>**Note**>>As a convenience, you can use the string `'category'` in place of a`CategoricalDtype` when you want the default behavior ofthe categories being unordered, and equal to the set values present in thearray. In other words, `dtype='category'` is equivalent to`dtype=CategoricalDtype()`.  Equality semanticsTwo instances of `CategoricalDtype` compare equalwhenever they have the same categories and order. When comparing twounordered categoricals, the order of the `categories` is not considered.<jupyter_code>c1 = CategoricalDtype(["a", "b", "c"], ordered=False)

# Equal, since order is not considered when ordered=False
c1 == CategoricalDtype(["b", "c", "a"], ordered=False)
# Unequal, since the second CategoricalDtype is ordered
c1 == CategoricalDtype(["a", "b", "c"], ordered=True)<jupyter_output><empty_output><jupyter_text>All instances of `CategoricalDtype` compare equal to the string `'category'`.<jupyter_code>c1 == "category"<jupyter_output><empty_output><jupyter_text>Since `dtype='category'` is essentially `CategoricalDtype(None, False)`,and since all instances `CategoricalDtype` compare equal to `'category'`,all instances of `CategoricalDtype` compare equal to a`CategoricalDtype(None, False)`, regardless of `categories` or`ordered`.  DescriptionUsing `describe()` on categorical data will produce similaroutput to a `Series` or `DataFrame` of type `string`.<jupyter_code>cat = pd.Categorical(["a", "c", "c", np.nan], categories=["b", "a", "c"])
df = pd.DataFrame({"cat": cat, "s": ["a", "c", "c", np.nan]})
df.describe()
df["cat"].describe()<jupyter_output><empty_output><jupyter_text>Working with categoriesCategorical data has a `categories` and a `ordered` property, which list theirpossible values and whether the ordering matters or not. These properties areexposed as `s.cat.categories` and `s.cat.ordered`. If you don’t manuallyspecify categories and ordering, they are inferred from the passed arguments.<jupyter_code>s = pd.Series(["a", "b", "c", "a"], dtype="category")
s.cat.categories
s.cat.ordered<jupyter_output><empty_output><jupyter_text>It’s also possible to pass in the categories in a specific order:<jupyter_code>s = pd.Series(pd.Categorical(["a", "b", "c", "a"], categories=["c", "b", "a"]))
s.cat.categories
s.cat.ordered<jupyter_output><empty_output><jupyter_text>>**Note**>>New categorical data are **not** automatically ordered. You must explicitlypass `ordered=True` to indicate an ordered `Categorical`.>**Note**>>The result of `unique()` is not always the same as `Series.cat.categories`,because `Series.unique()` has a couple of guarantees, namely that it returns categoriesin the order of appearance, and it only includes values that are actually present.<jupyter_code>s = pd.Series(list("babc")).astype(CategoricalDtype(list("abcd")))
s
# categories
s.cat.categories
# uniques
s.unique()<jupyter_output><empty_output><jupyter_text>Renaming categoriesRenaming categories is done by assigning new values to the`Series.cat.categories` property or by using the`rename_categories()` method:<jupyter_code>s = pd.Series(["a", "b", "c", "a"], dtype="category")
s
s.cat.categories = ["Group %s" % g for g in s.cat.categories]
s
s = s.cat.rename_categories([1, 2, 3])
s
# You can also pass a dict-like object to map the renaming
s = s.cat.rename_categories({1: "x", 2: "y", 3: "z"})
s<jupyter_output><empty_output><jupyter_text>>**Note**>>In contrast to R’s `factor`, categorical data can have categories of other types than string.>**Note**>>Be aware that assigning new categories is an inplace operation, while most other operationsunder `Series.cat` per default return a new `Series` of dtype `category`.Categories must be unique or a `ValueError` is raised:<jupyter_code>try:
    s.cat.categories = [1, 1, 1]
except ValueError as e:
    print("ValueError:", str(e))<jupyter_output><empty_output><jupyter_text>Categories must also not be `NaN` or a `ValueError` is raised:<jupyter_code>try:
    s.cat.categories = [1, 2, np.nan]
except ValueError as e:
    print("ValueError:", str(e))<jupyter_output><empty_output><jupyter_text>Appending new categoriesAppending categories can be done by using the`add_categories()` method:<jupyter_code>s = s.cat.add_categories([4])
s.cat.categories
s<jupyter_output><empty_output><jupyter_text>Removing categoriesRemoving categories can be done by using the`remove_categories()` method. Values which are removedare replaced by `np.nan`.:<jupyter_code>s = s.cat.remove_categories([4])
s<jupyter_output><empty_output><jupyter_text>Removing unused categoriesRemoving unused categories can also be done:<jupyter_code>s = pd.Series(pd.Categorical(["a", "b", "a"], categories=["a", "b", "c", "d"]))
s
s.cat.remove_unused_categories()<jupyter_output><empty_output><jupyter_text>Setting categoriesIf you want to do remove and add new categories in one step (which has somespeed advantage), or simply set the categories to a predefined scale,use `set_categories()`.<jupyter_code>s = pd.Series(["one", "two", "four", "-"], dtype="category")
s
s = s.cat.set_categories(["one", "two", "three", "four"])
s<jupyter_output><empty_output><jupyter_text>>**Note**>>Be aware that `Categorical.set_categories()` cannot know whether some category is omittedintentionally or because it is misspelled or (under Python3) due to a type difference (e.g.,NumPy S1 dtype and Python strings). This can result in surprising behaviour!  Sorting and orderIf categorical data is ordered (`s.cat.ordered == True`), then the order of the categories has ameaning and certain operations are possible. If the categorical is unordered, `.min()/.max()` will raise a `TypeError`.<jupyter_code>s = pd.Series(pd.Categorical(["a", "b", "c", "a"], ordered=False))
s.sort_values(inplace=True)
s = pd.Series(["a", "b", "c", "a"]).astype(CategoricalDtype(ordered=True))
s.sort_values(inplace=True)
s
s.min(), s.max()<jupyter_output><empty_output><jupyter_text>You can set categorical data to be ordered by using `as_ordered()` or unordered by using `as_unordered()`. These will bydefault return a *new* object.<jupyter_code>s.cat.as_ordered()
s.cat.as_unordered()<jupyter_output><empty_output><jupyter_text>Sorting will use the order defined by categories, not any lexical order present on the data type.This is even true for strings and numeric data:<jupyter_code>s = pd.Series([1, 2, 3, 1], dtype="category")
s = s.cat.set_categories([2, 3, 1], ordered=True)
s
s.sort_values(inplace=True)
s
s.min(), s.max()<jupyter_output><empty_output><jupyter_text>ReorderingReordering the categories is possible via the `Categorical.reorder_categories()` andthe `Categorical.set_categories()` methods. For `Categorical.reorder_categories()`, allold categories must be included in the new categories and no new categories are allowed. This willnecessarily make the sort order the same as the categories order.<jupyter_code>s = pd.Series([1, 2, 3, 1], dtype="category")
s = s.cat.reorder_categories([2, 3, 1], ordered=True)
s
s.sort_values(inplace=True)
s
s.min(), s.max()<jupyter_output><empty_output><jupyter_text>>**Note**>>Note the difference between assigning new categories and reordering the categories: the firstrenames categories and therefore the individual values in the `Series`, but if the firstposition was sorted last, the renamed value will still be sorted last. Reordering means that theway values are sorted is different afterwards, but not that individual values in the`Series` are changed.>**Note**>>If the `Categorical` is not ordered, `Series.min()` and `Series.max()` will raise`TypeError`. Numeric operations like `+`, `-`, `*`, `/` and operations based on them(e.g. `Series.median()`, which would need to compute the mean between two values if the lengthof an array is even) do not work and raise a `TypeError`.  Multi column sortingA categorical dtyped column will participate in a multi-column sort in a similar manner to other columns.The ordering of the categorical is determined by the `categories` of that column.<jupyter_code>dfs = pd.DataFrame(
    {
        "A": pd.Categorical(
            list("bbeebbaa"),
            categories=["e", "a", "b"],
            ordered=True,
        ),
        "B": [1, 2, 1, 2, 2, 1, 2, 1],
    }
)
dfs.sort_values(by=["A", "B"])<jupyter_output><empty_output><jupyter_text>Reordering the `categories` changes a future sort.<jupyter_code>dfs["A"] = dfs["A"].cat.reorder_categories(["a", "b", "e"])
dfs.sort_values(by=["A", "B"])<jupyter_output><empty_output><jupyter_text>ComparisonsComparing categorical data with other objects is possible in three cases:- Comparing equality (`==` and `!=`) to a list-like object (list, Series, array,  …) of the same length as the categorical data.  - All comparisons (`==`, `!=`, `>`, `>=`, `<`, and `<=`) of categorical data to  another categorical Series, when `ordered==True` and the `categories` are the same.  - All comparisons of a categorical data to a scalar.  All other comparisons, especially “non-equality” comparisons of two categoricals with differentcategories or a categorical with any list-like object, will raise a `TypeError`.>**Note**>>Any “non-equality” comparisons of categorical data with a `Series`, `np.array`, `list` orcategorical data with different categories or ordering will raise a `TypeError` because customcategories ordering could be interpreted in two ways: one with taking into account theordering and one without.<jupyter_code>cat = pd.Series([1, 2, 3]).astype(CategoricalDtype([3, 2, 1], ordered=True))
cat_base = pd.Series([2, 2, 2]).astype(CategoricalDtype([3, 2, 1], ordered=True))
cat_base2 = pd.Series([2, 2, 2]).astype(CategoricalDtype(ordered=True))

cat
cat_base
cat_base2<jupyter_output><empty_output><jupyter_text>Comparing to a categorical with the same categories and ordering or to a scalar works:<jupyter_code>cat > cat_base
cat > 2<jupyter_output><empty_output><jupyter_text>Equality comparisons work with any list-like object of same length and scalars:<jupyter_code>cat == cat_base
cat == np.array([1, 2, 3])
cat == 2<jupyter_output><empty_output><jupyter_text>This doesn’t work because the categories are not the same:<jupyter_code>try:
    cat > cat_base2
except TypeError as e:
    print("TypeError:", str(e))<jupyter_output><empty_output><jupyter_text>If you want to do a “non-equality” comparison of a categorical series with a list-like objectwhich is not categorical data, you need to be explicit and convert the categorical data back tothe original values:<jupyter_code>base = np.array([1, 2, 3])

try:
    cat > base
except TypeError as e:
    print("TypeError:", str(e))

np.asarray(cat) > base<jupyter_output><empty_output><jupyter_text>When you compare two unordered categoricals with the same categories, the order is not considered:<jupyter_code>c1 = pd.Categorical(["a", "b"], categories=["a", "b"], ordered=False)
c2 = pd.Categorical(["a", "b"], categories=["b", "a"], ordered=False)
c1 == c2<jupyter_output><empty_output><jupyter_text>OperationsApart from `Series.min()`, `Series.max()` and `Series.mode()`, thefollowing operations are possible with categorical data:`Series` methods like `Series.value_counts()` will use all categories,even if some categories are not present in the data:<jupyter_code>s = pd.Series(pd.Categorical(["a", "b", "c", "c"], categories=["c", "a", "b", "d"]))
s.value_counts()<jupyter_output><empty_output><jupyter_text>`DataFrame` methods like `DataFrame.sum()` also show “unused” categories.<jupyter_code>columns = pd.Categorical(
    ["One", "One", "Two"], categories=["One", "Two", "Three"], ordered=True
)
df = pd.DataFrame(
    data=[[1, 2, 3], [4, 5, 6]],
    columns=pd.MultiIndex.from_arrays([["A", "B", "B"], columns]),
)
df.groupby(axis=1, level=1).sum()<jupyter_output><empty_output><jupyter_text>Groupby will also show “unused” categories:<jupyter_code>cats = pd.Categorical(
    ["a", "b", "b", "b", "c", "c", "c"], categories=["a", "b", "c", "d"]
)
df = pd.DataFrame({"cats": cats, "values": [1, 2, 2, 2, 3, 4, 5]})
df.groupby("cats").mean()
cats2 = pd.Categorical(["a", "a", "b", "b"], categories=["a", "b", "c"])
df2 = pd.DataFrame(
    {
        "cats": cats2,
        "B": ["c", "d", "c", "d"],
        "values": [1, 2, 3, 4],
    }
)
df2.groupby(["cats", "B"]).mean()<jupyter_output><empty_output><jupyter_text>Pivot tables:<jupyter_code>raw_cat = pd.Categorical(["a", "a", "b", "b"], categories=["a", "b", "c"])
df = pd.DataFrame({"A": raw_cat, "B": ["c", "d", "c", "d"], "values": [1, 2, 3, 4]})
pd.pivot_table(df, values="values", index=["A", "B"])<jupyter_output><empty_output><jupyter_text>Data mungingThe optimized pandas data access methods  `.loc`, `.iloc`, `.at`, and `.iat`,work as normal. The only difference is the return type (for getting) andthat only values already in `categories` can be assigned.  GettingIf the slicing operation returns either a `DataFrame` or a column of type`Series`, the `category` dtype is preserved.<jupyter_code>idx = pd.Index(["h", "i", "j", "k", "l", "m", "n"])
cats = pd.Series(["a", "b", "b", "b", "c", "c", "c"], dtype="category", index=idx)
values = [1, 2, 2, 2, 3, 4, 5]
df = pd.DataFrame({"cats": cats, "values": values}, index=idx)
df.iloc[2:4, :]
df.iloc[2:4, :].dtypes
df.loc["h":"j", "cats"]
df[df["cats"] == "b"]<jupyter_output><empty_output><jupyter_text>An example where the category type is not preserved is if you take one singlerow: the resulting `Series` is of dtype `object`:<jupyter_code># get the complete "h" row as a Series
df.loc["h", :]<jupyter_output><empty_output><jupyter_text>Returning a single item from categorical data will also return the value, not a categoricalof length “1”.<jupyter_code>df.iat[0, 0]
df["cats"].cat.categories = ["x", "y", "z"]
df.at["h", "cats"]  # returns a string<jupyter_output><empty_output><jupyter_text>>**Note**>>The is in contrast to R’s `factor` function, where `factor(c(1,2,3))[1]`returns a single value `factor`.To get a single value `Series` of type `category`, you pass in a list witha single value:<jupyter_code>df.loc[["h"], "cats"]<jupyter_output><empty_output><jupyter_text>String and datetime accessorsThe accessors  `.dt` and `.str` will work if the `s.cat.categories` are ofan appropriate type:<jupyter_code>str_s = pd.Series(list("aabb"))
str_cat = str_s.astype("category")
str_cat
str_cat.str.contains("a")
date_s = pd.Series(pd.date_range("1/1/2015", periods=5))
date_cat = date_s.astype("category")
date_cat
date_cat.dt.day<jupyter_output><empty_output><jupyter_text>>**Note**>>The returned `Series` (or `DataFrame`) is of the same type as if you used the`.str.` / `.dt.` on a `Series` of that type (and not oftype `category`!).That means, that the returned values from methods and properties on the accessors of a`Series` and the returned values from methods and properties on the accessors of this`Series` transformed to one of type `category` will be equal:<jupyter_code>ret_s = str_s.str.contains("a")
ret_cat = str_cat.str.contains("a")
ret_s.dtype == ret_cat.dtype
ret_s == ret_cat<jupyter_output><empty_output><jupyter_text>>**Note**>>The work is done on the `categories` and then a new `Series` is constructed. This hassome performance implication if you have a `Series` of type string, where lots of elementsare repeated (i.e. the number of unique elements in the `Series` is a lot smaller than thelength of the `Series`). In this case it can be faster to convert the original `Series`to one of type `category` and use `.str.` or `.dt.` on that.  SettingSetting values in a categorical column (or `Series`) works as long as thevalue is included in the `categories`:<jupyter_code>idx = pd.Index(["h", "i", "j", "k", "l", "m", "n"])
cats = pd.Categorical(["a", "a", "a", "a", "a", "a", "a"], categories=["a", "b"])
values = [1, 1, 1, 1, 1, 1, 1]
df = pd.DataFrame({"cats": cats, "values": values}, index=idx)

df.iloc[2:4, :] = [["b", 2], ["b", 2]]
df
try:
    df.iloc[2:4, :] = [["c", 3], ["c", 3]]
except TypeError as e:
    print("TypeError:", str(e))
except ValueError as e:
    print("ValueError:", str(e))<jupyter_output><empty_output><jupyter_text>Setting values by assigning categorical data will also check that the `categories` match:<jupyter_code>df.loc["j":"k", "cats"] = pd.Categorical(["a", "a"], categories=["a", "b"])
df
try:
    df.loc["j":"k", "cats"] = pd.Categorical(["b", "b"], categories=["a", "b", "c"])
except TypeError as e:
    print("TypeError:", str(e))
except ValueError as e:
    print("ValueError:", str(e))<jupyter_output><empty_output><jupyter_text>Assigning a `Categorical` to parts of a column of other types will use the values:<jupyter_code>df = pd.DataFrame({"a": [1, 1, 1, 1, 1], "b": ["a", "a", "a", "a", "a"]})
df.loc[1:2, "a"] = pd.Categorical(["b", "b"], categories=["a", "b"])
df.loc[2:3, "b"] = pd.Categorical(["b", "b"], categories=["a", "b"])
df
df.dtypes<jupyter_output><empty_output><jupyter_text>Merging / concatenationBy default, combining `Series` or `DataFrames` which contain the samecategories results in `category` dtype, otherwise results will depend on thedtype of the underlying categories. Merges that result in non-categoricaldtypes will likely have higher memory usage. Use `.astype` or`union_categoricals` to ensure `category` results.<jupyter_code>from pandas.api.types import union_categoricals

# same categories
s1 = pd.Series(["a", "b"], dtype="category")
s2 = pd.Series(["a", "b", "a"], dtype="category")
pd.concat([s1, s2])
# different categories
s3 = pd.Series(["b", "c"], dtype="category")
pd.concat([s1, s3])
# Output dtype is inferred based on categories values
int_cats = pd.Series([1, 2], dtype="category")
float_cats = pd.Series([3.0, 4.0], dtype="category")
pd.concat([int_cats, float_cats])
pd.concat([s1, s3]).astype("category")
union_categoricals([s1.array, s3.array])<jupyter_output><empty_output><jupyter_text>The following table summarizes the results of merging `Categoricals`:|arg1|arg2|identical|result||:-----------------|:----------------------|:--------------------|:---------------------------||category|category|True|category||category (object)|category (object)|False|object (dtype is inferred)||category (int)|category (float)|False|float (dtype is inferred)|See also the section on [merge dtypes](25_merging_join_concat.ipynbmerging-dtypes) for notes aboutpreserving merge dtypes and performance.  UnioningIf you want to combine categoricals that do not necessarily have the samecategories, the `union_categoricals()` function willcombine a list-like of categoricals. The new categories will be the union ofthe categories being combined.<jupyter_code>from pandas.api.types import union_categoricals

a = pd.Categorical(["b", "c"])
b = pd.Categorical(["a", "b"])
union_categoricals([a, b])<jupyter_output><empty_output><jupyter_text>By default, the resulting categories will be ordered asthey appear in the data. If you want the categories tobe lexsorted, use `sort_categories=True` argument.<jupyter_code>union_categoricals([a, b], sort_categories=True)<jupyter_output><empty_output><jupyter_text>`union_categoricals` also works with the “easy” case of combining twocategoricals of the same categories and order information(e.g. what you could also `append` for).<jupyter_code>a = pd.Categorical(["a", "b"], ordered=True)
b = pd.Categorical(["a", "b", "a"], ordered=True)
union_categoricals([a, b])<jupyter_output><empty_output><jupyter_text>The below raises `TypeError` because the categories are ordered and not identical. ```pythonIn [1]: a = pd.Categorical(["a", "b"], ordered=True)In [2]: b = pd.Categorical(["a", "b", "c"], ordered=True)In [3]: union_categoricals([a, b])Out[3]:TypeError: to union ordered Categoricals, all categories must be the same``` Ordered categoricals with different categories or orderings can be combined byusing the `ignore_ordered=True` argument.<jupyter_code>a = pd.Categorical(["a", "b", "c"], ordered=True)
b = pd.Categorical(["c", "b", "a"], ordered=True)
union_categoricals([a, b], ignore_order=True)<jupyter_output><empty_output><jupyter_text>`union_categoricals()` also works with a`CategoricalIndex`, or `Series` containing categorical data, but note thatthe resulting array will always be a plain `Categorical`:<jupyter_code>a = pd.Series(["b", "c"], dtype="category")
b = pd.Series(["a", "b"], dtype="category")
union_categoricals([a, b])<jupyter_output><empty_output><jupyter_text>>**Note**>>`union_categoricals` may recode the integer codes for categorieswhen combining categoricals.  This is likely what you want,but if you are relying on the exact numbering of the categories, beaware.<jupyter_code>c1 = pd.Categorical(["b", "c"])
c2 = pd.Categorical(["a", "b"])

c1
# "b" is coded to 0
c1.codes
c2
# "b" is coded to 1
c2.codes
c = union_categoricals([c1, c2])
c
# "b" is coded to 0 throughout, same as c1, different from c2
c.codes<jupyter_output><empty_output><jupyter_text>Getting data in/outYou can write data that contains `category` dtypes to a `HDFStore`.See here for an example and caveats.It is also possible to write data to and reading data from *Stata* format files.See here for an example and caveats.Writing to a CSV file will convert the data, effectively removing any information about thecategorical (categories and ordering). So if you read back the CSV file you have to convert therelevant columns back to `category` and assign the right categories and categories ordering.<jupyter_code>import io

s = pd.Series(pd.Categorical(["a", "b", "b", "a", "a", "d"]))
# rename the categories
s.cat.categories = ["very good", "good", "bad"]
# reorder the categories and add missing categories
s = s.cat.set_categories(["very bad", "bad", "medium", "good", "very good"])
df = pd.DataFrame({"cats": s, "vals": [1, 2, 3, 4, 5, 6]})
csv = io.StringIO()
df.to_csv(csv)
df2 = pd.read_csv(io.StringIO(csv.getvalue()))
df2.dtypes
df2["cats"]
# Redo the category
df2["cats"] = df2["cats"].astype("category")
df2["cats"].cat.set_categories(
    ["very bad", "bad", "medium", "good", "very good"], inplace=True
)
df2.dtypes
df2["cats"]<jupyter_output><empty_output><jupyter_text>The same holds for writing to a SQL database with `to_sql`.  Missing datapandas primarily uses the value `np.nan` to represent missing data. It is bydefault not included in computations. See the [Missing Data section](28_Working_with_missing_data.ipynbworking-with-missing-data).Missing values should **not** be included in the Categorical’s `categories`,only in the `values`.Instead, it is understood that NaN is different, and is always a possibility.When working with the Categorical’s `codes`, missing values will always havea code of `-1`.<jupyter_code>s = pd.Series(["a", "b", np.nan, "a"], dtype="category")
# only two categories
s
s.cat.codes<jupyter_output><empty_output><jupyter_text>Methods for working with missing data, e.g. `isna()`, `fillna()`,`dropna()`, all work normally:<jupyter_code>s = pd.Series(["a", "b", np.nan], dtype="category")
s
pd.isna(s)
s.fillna("a")<jupyter_output><empty_output><jupyter_text>Differences to R’s `factor`The following differences to R’s factor functions can be observed:- R’s `levels` are named `categories`.  - R’s `levels` are always of type string, while `categories` in pandas can be of any dtype.  - It’s not possible to specify labels at creation time. Use `s.cat.rename_categories(new_labels)`  afterwards.  - In contrast to R’s `factor` function, using categorical data as the sole input to create a  new categorical series will *not* remove unused categories but create a new categorical series  which is equal to the passed in one!  - R allows for missing values to be included in its `levels` (pandas’ `categories`). pandas  does not allow `NaN` categories, but missing values can still be in the `values`.    Gotchas  Memory usageThe memory usage of a `Categorical` is proportional to the number of categories plus the length of the data. In contrast,an `object` dtype is a constant times the length of the data.<jupyter_code>s = pd.Series(["foo", "bar"] * 1000)

# object dtype
s.nbytes
# category dtype
s.astype("category").nbytes<jupyter_output><empty_output><jupyter_text>>**Note**>>If the number of categories approaches the length of the data, the `Categorical` will use nearly the same ormore memory than an equivalent `object` dtype representation.<jupyter_code>s = pd.Series(["foo%04d" % i for i in range(2000)])

# object dtype
s.nbytes
# category dtype
s.astype("category").nbytes<jupyter_output><empty_output><jupyter_text>`Categorical` is not a `numpy` arrayCurrently, categorical data and the underlying `Categorical` is implemented as a Pythonobject and not as a low-level NumPy array dtype. This leads to some problems.NumPy itself doesn’t know about the new `dtype`:<jupyter_code>try:
    np.dtype("category")
except TypeError as e:
    print("TypeError:", str(e))

dtype = pd.Categorical(["a"]).dtype
try:
    np.dtype(dtype)
except TypeError as e:
    print("TypeError:", str(e))<jupyter_output><empty_output><jupyter_text>Dtype comparisons work:<jupyter_code>dtype == np.str_
np.str_ == dtype<jupyter_output><empty_output><jupyter_text>To check if a Series contains Categorical data, use `hasattr(s, 'cat')`:<jupyter_code>hasattr(pd.Series(["a"], dtype="category"), "cat")
hasattr(pd.Series(["a"]), "cat")<jupyter_output><empty_output><jupyter_text>Using NumPy functions on a `Series` of type `category` should not work as `Categoricals`are not numeric data (even in the case that `.categories` is numeric).<jupyter_code>s = pd.Series(pd.Categorical([1, 2, 3, 4]))
try:
    np.sum(s)
    # same with np.log(s),...
except TypeError as e:
    print("TypeError:", str(e))<jupyter_output><empty_output><jupyter_text>>**Note**>>If such a function works, please file a bug at [https://github.com/pandas-dev/pandas](https://github.com/pandas-dev/pandas)!  dtype in applypandas currently does not preserve the dtype in apply functions: If you apply along rows you geta `Series` of `object` `dtype` (same as getting a row -> getting one element will return abasic type) and applying along columns will also convert to object. `NaN` values are unaffected.You can use `fillna` to handle missing values before applying a function.<jupyter_code>df = pd.DataFrame(
    {
        "a": [1, 2, 3, 4],
        "b": ["a", "b", "c", "d"],
        "cats": pd.Categorical([1, 2, 3, 2]),
    }
)
df.apply(lambda row: type(row["cats"]), axis=1)
df.apply(lambda col: col.dtype, axis=0)<jupyter_output><empty_output><jupyter_text>Categorical index`CategoricalIndex` is a type of index that is useful for supportingindexing with duplicates. This is a container around a `Categorical`and allows efficient indexing and storage of an index with a large number of duplicated elements.See the advanced indexing docs for a more detailedexplanation.Setting the index will create a `CategoricalIndex`:<jupyter_code>cats = pd.Categorical([1, 2, 3, 4], categories=[4, 2, 3, 1])
strings = ["a", "b", "c", "d"]
values = [4, 2, 3, 1]
df = pd.DataFrame({"strings": strings, "values": values}, index=cats)
df.index
# This now sorts by the categories order
df.sort_index()<jupyter_output><empty_output><jupyter_text>Side effectsConstructing a `Series` from a `Categorical` will not copy the input`Categorical`. This means that changes to the `Series` will in most caseschange the original `Categorical`:<jupyter_code>cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])
s = pd.Series(cat, name="cat")
cat
s.iloc[0:2] = 10
cat
df = pd.DataFrame(s)
df["cat"].cat.categories = [1, 2, 3, 4, 5]
cat<jupyter_output><empty_output><jupyter_text>Use `copy=True` to prevent such a behaviour or simply don’t reuse `Categoricals`:<jupyter_code>cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])
s = pd.Series(cat, name="cat", copy=True)
cat
s.iloc[0:2] = 10
cat<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14392721845941914, 'fraction_numerical': 0.009040293881299507, 'mean_word_length': 6.1786155747836835, 'pattern_counts': {'":': 37, '<': 291, '<?xml version=': 0, '>': 351, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '72', 'dir': 'jupyter-structured-clean-dedup', 'id': '93940', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 11637, 'n_tokens_neox': 11273, 'n_words': 3692, 'path': '04_user_guide/30_categorical.ipynb', 'repo_name': 'olippuner/Pandas_Docu_as_Notebooks', 'size': '34844'}