Q:

How do I remove non-ascii characters (e.g Ð±§•¿µ´‡»Ž®ºÏƒ¶¹) from texts in pandas dataframe columns?

How do I remove non-ascii characters (e.g Ð±§•¿µ´‡»Ž®ºÏƒ¶¹) from texts in pandas dataframe columns?
I have tried the following but no luck
df = pd.read_csv(path, index_col=0)
for col in df.columns:
for j in df.index:
    markup1 = str(df.ix[j, col]).replace("\r", "")
    markup1 = markup1.replace("\n", "")
    markup1 = markup1.decode('unicode_escape').encode('ascii','ignore').strip()
soup = BeautifulSoup(markup1, 'lxml')
df.ix[j, col] = soup.get_text()
print df.ix[j, 'requirements']

I tried using regex yet it wouldn't work.
markup1 = str(df.ix[j, 'requirements']).replace("\r", "")
markup1 = markup1.replace("\n", "")
markup1 = re.sub(r'[^\x00-\x7F]+', ' ', markup1)

I still keep getting the non-ascii characters. Any suggestion would be appreciated.
I have added the first three rows of the df below:
                                              col1               col2  \
1.0                          H1B SPONSOR FOR L1/L2/OPT  US, NY, New York
2.0                             Graphic / Web Designer     US, TX, Austin
3.0  Full Stack Developer (.NET or equivalent + Jav...             GR, ,

                col3  col4  \
1.0                  NaN   NaN
2.0  Sales and Marketing   NaN
3.0                  NaN   NaN

                                              col5  \ 
1.0  i28 Technologies has demonstrated expertise in...
2.0  outstanding people who believe that more is po...
3.0                                                NaN

                                              col6  \
1.0  Hello,Wish you are doing good...              ...
2.0  The Graphic / Web Designer will manage, popula...
3.0   You?ll have to join the Moosend dojo. But, yo...

                                              col7  \
1.0  JAVA, .NET, SQL, ORACLE, SAP, Informatica, Big...
2.0  Bachelor?s degree in Graphic Design, Web Desig...
3.0  ? .NET or equivalent (Java etc.)? MVC? Javascr...

                                              col8 col9
1.0                                                NaN    f
2.0  CSD offers a competitive benefits package for ...    f
3.0  You?ll be working with the best team in town.....    f

A:

Option 1 - if you know the complete set of non-ascii characters:
df
Out[36]: 
         col1  col2
0  aaÐ±§•¿µbb  abcd
1         hf4  efgh
2         xxx  ijk9

df.replace(regex=True, to_replace=['Ð', '§', '±'], value='') # incomplete here
Out[37]: 
      col1  col2
0  aa•¿µbb  abcd
1      hf4  efgh
2      xxx  ijk9

Option 2 - if you can't specify the whole set of non-ascii characters:
Consider using string.printable:

String of ASCII characters which are considered printable.

from string import printable

printable
Out[38]: '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c'

df.applymap(lambda y: ''.join(filter(lambda x: 
            x in string.printable, y)))
Out[14]: 
   col1  col2
0  aabb  abcd
1   hf4  asdf
2   xxx      

Note that if an element in the DataFrame is all-non-ascii, it will be replaced with just ''.

