<jupyter_start><jupyter_text>As colunas correspondentes a quantidades começam com `qt_`<jupyter_code>df.nunique()
col_3_20_unique = df.loc[:, df.nunique().between(3,20)].columns
col_3_20_unique
len(col_3_20_unique)<jupyter_output><empty_output><jupyter_text>`Unnamed` é uma lista de inteiros sequencial de 0 a 462 297 (comprimento de df).<jupyter_code>df.iloc[:,0].head()
df.iloc[:,0].tail()<jupyter_output><empty_output><jupyter_text>`id` é uma string com 64 caracteres com valores únicos para cada um dos 462 298 registros.<jupyter_code>df.iloc[:,1].head()
df.iloc[:,1].nunique()<jupyter_output><empty_output><jupyter_text>Como as variáveis `Unnamed` e `id` são únicas, sua função já está contemplada no índice do DataFrame. Por isso, ambas podem ser descartadas.<jupyter_code>df = df.loc[:, ~df.columns.str.contains('^Unnamed')]
df.drop(columns=['id'], inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>`fl_matriz` é uma variável booleana (`True` ou `False`), que é `True` se o CNPJ for da matriz.As variáveis booleanas do dataset iniciam com `fl_` e terminam com uma string com o valor correspondente a `True`.* Observa-se um predomínio de CNPJ da matriz (93%)<jupyter_code>col = 'fl_matriz'

fig = plt.figure()
df[col].value_counts().plot(kind='pie', autopct='%d%%')
plt.ylabel('')


df[col].value_counts()<jupyter_output><empty_output><jupyter_text>`de_natureza_juridica` é uma string que corresponde à natureza jurídica da empresa* há 67 tipos de natureza jurídica no dataset* 5 tipos correspondem a 95% dos dados * EMPRESARIO INDIVIDUAL: 63.98% * SOCIEDADE EMPRESARIA LIMITADA: 19.73% * ASSOCIACAO PRIVADA:	6.42% * EMPRESA INDIVIDUAL DE RESPONSABILIDADE LIMITADA DE NATUREZA EMPRESARIA:	3.28% * ORGAO DE DIRECAO LOCAL DE PARTIDO POLITICO: 1.58%<jupyter_code>def col_value_pct_counts(df,col,n=len(df[col].value_counts())):
  df_col_value_counts = df[col].value_counts().nlargest(n)

  df_col_value_pct_counts = pd.concat([df_col_value_counts,
                                       df[col].value_counts(normalize=True)
                                       .nlargest(n)], axis=1)
  df_col_value_pct_counts.columns = ['Value', 'Pct']
  df_col_value_pct_counts['Value'] = (df_col_value_pct_counts['Value']
                                      .apply(lambda x : "{:,}".format(x)))
  df_col_value_pct_counts['Pct'] = ((df_col_value_pct_counts['Pct'] * 100)
                                     .round(2).astype(str) + '%')
  display(df_col_value_pct_counts)

  fig = plt.figure()
  # Divide the figure into a 1x2 grid, and give me the first section
  ax1 = fig.add_subplot(121)
  # Divide the figure into a 1x2 grid, and give me the second section
  ax2 = fig.add_subplot(122)

  df_col_value_counts.plot(kind='bar', ax=ax1);

  plt2 = df_col_value_counts.plot(kind='pie', autopct='%d%%', ax=ax2);
  ax2.set_ylabel('')

  return df_col_value_counts
col = 'de_natureza_juridica'
col_value_pct_counts(df, col, n=5);
np.cumsum(df[col].value_counts(normalize=True).nlargest(10)).plot()
plt.axhline(y=0.95, color='r', ls='--')
plt.xticks(rotation=90)
plt.show()
df[col].value_counts(normalize=True).nlargest(5).sum().round(2)
len(df[col].value_counts())<jupyter_output><empty_output><jupyter_text>`sg_uf` contém empresas distribuídas em 6 estados: MA, RN, AM, PI, RO, AC.* 2: MA, PI situam-se na Região Nordeste do Brasil.* 4: RN, AM, RO, AC situam-se na Região Norte do Brasil.* Não contém `NaN`.<jupyter_code>col = 'sg_uf'

df_col_value_counts = col_value_pct_counts(df, col)

df[col].isna().sum()<jupyter_output><empty_output><jupyter_text><jupyter_code>df.describe()
def list_df_col_starts_with_str(df, str):
  filter_col = [col for col in df if col.startswith(str)]
  return filter_col

filter_col = list_df_col_starts_with_str(df, str='qt')
start_with = ['idade', 'grau', 'vl_frota', 'vl_total']
[filter_col.extend(list_df_col_starts_with_str(df, str)) for str in start_with]
filter_col<jupyter_output><empty_output><jupyter_text>Essas poderiam ser convertidas de `float` para `int`, contudo o `Pandas` não suporta `int` com `NaN`, por isso adotam-se duas práticas usuais:1. Trabalhar com `float`2. Trabalhar com `Int64`Por simplicidade, optou-se pela 1a.<jupyter_code># filter_col = [df[col].astype('int32').dtypes for col in filter_col]
# filter_col = [df[col].dtypes('Int64') for col in filter_col]
df[filter_]<jupyter_output><empty_output><jupyter_text><jupyter_code>display(df.dtypes)<jupyter_output><empty_output><jupyter_text>0. fl_matriz: boolean value, true if the CNPJ corresponds to the "matriz".1. de_natureza_juridica: character, juridic nature of the company.<jupyter_code>response.content<jupyter_output><empty_output><jupyter_text>Não consegui extrair texto do pdf<jupyter_code>!pip install pdfplumber
import pdfplumber

with BytesIO(response.content) as open_pdf_file:
    pdf = pdfplumber.open(open_pdf_file)
    page = pdf.pages[0]
    text = page.extract_text()
    print(text)
    pdf.close()
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.converter import TextConverter
from pdfminer.layout import LAParams
from pdfminer.pdfpage import PDFPage
from io import StringIO

def convert_pdf_to_txt(path):
    rsrcmgr = PDFResourceManager()
    retstr = StringIO()
    codec = 'utf-8'
    laparams = LAParams()
    device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)
    fp = open(path, 'rb')
    interpreter = PDFPageInterpreter(rsrcmgr, device)
    password = ""
    maxpages = 0
    caching = True
    pagenos=set()

    for page in PDFPage.get_pages(fp, pagenos, maxpages=maxpages, 
                                  password=password,caching=caching,
                                  check_extractable=True):
        interpreter.process_page(page)

    text = retstr.getvalue()

    fp.close()
    device.close()
    retstr.close()
    return text
#!pip install pdfminer.six
from pdfminer.high_level import extract_text



text = extract_text(BytesIO(response.content))
print(text)
!pip install pdfplumber
import pdfplumber

with BytesIO(response.content) as open_pdf_file:
    pdf = pdfplumber.open(open_pdf_file)
    page = pdf.pages[0]
    text = page.extract_text()
    print(text)
    pdf.close()
!pip install tika
from tika import parser

with BytesIO(response.content) as open_pdf_file:
    rawText = parser.from_file(open_pdf_file)
    rawList = rawText['content'].splitlines()
#!pip install PyPDF4==1.27.0
import PyPDF4

with BytesIO(response.content) as open_pdf_file:
    read_pdf = PyPDF4.PdfFileReader(open_pdf_file)
    num_pages = read_pdf.getNumPages()
    print(num_pages)
    text_list=[]
    for i in range(num_pages):
      page = read_pdf.getPage(i)
      page_content = page.extractText()
      text_list.append(page_content)


print(page_content)
print(read_pdf)
print(text_list)
# merge elements of a list to one single string
text = ''.join(text_list[:])

# eliminates \n characters (pdf text break)
test_str = text.replace('\n', '')
text
df.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1218607305936073, 'fraction_numerical': 0.015839041095890412, 'mean_word_length': 4.88003355704698, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '129304', 'license': 'MIT', 'n_tokens_mistral': 2731, 'n_tokens_neox': 2622, 'n_words': 553, 'path': 'GoogleColab/ProjetoAceleraDev.ipynb', 'repo_name': 'viniciusriosfuck/dscodenation', 'size': '7008'}