task_categories:text-generation license:mit region:us
---
dataset_info:
  features:
  - name: hexsha
    dtype: string
  - name: repo
    dtype: string
  - name: path
    dtype: string
  - name: license
    sequence: string
  - name: language
    dtype: string
  - name: identifier
    dtype: string
  - name: return_type
    dtype: string
  - name: original_string
    dtype: string
  - name: original_docstring
    dtype: string
  - name: docstring
    dtype: string
  - name: docstring_tokens
    sequence: string
  - name: code
    dtype: string
  - name: code_tokens
    sequence: string
  - name: short_docstring
    dtype: string
  - name: short_docstring_tokens
    sequence: string
  - name: comment
    sequence: string
  - name: parameters
    list:
    - name: param
      dtype: string
    - name: type
      dtype: string
  - name: docstring_params
    struct:
    - name: returns
      list:
      - name: docstring
        dtype: string
      - name: docstring_tokens
        sequence: string
      - name: type
        dtype: string
    - name: raises
      list:
      - name: docstring
        dtype: string
      - name: docstring_tokens
        sequence: string
      - name: type
        dtype: string
    - name: params
      list:
      - name: identifier
        dtype: string
      - name: type
        dtype: string
      - name: docstring
        dtype: string
      - name: docstring_tokens
        sequence: string
      - name: default
        dtype: string
      - name: is_optional
        dtype: bool
    - name: outlier_params
      list:
      - name: identifier
        dtype: string
      - name: type
        dtype: string
      - name: docstring
        dtype: string
      - name: docstring_tokens
        sequence: string
      - name: default
        dtype: string
      - name: is_optional
        dtype: bool
    - name: others
      list:
      - name: identifier
        dtype: string
      - name: docstring
        dtype: string
      - name: docstring_tokens
        sequence: string
  - name: code_with_imports
    dtype: string
  - name: idxs
    dtype: int64
  - name: cluster
    dtype: int64
  splits:
  - name: train
    num_bytes: 1555988881.6663418
    num_examples: 544627
  download_size: 773215769
  dataset_size: 1555988881.6663418
license: mit
task_categories:
- text-generation
---
# Dataset Card for "thevault-filtered"

Filtered version of [The Vault (function)](https://huggingface.co/datasets/Fsoft-AIC/the-vault-function). Restricted only to Python, then:
- Light AST filtering for self-contained functions
- Run through CodeBERT embeddings, clustered with k-means to 1024 clusters, and then the clusters were manually skimmed for seemingly uninformative functions.

The clusters excluded and their reasons are as follows:
```
  excluded = [
    4, # biochem stuff? DEcompiled code
    9, # Empty functions
    33, # Empty functions
    34, # UI stuff, just returns arguments
    37, # Empty functions
    40, # Empty functions
    42, # Empty functions
    44, # _namespace_SIO stuff
    55, # Trivial, e.g. add(a, b) = a + b
    66, # find_by class methods
    67, # Mostly methods, seems not very informative
    77, # openapi_types, returns a fixed dictionary
    78, # Minimal, method stuff
    83, # Locale configuration
    87, # Just returns argument
    101, # Incomplete
    102, # Class methods
    108, # openapi_types
    156, # Empty functions
    164, # Trivial, function aliases
    168, # Class methods
    172, # Empty functions
    173, # Class methods
    175, # Class methods
    181, # Empty functions
    182, # Fixed API stuff
    190, # Fixed specific stuff
    197, # from_dictionary class methods
    198, # Empty functions
    234, # Unimplemented
    246, # Fixed specific stuff
    277, # Empty functions
    280, # Empty functions
    282, # Empty functions
    287, # Trivial, e.g. helloWorld()
    299, # Mostly unfinished
    304, # Empty functions
    310, # Fixed API stuff
    313, # Just modifies globals
    320, # Empty functions
    329, # Takes a credentials object, and runs methods on it
    332, # MangoPi bot
    334, # Empty
    338, # namespace_SIO nonsense
    339, # fn(x) = x
    363, # Empty functions
    370, # Empty
    379, # Empty
    388, # Empty
    392, # Empty functions
    393, # Fixed lists
    409, # Fixed dictionaries
    416, # Aliases to print
    428, # Empty functions
    437, # Empty functions
    444, # Empty
    454, # Mostly just calls methods on arguments
    463, # Mostly just calls methods on arguments
    470, # Fixed dictionaries
    474, # Mostly fixed printing
    465, # OpenAPI fixed dictionaries
    476, # Empty
    477, # Fixed dictionaries
    491, # Trivial
    494, # Lots of fixed string stuff
    496, # Empty
    511, # Empty
    518, # OpenAPI
    521, # Fixed API stuff
    536, # Empty
    540, # Fixed API stuff
    553, # Empty
    555, # Empty
    564, # Empty
    566, # Empty
    568, # cls methods
    573, # Mostly fixed dict stuff
    574, # namespace_SO stuff, more biochem?
    582, # namespace_SO stuff, more biochem?
    602, # Fixed lists
    608, # Mostly cls methods
    617, # Mostly cls methods
    629, # cls methods, fixed lists
    641, # Fixed API stuff
    642, # Empty
    647, # Windows API stuff
    648, # jupyter stuff
    649, # mostly fixed dicts
    652, # Empty
    660, # Empty
    665, # cls methods
    666, # Empty
    672, # Empty
    680, # fixed dicts
    682, # Empty
    686, # Empty
    687, # Fixed lists elements_sequence
    692, # cls methods
    693, # ASCII art
    704, # Empty
    709, # mqtt send message
    712, # Empty
    715, # Fixed data recoding
    717, # Empty
    722, # cls methods
    725, # cls methods
    734, # cls methods
    737, # Empty
    741, # Trivial cls methods
    742, # Empty
    745, # Fixed strings
    752, # Empty
    758, # Mostly fixed printing
    768, # Empty
    783, # Empty
    784, # Mostly fixed dicts
    802, # Fixed printing
    806, # Empty
    821, # Empty
    824, # stuff like load_performance_win_x64_win_x64_vs2017_settings
    825, # Trivial
    835, # Empty
    851, # Empty
    862, # Empty
    876, # Trivial
    878, # Empty
    887, # Empty
    888, # Mostly fixed dicts
    890, # Mostly fixed dicts
    893, # Empty
    898, # cls methods
    899, # Fixed ['str'] stuff
    906, # Auto-generated or something
    912, # Empty
    924, # Empty
    933, # namespace_SO biochem stuff
    938, # Trivial
    959, # Mostly fixed printing
    963, # API-specific
    965, # cls methods
    967, # cls methods
    970, # Mostly fixed printing
    971, # cls methods
    972, # cls methods
    973, # Empty
    979, # cls methods
    982, # Empty
    983, # Empty
    989, # cls methods
    990, # API specific
    1007, # API specific
    1014, # Empty
]
```

MIT licensed, like the original dataset