task_categories:text-classification size_categories:10K<n<100K region:us
---
configs:
- config_name: default
  data_files:
  - split: java_Pointer
    path: data/java_Pointer-*
  - split: java_Expand
    path: data/java_Expand-*
  - split: java_Ownership
    path: data/java_Ownership-*
  - split: java_deprecation
    path: data/java_deprecation-*
  - split: java_rational
    path: data/java_rational-*
  - split: java_summary
    path: data/java_summary-*
  - split: java_usage
    path: data/java_usage-*
  - split: python_Expand
    path: data/python_Expand-*
  - split: python_Summary
    path: data/python_Summary-*
  - split: python_DevelopmentNotes
    path: data/python_DevelopmentNotes-*
  - split: python_Parameters
    path: data/python_Parameters-*
  - split: python_Usage
    path: data/python_Usage-*
  - split: pharo_Example
    path: data/pharo_Example-*
  - split: pharo_Keymessages
    path: data/pharo_Keymessages-*
  - split: pharo_Responsibilities
    path: data/pharo_Responsibilities-*
  - split: pharo_Keyimplementationpoints
    path: data/pharo_Keyimplementationpoints-*
  - split: pharo_Collaborators
    path: data/pharo_Collaborators-*
  - split: pharo_Intent
    path: data/pharo_Intent-*
  - split: pharo_Classreferences
    path: data/pharo_Classreferences-*
dataset_info:
  features:
  - name: comment_sentence_id
    dtype: int64
  - name: class
    dtype: string
  - name: comment_sentence
    dtype: string
  - name: partition
    dtype: int64
  - name: instance_type
    dtype: int64
  - name: category
    dtype: string
  - name: label
    dtype: int64
  - name: combo
    dtype: string
  - name: __index_level_0__
    dtype: int64
  splits:
  - name: java_Pointer
    num_bytes: 483600
    num_examples: 2418
  - name: java_Expand
    num_bytes: 481182
    num_examples: 2418
  - name: java_Ownership
    num_bytes: 488436
    num_examples: 2418
  - name: java_deprecation
    num_bytes: 493272
    num_examples: 2418
  - name: java_rational
    num_bytes: 486018
    num_examples: 2418
  - name: java_summary
    num_bytes: 483600
    num_examples: 2418
  - name: java_usage
    num_bytes: 478764
    num_examples: 2418
  - name: python_Expand
    num_bytes: 421025
    num_examples: 2555
  - name: python_Summary
    num_bytes: 423580
    num_examples: 2555
  - name: python_DevelopmentNotes
    num_bytes: 446575
    num_examples: 2555
  - name: python_Parameters
    num_bytes: 431245
    num_examples: 2555
  - name: python_Usage
    num_bytes: 418470
    num_examples: 2555
  - name: pharo_Example
    num_bytes: 368156
    num_examples: 1765
  - name: pharo_Keymessages
    num_bytes: 375216
    num_examples: 1765
  - name: pharo_Responsibilities
    num_bytes: 384041
    num_examples: 1765
  - name: pharo_Keyimplementationpoints
    num_bytes: 396396
    num_examples: 1765
  - name: pharo_Collaborators
    num_bytes: 378746
    num_examples: 1765
  - name: pharo_Intent
    num_bytes: 366391
    num_examples: 1765
  - name: pharo_Classreferences
    num_bytes: 382276
    num_examples: 1765
  download_size: 3231436
  dataset_size: 8186989
task_categories:
- text-classification
size_categories:
- 10K<n<100K
---
# Dataset Card for "nlbse_ccc"

A dataset object for the NLBSE'23 Code Comment Classification competition. Please refer to the original [Github repo for more details](https://github.com/nlbse2023/code-comment-classification).

## Category distribution in the training and test sets

The table below shows the distribution of positive/negative sentences for each category in the training and testing sets.

| Language | Category           | Training | Training | Testing  | Testing  | Total  |
|----------|--------------------|---------:|---------:|---------:|---------:|-------:|
|          |                    | **Positive** | **Negative** | **Positive** | **Negative** |        |
| Java     | Expand             |      505 |     1426 |      127 |      360 |   2418 |
| Java     | Ownership          |       90 |     1839 |       25 |      464 |   2418 |
| Java     | Deprecation        |      100 |     1831 |       27 |      460 |   2418 |
| Java     | Rational           |      223 |     1707 |       57 |      431 |   2418 |
| Java     | Summary            |      328 |     1600 |       87 |      403 |   2418 |
| Java     | Pointer            |      289 |     1640 |       75 |      414 |   2418 |
| Java     | Usage              |      728 |     1203 |      184 |      303 |   2418 |
|          |                    	 | **Positive** | **Negative** | **Positive** | **Negative** |        |
| Pharo    | Responsibilities   	 |      267 |     1139 |       69 |      290 |   1765 |
| Pharo    | Keymessages        	 |      242 |     1165 |       63 |      295 |   1765 |
| Pharo    | Keyimplementationpoints |      184 |     1222 |       48 |      311 |   1765 |
| Pharo    | Collaborators      	 |       99 |     1307 |       28 |      331 |   1765 |
| Pharo    | Example            	 |      596 |      812 |      152 |      205 |   1765 |
| Pharo    | Classreferences    	 |       60 |     1348 |       17 |      340 |   1765 |
| Pharo    | Intent             	 |      173 |     1236 |       45 |      311 |   1765 |
|          |                    | **Positive** | **Negative** | **Positive** | **Negative** |        |
| Python   | Expand             |      402 |     1637 |      102 |      414 |   2555 |
| Python   | Parameters         |      633 |     1404 |      161 |      357 |   2555 |
| Python   | Summary            |      361 |     1678 |       93 |      423 |   2555 |
| Python   | Developmentnotes   |      247 |     1792 |       65 |      451 |   2555 |
| Python   | Usage              |      637 |     1401 |      163 |      354 |   2555 |

## Code

The following code snippet was used to create the dataset:

```
# !git clone https://github.com/nlbse2023/code-comment-classification.git 
from datasets import DatasetDict

langs = ['java', 'python', 'pharo']
lan_cats = []
dataset_dict = DatasetDict()
for lan in langs: # for each language
    df = pd.read_csv(f'./code-comment-classification/{lan}/input/{lan}.csv')
    df['label'] = df.instance_type
    df['combo'] = df[['comment_sentence', 'class']].agg(' | '.join, axis=1)
    print(df.columns)
    cats = list(map(lambda x: lan + '_' + x, list(set(df.category))))
    lan_cats = lan_cats + cats
    for cat in list(set(df.category)): # for each category
        filtered =  df[df.category == cat]
        dataset_dict[f'{lan}_{cat}'] = Dataset.from_pandas(filtered)
dataset_dict.push_to_hub("AISE-TUDelft/nlbse_ccc", token='hf_********************')
```
