<commit_before># SPIRIT Data Collection

The source code for data collection is in the [`src/data`](../src/data) directory.

* [`record_bag.py`](../src/data/record_bag.py) is a command line tool which records the relevant data into a ROS Bag file.
* [`bag_to_csv.py`](../src/data/bag_to_csv.py) is a script which converts new bagfiles into CSV 
* [`survey.py`](../src/data/survey.py) is a web interface to collect user information, and their opinions after each run of the experiment.
Since [`survey.py`](../src/data/survey.py) requires a `__main__` module to run, a convenience script, [`run_survey`](../src/data/run_survey), which can be run from anywhere.

[`config.py`](../src/data/config.py) contains the configuration options for all the data collection tools.
By default, bagfiles and survey data are stored in [`data/raw`](../data/raw), while the CSV data is stored in [`data/interim`](../data/interim).

[`record_bag.py`](../src/data/record_bag.py) and [`bag_to_csv.py](../src/data/bag_to_csv.py)` can only be used from inside the Docker container.

## Checklist
* [ ] If you will be recording the screen and camera input, make sure OBS is set up properly prior to running the experiment.
  * Set a hotkey which would allow you to start and stop recording, so that you don't need to switch to the application.
  * A good location to store the videos is [`data/raw/recordings`](../data/raw/recordings).
* [ ] Have the users sign the consent and waiver forms.
* [ ] Run the survey.
* [ ] Select the user.
  * If the user has done previous experiments, select their name from the list below the table.
  * Otherwise, click "Add user", input their information, and press "Ok".
* [ ] Move to the corresponding tab and click on the "Run experiment" button.
For my experiment, the users with an even ID started with the onboard view, while those with an odd ID started with SPIRIT.
It is also possible to choose a random experiment, or an experiment which allows the user to see both interfaces at the same time.
* [ ] Now, you want to run the Docker container and connect to the drone according to the [checklist](readme_spirit.md#checklist) in the corresponding [readme](readme_spirit.md).
* [ ] Depending on the experiment, you may need to hide one of the video windows or mute the sounds at this point.
* [ ] In a separate terminal, `join` the running container and start collecting data:
  * [ ] `cd` to the /root/analysis/scripts directory.
  * [ ] Run `record_bag.py` with two arguments: the experimenter ID and the experiment type. Use `./record_bag.py --help` for more details.
  * [ ] Press the OBS hotkey to start recording.
* [ ] <a name="checklist-stop-collection" />Return to the SPIRIT [checklist](readme_spirit.md#checklist-collect_data) and continue normally.
* [ ] When done with the run, stop the data collection:
  * [ ] Press the OBS hotkey to stop recording.<commit_msg>Clarification and notes on survey code quality.<commit_after># SPIRIT Data Collection

The source code for data collection is in the [`src/data`](../src/data) directory.

* [`record_bag.py`](../src/data/record_bag.py) is a command line tool which records the relevant data into a ROS Bag file.
* [`bag_to_csv.py`](../src/data/bag_to_csv.py) is a script which converts new bagfiles into CSV for consumption offline.
* [`survey.py`](../src/data/survey.py) is a web interface to collect user information, and their opinions after each run of the experiment.
Since [`survey.py`](../src/data/survey.py) requires a `__main__` module to run, a convenience script, [`run_survey`](../src/data/run_survey), which can be run from anywhere.
Note that this code is extremely messy since it was my first time making something with a GUI, and future users may want to use something else to create a better, cleaner, more modular interface.

[`config.py`](../src/data/config.py) contains the configuration options for all the data collection tools.
By default, bagfiles and survey data are stored in [`data/raw`](../data/raw), while the CSV data is stored in [`data/interim`](../data/interim).

[`record_bag.py`](../src/data/record_bag.py) and [`bag_to_csv.py](../src/data/bag_to_csv.py)` can only be used from inside the Docker container.

## Checklist
* [ ] If you will be recording the screen and camera input, make sure OBS is set up properly prior to running the experiment.
  * Set a hotkey which would allow you to start and stop recording, so that you don't need to switch to the application.
  * A good location to store the videos is [`data/raw/recordings`](../data/raw/recordings).
* [ ] Have the users sign the consent and waiver forms.
* [ ] Run the survey.
* [ ] Select the user.
  * If the user has done previous experiments, select their name from the list below the table.
  * Otherwise, click "Add user", input their information, and press "Ok".
* [ ] Move to the corresponding tab and click on the "Run experiment" button.
For my experiment, the users with an even ID started with the onboard view, while those with an odd ID started with SPIRIT.
It is also possible to choose a random experiment, or an experiment which allows the user to see both interfaces at the same time.
* [ ] Now, you want to run the Docker container and connect to the drone according to the [checklist](readme_spirit.md#checklist) in the corresponding [readme](readme_spirit.md).
* [ ] Depending on the experiment, you may need to hide one of the video windows or mute the sounds at this point.
* [ ] In a separate terminal, `join` the running container and start collecting data:
  * [ ] `cd` to the /root/analysis/scripts directory.
  * [ ] Run `record_bag.py` with two arguments: the experimenter ID and the experiment type. Use `./record_bag.py --help` for more details.
  * [ ] Press the OBS hotkey to start recording.
* [ ] <a name="checklist-stop-collection" />Return to the SPIRIT [checklist](readme_spirit.md#checklist-collect_data) and continue normally.
* [ ] When done with the run, stop the data collection:
  * [ ] Press the OBS hotkey to stop recording.
{'fraction_non_alphanumeric': 0.1012909632571996, 'fraction_numerical': 0.0, 'mean_word_length': 4.0824222035323805, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3594391', 'n_tokens_mistral': 1722, 'n_tokens_neox': 1682, 'n_words': 807}