<commit_before># DCDataBaseWithSpark
Java based project that utilzes [SparkSQL](https://spark.apache.org/sql/), [groot](https://github.com/gavalian/groot/), [clas12-offline software](https://github.com/JeffersonLab/clas12-offline-software), [Apache Commons](https://commons.apache.org/), [JDBC](https://en.wikipedia.org/wiki/JDBC_driver), Swing and AWT to provide a GUI for the user to visualize, categorize and document Drift Chamber (DC) wire faults for the CLAS12 detector

# Quick-Start and Reference 

## Data Processing and Visualization
* Processing uses the clas12-offline-software HipoDataSource found in the COATJava package
* Visualization uses the groot package to create histograms of "Layer vs. Wire" for each segment of Sector and Superlayer. It should be noted that "Layer" in the graphical display is "Local Layer", i.e. for each Sector and Superlayer there are 6 "Local Layers" This can be seen on the right side panel of ###NEED IMAGE REFERENCE###
# Inline
![alt text] (https://github.com/mckunkel/DCFaultFinder/blob/master/images/LabeledProcessedPlot.png "Labeled Plot")
<img src="https://github.com/mckunkel/DCFaultFinder/blob/master/images/LabeledProcessedPlot.png" width="800">
## Fault Categorization 
* At the lowest level there is a A.I. that searches for dead wires according to a set value of percent efficiency given by the user. A generated table of possible dead wires will be initially displayed based upon this criterion. This table is the left side panel of the GUI. 
The user is responsible for ensuring the proper fault is selected for the given section of faults displayed on the "Histogram Panel". 
* To aide the user, there is a "Type of Fault" panel (Bottom-Right). 
* Once a fault is selected, the user can place the mouse over the provided histogram to see what that specific fault would appear to be.
  * Double click the histogram panel, with the proper fault and place the selected wires into the "SQL Query" panel (Top-Middle)

## Fault Documentation
* Once a run's entire sector and superlayer has been investigated for possible faults and those possible faults have been proper categorized and placed into the "SQL Query" panel, the entries should be placed into the MySQL database but clicking the button located in the "Send to Database" panel (Bottom-Middle)
  * To place faults into the CCDB, faults must be placed into the MySQL database first.
* Placing entries into the CCDB can be done only after faults into MySQL has been pushed into the database.
  * If entires are in the MySQL database, and the user does not want to process data, just the CCDB entry list, the "Send to CCDB" can be used<commit_msg>more documantation again XI<commit_after># DCDataBaseWithSpark
Java based project that utilzes [SparkSQL](https://spark.apache.org/sql/), [groot](https://github.com/gavalian/groot/), [clas12-offline software](https://github.com/JeffersonLab/clas12-offline-software), [Apache Commons](https://commons.apache.org/), [JDBC](https://en.wikipedia.org/wiki/JDBC_driver), Swing and AWT to provide a GUI for the user to visualize, categorize and document Drift Chamber (DC) wire faults for the CLAS12 detector

# Quick-Start and Reference 

## Data Processing and Visualization
* Processing uses the clas12-offline-software HipoDataSource found in the COATJava package
* Visualization uses the groot package to create histograms of "Layer vs. Wire" for each segment of Sector and Superlayer. It should be noted that "Layer" in the graphical display is "Local Layer", i.e. for each Sector and Superlayer there are 6 "Local Layers" This can be seen on the right side panel of ###NEED IMAGE REFERENCE###
# Inline
! [alt text] (https://github.com/mckunkel/DCFaultFinder/blob/master/images/LabeledProcessedPlot.png "Labeled Plot")

<img src="https://github.com/mckunkel/DCFaultFinder/blob/master/images/LabeledProcessedPlot.png" width="800">
## Fault Categorization 
* At the lowest level there is a A.I. that searches for dead wires according to a set value of percent efficiency given by the user. A generated table of possible dead wires will be initially displayed based upon this criterion. This table is the left side panel of the GUI. 
The user is responsible for ensuring the proper fault is selected for the given section of faults displayed on the "Histogram Panel". 
* To aide the user, there is a "Type of Fault" panel (Bottom-Right). 
* Once a fault is selected, the user can place the mouse over the provided histogram to see what that specific fault would appear to be.
  * Double click the histogram panel, with the proper fault and place the selected wires into the "SQL Query" panel (Top-Middle)

## Fault Documentation
* Once a run's entire sector and superlayer has been investigated for possible faults and those possible faults have been proper categorized and placed into the "SQL Query" panel, the entries should be placed into the MySQL database but clicking the button located in the "Send to Database" panel (Bottom-Middle)
  * To place faults into the CCDB, faults must be placed into the MySQL database first.
* Placing entries into the CCDB can be done only after faults into MySQL has been pushed into the database.
  * If entires are in the MySQL database, and the user does not want to process data, just the CCDB entry list, the "Send to CCDB" can be used
{'fraction_non_alphanumeric': 0.06934032983508245, 'fraction_numerical': 0.004497751124437781, 'mean_word_length': 4.726394849785407, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4371928', 'n_tokens_mistral': 1479, 'n_tokens_neox': 1382, 'n_words': 730}