<commit_before>BUG:
- High-DM astropysical pulses are removed (es. L199854_SAP0_BEAM72)
- Top_candidates in inf are wrong
- Only 10 top_candidates in inc beams
- Rank in top_candidates.inf wrong without inc beam
- Ignore corrupted tar files
- Check dimension of squares, sometimes not related with sigma apparently

- Remove space from output file names
- Folder with general best plots
- Sometimes top_candidates reported in the output are not the best and the same reported in the plot (es. L232335_SAP0_BEAM24)
- SAP0_BEAM12 has repeated best_pulses (es. L232335)
- Single pulse plots: enlarge points dimension
- General plots: enlarge points dimension and diminish square dimension
- Single pulse plots: report obs name, sigma max, rank number
- Histogram bins smaller
- Histogram bins logarithmic or plot non logarithmic
- Beam plots: in top-right plot move numbers to the left
- Beam plots: give numbers to squares
- Move outside grid in plots
- In top_candidates.inf incoherent pulses are reported twice
- Remove time spans affected by RFI in general plot

- Add filter duration/sigma 
- Do not plot if plot is empty
- Better study parameters of grouping
  - Better study filter values
- Substitute duration filter with downfact filter
- Divide the general plot for the three saps
- Plot horiontal lines in DM trial changes
- Plots of pulse shapes larger when few pulses

TO TEST:
- Plots smaller
- Single pulse plots: circle dimensions relative to max
- Histogram also for only pulses brighter than threshold
- Strongest pulse in other list
- Strongest pulse only for DM > x

- General plot with counts in every beam
- Alert for pulsars (more than x pulses at one DM, more than y pulses brighter than S in one DM)
- Pulse spectra<commit_msg>Things to do<commit_after>BUGS:
- High-DM astropysical pulses are removed (es. L199854_SAP0_BEAM72)
- Top_candidates in inf are wrong
- Only 10 top_candidates in inc beams
- Rank in top_candidates.inf wrong without inc beam
- Ignore corrupted tar files
- Check dimension of squares, sometimes not related with sigma apparently


PLOT:
- General plots: enlarge points dimension and diminish square dimension
- Single pulse plots: enlarge points dimension
- Single pulse plots: report obs name, sigma max, rank number
- Histogram bins smaller
- Histogram bins logarithmic or plot non logarithmic
- Beam plots: in top-right plot move numbers to the left
- Beam plots: give numbers to squares
- Move outside grid in plots
- Remove time spans affected by RFI in general plot
- Do not plot if plot is empty
- Divide the general plot for the three saps
- Plot horiontal lines in DM trial changes
- Plots of pulse shapes larger when few pulses
- Plots smaller
- Single pulse plots: circle dimensions relative to max
- Histogram also for only pulses brighter than threshold
- General plot with counts in every beam


OUTPUT:
- Remove space from output file names
- Folder with general best plots
- Sometimes top_candidates reported in the output are not the best and the same reported in the plot (es. L232335_SAP0_BEAM24)
- SAP0_BEAM12 has repeated best_pulses (es. L232335)
- In top_candidates.inf incoherent pulses are reported twice


ANALYSIS:
- Add filter duration/sigma 
- Better study parameters of grouping
  - Better study filter values
- Substitute duration filter with downfact filter
- Strongest pulse in other list
- Strongest pulse only for DM > x
- Alert for pulsars (more than x pulses at one DM, more than y pulses brighter than S in one DM)
- Pulse spectra
{'fraction_non_alphanumeric': 0.03926031294452347, 'fraction_numerical': 0.016500711237553343, 'mean_word_length': 4.08092485549133, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7157425', 'n_tokens_mistral': 1027, 'n_tokens_neox': 903, 'n_words': 514}