<commit_before>@RALLY_DE2347
Feature: Duplicate detection modes

#  As an ingestion user, I want to be able to control duplicate detection (record hashing) to ignore suspect data, disable it completely and test its performance

Background: I have a landing zone route configured
Given I am using local data store

Scenario: Ingest and reingest with default mode
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
When the landing zone is reinitialized
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 0" in the resulting batch job file
  And I should see "InterchangeEducationOrganization.xml stateEducationAgency 1 deltas" in the resulting batch job file
  
Scenario: Ingest then use mode reset
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
When I post "TinyDataSetDDreset.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
  And I should see "duplicate-detection: reset" in the resulting batch job file
  
Scenario: Ingest then use mode disable
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
When I post "TinyDataSetDDdisable.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 0                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
  And I should see "duplicate-detection: disable" in the resulting batch job file
  
Scenario: Ingest initially with debugdrop
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSetDDdebugdrop.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |<commit_msg>[DE2347] - wip feature to fix build<commit_after>@RALLY_DE2347

@wip
Feature: Duplicate detection modes

#  As an ingestion user, I want to be able to control duplicate detection (record hashing) to ignore suspect data, disable it completely and test its performance

Background: I have a landing zone route configured
Given I am using local data store

Scenario: Ingest and reingest with default mode
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
When the landing zone is reinitialized
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 0" in the resulting batch job file
  And I should see "InterchangeEducationOrganization.xml stateEducationAgency 1 deltas" in the resulting batch job file

Scenario: Ingest then use mode reset
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
When I post "TinyDataSetDDreset.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
  And I should see "duplicate-detection: reset" in the resulting batch job file

Scenario: Ingest then use mode disable
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSet.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 1                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
When I post "TinyDataSetDDdisable.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
     | recordHash               | 0                   | t                           | Midgar                  | string               |
  And I should see "InterchangeEducationOrganization.xml records ingested successfully: 1" in the resulting batch job file
  And I should see "duplicate-detection: disable" in the resulting batch job file

Scenario: Ingest initially with debugdrop
Given I am using preconfigured Ingestion Landing Zone
  And I post "TinyDataSetDDdebugdrop.zip" file as the payload of the ingestion job
  And zip file is scp to ingestion landing zone
  And I am willing to wait upto 60 seconds for ingestion to complete
  And a batch job log has been created
  And I should not see a warning log file created
  And I should not see an error log file created
Then I check to find if record is in batch job collection:
     | collectionName           | expectedRecordCount | searchParameter             | searchValue             | searchType           |
{'fraction_non_alphanumeric': 0.02520740268028079, 'fraction_numerical': 0.005264837268666241, 'mean_word_length': 2.242886704604242, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 14}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 37, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6651033', 'n_tokens_mistral': 3001, 'n_tokens_neox': 2796, 'n_words': 1644}