<commit_before>@RALLY_US4112
Feature: Ingestion Delta Test

Background: I have a landing zone route configured
Given I am using local data store
 And I am using preconfigured Ingestion Landing Zone

Scenario: Job report should report deltas when SDS is ingested twice
    Given I post "StoriedDataSet_IL_Daybreak.zip" file as the payload of the ingestion job
    And the following collections are empty in batch job datastore:
        | collectionName                        |
        | assessment                            |
        | attendance                            |
        | calendarDate                          |
        | cohort                                |
        | competencyLevelDescriptor             |
        | course                                |
        | courseOffering                        |
        | disciplineAction                      |
        | disciplineIncident                    |
        | educationOrganization                 |
        | grade                                 |
        | gradebookEntry                        |
        | gradingPeriod                         |
        | graduationPlan                        |
        | learningObjective                     |
        | learningStandard                      |
        | parent                                |
        | program                               |
        | reportCard                            |
        | section                               |
        | session                               |
        | staff                                 |
        | staffCohortAssociation                |
        | staffEducationOrganizationAssociation |
        | staffProgramAssociation               |
        | student                               |
        | studentAcademicRecord                 |
        | studentAssessment          |
        | studentCohortAssociation              |
        | studentCompetency                     |
        | studentCompetencyObjective            |
        | studentDisciplineIncidentAssociation  |
        | studentGradebookEntry                 |
        | studentParentAssociation              |
        | studentProgramAssociation             |
        | studentSchoolAssociation              |
        | studentSectionAssociation             |
        | courseTranscript                      |
        | teacherSchoolAssociation              |
        | teacherSectionAssociation             |

    And zip file is scp to ingestion landing zone
    And a batch job log has been created
    And zip file is scp to ingestion landing zone with name "Reingest-StoriedDataSet_IL_Daybreak.zip"
    And a batch job for file "Reingest-StoriedDataSet_IL_Daybreak.zip" is completed in database
And I should see "InterchangeStudent.xml student 78 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml course 95 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml school 3 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml program 2 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml competencyLevelDescriptor 6 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml localEducationAgency 2 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml stateEducationAgency 1 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrgCalendar.xml calendarDate 556 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrgCalendar.xml session 22 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrgCalendar.xml gradingPeriod 17 deltas!" in the resulting batch job file
And I should see "InterchangeMasterSchedule.xml courseOffering 95 deltas!" in the resulting batch job file
And I should see "InterchangeMasterSchedule.xml section 97 deltas!" in the resulting batch job file
And I should see "InterchangeStudentProgram.xml studentProgramAssociation 6 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml teacherSchoolAssociation 3 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml staff 11 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml teacherSectionAssociation 11 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml staffEducationOrganizationAssociation 10 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml teacher 3 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml staffProgramAssociation 7 deltas!" in the resulting batch job file
And I should see "InterchangeStudentEnrollment.xml studentSchoolAssociation 167 deltas!" in the resulting batch job file
And I should see "InterchangeStudentEnrollment.xml graduationPlan 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentEnrollment.xml studentSectionAssociation 325 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-English.xml learningStandard 954 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-Math.xml learningStandard 509 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CommonCore.xml learningStandard 36 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml courseTranscript 196 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentAcademicRecord 117 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentCompetencyObjective 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml reportCard 2 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentCompetency 60 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml grade 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentGradebookEntry 315 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml gradebookEntry 12 deltas!" in the resulting batch job file
And I should see "InterchangeStudentParent.xml parent 9 deltas!" in the resulting batch job file
And I should see "InterchangeStudentParent.xml studentParentAssociation 9 deltas!" in the resulting batch job file
And I should see "InterchangeStudentCohort.xml cohort 3 deltas!" in the resulting batch job file
And I should see "InterchangeStudentCohort.xml studentCohortAssociation 6 deltas!" in the resulting batch job file
And I should see "InterchangeStudentCohort.xml staffCohortAssociation 3 deltas!" in the resulting batch job file
And I should see "InterchangeStudentDiscipline.xml disciplineAction 2 deltas!" in the resulting batch job file
And I should see "InterchangeStudentDiscipline.xml studentDisciplineIncidentAssociation 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentDiscipline.xml disciplineIncident 2 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-English.xml learningObjective 70 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-Math.xml learningObjective 65 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CommonCore.xml learningObjective 63 deltas!" in the resulting batch job file
# Uncomment when deltas for Attendance implemented
# And I should see "InterchangeAttendance.xml attendance 5550 deltas!" in the resulting batch job file


Scenario: Job report should not report deltas when SDS is ingested twice for different tenantId
Given I am using preconfigured Ingestion Landing Zone for "Midgar-Daybreak"
    And I post "Hyrule.zip" file as the payload of the ingestion job for "Midgar-Daybreak"
    And the following collections are empty in datastore:
        | collectionName              |
        | student                     |
        | studentSchoolAssociation    |
        | course                      |
        | educationOrganization       |
        | section                     |
        | studentSectionAssociation   |
        | staff                       |
        |staffEducationOrganizationAssociation|
        | teacherSchoolAssociation    |
        | teacherSectionAssociation   |
        | session                     |
        | assessment                  |
        | studentAssessment|
        | gradebookEntry              |
        | courseTranscript            |
        | studentGradebookEntry       |
        | parent                      |
        | studentParentAssociation    |
        | attendance                  |
        | program                     |
        | staffProgramAssociation     |
        | studentProgramAssociation   |<commit_msg>us4161 Enable attedance delta count check in the AT<commit_after>@RALLY_US4112
Feature: Ingestion Delta Test

Background: I have a landing zone route configured
Given I am using local data store
 And I am using preconfigured Ingestion Landing Zone

Scenario: Job report should report deltas when SDS is ingested twice
    Given I post "StoriedDataSet_IL_Daybreak.zip" file as the payload of the ingestion job
    And the following collections are empty in batch job datastore:
        | collectionName                        |
        | assessment                            |
        | attendance                            |
        | calendarDate                          |
        | cohort                                |
        | competencyLevelDescriptor             |
        | course                                |
        | courseOffering                        |
        | disciplineAction                      |
        | disciplineIncident                    |
        | educationOrganization                 |
        | grade                                 |
        | gradebookEntry                        |
        | gradingPeriod                         |
        | graduationPlan                        |
        | learningObjective                     |
        | learningStandard                      |
        | parent                                |
        | program                               |
        | reportCard                            |
        | section                               |
        | session                               |
        | staff                                 |
        | staffCohortAssociation                |
        | staffEducationOrganizationAssociation |
        | staffProgramAssociation               |
        | student                               |
        | studentAcademicRecord                 |
        | studentAssessment          |
        | studentCohortAssociation              |
        | studentCompetency                     |
        | studentCompetencyObjective            |
        | studentDisciplineIncidentAssociation  |
        | studentGradebookEntry                 |
        | studentParentAssociation              |
        | studentProgramAssociation             |
        | studentSchoolAssociation              |
        | studentSectionAssociation             |
        | courseTranscript                      |
        | teacherSchoolAssociation              |
        | teacherSectionAssociation             |

    And zip file is scp to ingestion landing zone
    And a batch job log has been created
    And zip file is scp to ingestion landing zone with name "Reingest-StoriedDataSet_IL_Daybreak.zip"
    And a batch job for file "Reingest-StoriedDataSet_IL_Daybreak.zip" is completed in database
And I should see "InterchangeStudent.xml student 78 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml course 95 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml school 3 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml program 2 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml competencyLevelDescriptor 6 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml localEducationAgency 2 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrganization.xml stateEducationAgency 1 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrgCalendar.xml calendarDate 556 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrgCalendar.xml session 22 deltas!" in the resulting batch job file
And I should see "InterchangeEducationOrgCalendar.xml gradingPeriod 17 deltas!" in the resulting batch job file
And I should see "InterchangeMasterSchedule.xml courseOffering 95 deltas!" in the resulting batch job file
And I should see "InterchangeMasterSchedule.xml section 97 deltas!" in the resulting batch job file
And I should see "InterchangeStudentProgram.xml studentProgramAssociation 6 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml teacherSchoolAssociation 3 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml staff 11 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml teacherSectionAssociation 11 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml staffEducationOrganizationAssociation 10 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml teacher 3 deltas!" in the resulting batch job file
And I should see "InterchangeStaffAssociation.xml staffProgramAssociation 7 deltas!" in the resulting batch job file
And I should see "InterchangeStudentEnrollment.xml studentSchoolAssociation 167 deltas!" in the resulting batch job file
And I should see "InterchangeStudentEnrollment.xml graduationPlan 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentEnrollment.xml studentSectionAssociation 325 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-English.xml learningStandard 954 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-Math.xml learningStandard 509 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CommonCore.xml learningStandard 36 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml courseTranscript 196 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentAcademicRecord 117 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentCompetencyObjective 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml reportCard 2 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentCompetency 60 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml grade 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml studentGradebookEntry 315 deltas!" in the resulting batch job file
And I should see "InterchangeStudentGrade.xml gradebookEntry 12 deltas!" in the resulting batch job file
And I should see "InterchangeStudentParent.xml parent 9 deltas!" in the resulting batch job file
And I should see "InterchangeStudentParent.xml studentParentAssociation 9 deltas!" in the resulting batch job file
And I should see "InterchangeStudentCohort.xml cohort 3 deltas!" in the resulting batch job file
And I should see "InterchangeStudentCohort.xml studentCohortAssociation 6 deltas!" in the resulting batch job file
And I should see "InterchangeStudentCohort.xml staffCohortAssociation 3 deltas!" in the resulting batch job file
And I should see "InterchangeStudentDiscipline.xml disciplineAction 2 deltas!" in the resulting batch job file
And I should see "InterchangeStudentDiscipline.xml studentDisciplineIncidentAssociation 4 deltas!" in the resulting batch job file
And I should see "InterchangeStudentDiscipline.xml disciplineIncident 2 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-English.xml learningObjective 70 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CCS-Math.xml learningObjective 65 deltas!" in the resulting batch job file
And I should see "InterchangeAssessmentMetadata-CommonCore.xml learningObjective 63 deltas!" in the resulting batch job file
And I should see "InterchangeAttendance.xml attendance 5550 deltas!" in the resulting batch job file


Scenario: Job report should not report deltas when SDS is ingested twice for different tenantId
Given I am using preconfigured Ingestion Landing Zone for "Midgar-Daybreak"
    And I post "Hyrule.zip" file as the payload of the ingestion job for "Midgar-Daybreak"
    And the following collections are empty in datastore:
        | collectionName              |
        | student                     |
        | studentSchoolAssociation    |
        | course                      |
        | educationOrganization       |
        | section                     |
        | studentSectionAssociation   |
        | staff                       |
        |staffEducationOrganizationAssociation|
        | teacherSchoolAssociation    |
        | teacherSectionAssociation   |
        | session                     |
        | assessment                  |
        | studentAssessment|
        | gradebookEntry              |
        | courseTranscript            |
        | studentGradebookEntry       |
        | parent                      |
        | studentParentAssociation    |
        | attendance                  |
        | program                     |
        | staffProgramAssociation     |
        | studentProgramAssociation   |
{'fraction_non_alphanumeric': 0.03828223550704766, 'fraction_numerical': 0.009323753633521637, 'mean_word_length': 2.437134778510839, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 90}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7490355', 'n_tokens_mistral': 4453, 'n_tokens_neox': 4005, 'n_words': 1698}