<commit_before>@wip
Feature: Sequel database connections are fault tolerant using PoolCleaner
  Scenario:
    Given no "/tmp/cranium_archive" directory
    And a "products_2.csv" data file containing:
    """
    id,name,category
    JNI-123,Just a product name,Main category > Subcategory > Sub-subcategory > Ultra-subcategory
    """
    And the following definition:
    """
    source :products do
      file "products_*.csv"
    end

    source :products_transformed do end

    transform :products => :products_transformed do |record|
      connection = Cranium::Database.connection
      procpid = connection.fetch("SELECT procpid FROM pg_stat_activity").first[:procpid]
      connection.run("SELECT PG_TERMINATE_BACKEND(#{procpid})")
      output record
    end
    """
    When I execute the definition
    Then the process should exit successfully
<commit_msg>AT for terminating database connection @wip<commit_after>@wip
Feature: Sequel database connections are fault tolerant using PoolCleaner
  Scenario:
    Given a database table called "dim_product" with the following fields:
      | field_name | field_type |
      | item       | TEXT       |
      | title      | TEXT       |
    And a "products.csv" data file containing:
    """
    id,name,category
    JNI-123,Just a product name,Main category > Subcategory > Sub-subcategory > Ultra-subcategory
    CDI-234,Another product name,Smart Insight > Cool stuff | 3dim > 2dim > 1dim
    """
    And the following definition:
    """
    def terminate_connections
      require 'sequel'
      connection = Sequel.connect "postgres://database_administrator:emarsys@192.168.56.43:5432/cranium"
      connection.run("SELECT pg_terminate_backend(procpid) FROM pg_stat_activity WHERE procpid <> pg_backend_pid() AND datname = 'cranium'")
    end

    source :products do
      encoding "UTF-8"
      delimiter ','

      field :id, String
      field :name, String
    end

    source :transformed_products do
      field :id, String
      field :name, String
    end

    transform :products => :transformed_products do |record|
      terminate_connections

      output record
    end

    import :transformed_products do
      into :dim_product

      put :id => :item
      put :name => :title
    end
    """
    When I execute the definition
    Then the process should exit successfully
    And the "dim_product" table should contain:
      | item    | title                |
      | JNI-123 | Just a product name  |
      | CDI-234 | Another product name |

{'fraction_non_alphanumeric': 0.07996832937450514, 'fraction_numerical': 0.013460015835312747, 'mean_word_length': 2.8462709284627095, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7531329', 'n_tokens_mistral': 741, 'n_tokens_neox': 688, 'n_words': 222}