<commit_before>= Darwin Core Archive

Darwin Core Archive format is a current standard for information exchange between Global Names Architecture modules. This gem allows to work with Darwin Core Archive data compressed to either zip or tar.gz files. More information about Darwing Core Archive can be found on a GBIF page[http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/]:

== Installation

Update to latest rubygems (v >= 1.3.6) which adds gemcutter sources by default.

    sudo gem install dwc-archive

== Usage

    require 'rubygems'
    require 'dwc-archive'

    dwc = DarwinCore.new('/path_to_file/archive_file.tar.gz')
    dwc.archive.files      # the archive file list
    dwc.metadata.data      # summary of metadata from eml.xml if it exists
    dwc.metadata.authors   # authors of the archive
    dwc.core.data          # summary of DarwinCore main file
    dwc.core.file_path     # path to the DarwinCore main file
    dwc.extensions         # array of DarwinCore Star extensions
    dwc.extensions[0].data # summary for an extension
<commit_msg>added more information about importing data into README<commit_after>= Darwin Core Archive

Darwin Core Archive format is a current standard for information exchange between Global Names Architecture modules. This gem allows to work with Darwin Core Archive data compressed to either zip or tar.gz files. More information about Darwing Core Archive can be found on a GBIF page[http://www.gbif.org/informatics/standards-and-tools/publishing-data/data-standards/darwin-core-archives/]:

== Installation

Update to latest rubygems (v >= 1.3.6) which adds gemcutter sources by default.

    sudo gem install dwc-archive

== Usage

    require 'rubygems'
    require 'dwc-archive'

    dwc = DarwinCore.new('/path_to_file/archive_file.tar.gz')
    dwc.archive.files      # the archive file list
    dwc.metadata.data      # summary of metadata from eml.xml if it exists
    dwc.metadata.authors   # authors of the archive
    dwc.core.data          # summary of DarwinCore main file
    dwc.core.file_path     # path to the DarwinCore main file
    dwc.extensions         # array of DarwinCore Star extensions
    dwc.extensions[0].data # summary for an extension

    # read content of the core data file into memory or used with a block
    # it returns array of arrays of data
    # rows that had a wrong encoding will be collected into errors array
    data, errors = dwc.core.read 

    # read content using a block with getting back results in sets 100 rows each
    results = []
    tail_data, tail_errors = dwc.core.read(100) do |data, errors|
      results << [data, errors]
    end
    results << [tail_data, tail_errors]

    # read content of an extension data file into memory
    data, errors = dwc.core.extensions[0].read

    # read content of an extension data using block
    results = []
    tail_data, tail_errors = dwc.core.extensions[0](100) do |data, errors|
      results << [data, errors]
    end
    results << [tail_data, tail_errors]

{'fraction_non_alphanumeric': 0.07059206245933637, 'fraction_numerical': 0.006180871828236825, 'mean_word_length': 3.2472375690607733, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2035436', 'n_tokens_mistral': 882, 'n_tokens_neox': 836, 'n_words': 339}