<commit_before>require 'tempfile'
require 'delegate'
require 'tmpdir'
require 'thread'

module SpoolPool
=begin rdoc
A class to deal with the writing of spool files. Currently uses Tempfile
to do most of the heavy lifting.

Most of this file has been adapted from the Tempfile code in the Ruby 1.9.1
class library, written by yugui.
=end
  class File < DelegateClass( ::File )
    attr_reader :path

=begin rdoc
  Yields the contents of the given +filename+ to the block, and deletes the
  file only if no exception was raised within the block.
=end
    def self.safe_read( filename )
      data = nil

      ::File.open( filename, "r" ) do |filehandle|
        data = filehandle.read
        yield data if block_given?
      end
      ::File.unlink( filename )
      data
    end

=begin rdoc
Stores the given +data+ in a unique file in the directory +basepath+.
+basepath+ can be either a file path as a String or a Pathname.

If the data can't be written to the file (permissions, quota, I/O errors...),
it will attempt to delete the file before throwing an exception.

Returns the path of the file storing the data.
=end
    def self.write( basepath, data ) 
      file = nil
      begin
        file = new( basepath.to_s )
        file.write data<commit_msg>Use File.read instead of opening the spool file<commit_after>require 'tempfile'
require 'delegate'
require 'tmpdir'
require 'thread'

module SpoolPool
=begin rdoc
A class to deal with the writing of spool files. Currently uses Tempfile
to do most of the heavy lifting.

Most of this file has been adapted from the Tempfile code in the Ruby 1.9.1
class library, written by yugui.
=end
  class File < DelegateClass( ::File )
    attr_reader :path

=begin rdoc
  Yields the contents of the given +filename+ to the block, and deletes the
  file only if no exception was raised within the block.
=end
    def self.safe_read( filename )
      data = ::File.read( filename )
      yield data if block_given?
      ::File.unlink( filename )
      data
    end

=begin rdoc
Stores the given +data+ in a unique file in the directory +basepath+.
+basepath+ can be either a file path as a String or a Pathname.

If the data can't be written to the file (permissions, quota, I/O errors...),
it will attempt to delete the file before throwing an exception.

Returns the path of the file storing the data.
=end
    def self.write( basepath, data ) 
      file = nil
      begin
        file = new( basepath.to_s )
        file.write data
{'fraction_non_alphanumeric': 0.06599597585513078, 'fraction_numerical': 0.002414486921529175, 'mean_word_length': 3.109090909090909, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6400513', 'n_tokens_mistral': 743, 'n_tokens_neox': 723, 'n_words': 357}