<commit_before>=================
API Documentation
=================

:class:`Dedupe` Objects
--------------------------
Class for active learning deduplication. Use deduplication when you have
data that can contain multiple records that can all refer to the same
entity. 

.. py:class:: Dedupe(field_definition, [data_sample=None, [num_processes]])

   Initialize a Dedupe object with a field definition

   :param dict field_definition: A field definition is a dictionary
				 where the keys are the fields that
				 will be used for training a model
				 and the values are the field
				 specification
   :param data_sample: is an optional argument that `we'll discuss fully
		       below <#wiki-sample-dedupe>`__
   :param int num_processes: the number of processes to use for parallel
			     processing, defaults to 1


   .. code:: python

      # initialize from a defined set of fields
      fields = {
	        'Site name': {'type': 'String'},
		'Address': {'type': 'String'},
		'Zip': {'type': 'String', 'Has Missing':True},
		'Phone': {'type': 'String', 'Has Missing':True},
		}

      deduper = dedupe.Dedupe(fields)

   If can't use this method because of the size your data (see the
   `MySQL
   <http://open-city.github.com/dedupe/doc/mysql_example.html>`__),
   you'll need to initialize Dedupe with the sample

   .. code:: python
	  
      deduper = Dedupe(field_definition, data_sample)

   ``data_sample`` should be a sequence of tuples, where each tuple
   contains a pair of records, and each record is a dictionary-like
   object that contains the field names you declared in
   field\_definitions as keys.

   For example, a data\_sample with only one pair of records,

   .. code:: python

      [
      (
       (854, {'city': 'san francisco',
              'address': '300 de haro st.',
              'name': "<NAME> & bakery",
              'cuisine': 'american'}),
       (855, {'city': 'san francisco',
             'address': '1328 18th st.',
             'name': 'san francisco bbq',
             'cuisine': 'thai'})
       )
       ]


   or ``deduper = dedupe.Dedupe(fields, data_sample)``


   .. py:method:: sample(data[, sample_size=150000])

      If you did not initialize the Dedupe object with a data_sample, you
      will need to call this method to take a random sample of your data to be
      used for training.

      :param dict data: A dictionary-like object indexed by record ID
			where the values are dictionaries representing records.
      :param int sample_size: Number of record tuples to return. Defaults
			      to 150,000.

      .. code:: python

	 data_sample = deduper.sample(data_d, 150000)



   .. include:: common_learning_methods.rst
   .. include:: common_dedupe_methods.rst
   .. include:: common_methods.rst
<commit_msg>better description of data_sample<commit_after>=================
API Documentation
=================

:class:`Dedupe` Objects
--------------------------
Class for active learning deduplication. Use deduplication when you have
data that can contain multiple records that can all refer to the same
entity. 

.. py:class:: Dedupe(field_definition, [data_sample=None, [num_processes]])

   Initialize a Dedupe object with a :doc:`field definition
   <Field-definition.rst>`

   :param dict field_definition: A field definition is a dictionary
				 where the keys are the fields that
				 will be used for training a model
				 and the values are the field
				 specifications 
   :param data_sample: is an optional argument that we discuss below
   :param int num_processes: the number of processes to use for parallel
			     processing, defaults to 1

   In order to learn how to deduplicate records, dedupe needs a sample
   of records you are trying to deduplicate. If your data is not too
   large (fits in memory), you can pass your data to the
   :py:meth:`~Dedupe.sample` method and dedupe will take a sample for
   you.

   .. code:: python

      # initialize from a defined set of fields
      fields = {
	        'Site name': {'type': 'String'},
		'Address': {'type': 'String'},
		'Zip': {'type': 'String', 'Has Missing':True},
		'Phone': {'type': 'String', 'Has Missing':True},
		}

      deduper = dedupe.Dedupe(fields)

      deduper.sample(your_data)

   If your data won't fit in memory, you'll have to prepare a sample
   of the data yourself and pass it to Dedupe.

   ``data_sample`` should be a sequence of tuples, where each tuple
   contains a pair of records, and each record is a dictionary-like
   object that contains the field names you declared in
   field\_definitions as keys.

   For example, a data_sample with only one pair of records,

   .. code:: python

      data_sample = [(
                      (854, {'city': 'san francisco',
	                     'address': '300 de haro st.',
		             'name': "<NAME>",
		             'cuisine': 'american'}),
	               (855, {'city': 'san francisco',
	                      'address': '1328 18th st.',
                              'name': 'san francisco bbq',
                              'cuisine': 'thai'})
	               )
	              ]

      deduper = dedupe.Dedupe(fields)
      
   See `MySQL
   <http://open-city.github.com/dedupe/doc/mysql_example.html>`__ for
   an example of how to create a data sample yourself.

   .. py:method:: sample(data[, sample_size=150000])

      If you did not initialize the Dedupe object with a data_sample, you
      will need to call this method to take a random sample of your data to be
      used for training.

      :param dict data: A dictionary-like object indexed by record ID
			where the values are dictionaries representing records.
      :param int sample_size: Number of record tuples to return. Defaults
			      to 150,000.

      .. code:: python

	 data_sample = deduper.sample(data_d, 150000)



   .. include:: common_learning_methods.rst
   .. include:: common_dedupe_methods.rst
   .. include:: common_methods.rst

{'fraction_non_alphanumeric': 0.10962118672477372, 'fraction_numerical': 0.011397921555481059, 'mean_word_length': 2.6141732283464565, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6581733', 'n_tokens_mistral': 1811, 'n_tokens_neox': 1584, 'n_words': 667}