<commit_before>#TL:1:BSON::Document:

use v6.d;

#-------------------------------------------------------------------------------
=begin pod

=head1 BSON::Document

BSON Encodable and Decodable document


=head1 Description

Document storage with Hash like behavior used mainly to communicate with a mongodb server. It can also be used as a serialized storage format. The main difference with the Hash is that this class keeps the input order of inserted key-value pairs which is important for the use with mongodb.

=head2 Raku type mapping

To get a proper mapping from Raku types to what BSON can handle, there are some rules and restrictions implemented.

At the top of BSON there is a type called document. (See also L<the bson spec|https://bsonspec.org/spec.html>). This is a representation of a series of keys with values. This could be represented by a B<Hash> but the problem is that the keys of a Hash are not ordered, that is, the order of keys is often not the same as on input while this is a requirement of BSON. So, the best option to represent a document, is a B<List> of B<Pair>s.

The values of a Pair can be the simple types like B<Bool>, B<Str>, B<Num> and B<Int>. B<Rat> and B<FatRat> are converted to Num. It can not be another Pair!

An B<Array> may also be used as a value. Its elements are like the values of a Pair, so the elements can be types like mentioned above for the Pair values or List of Pair. This also leads to the fact that Arrays may be nested.

Binary data can be provided as B<BSON::Binary>. There is support for several types of binary data. When a general type of binary data is offered, you can also give a B<Buf>. The Buf will then be converted into a BSON::Binary.

It is possible to run Javascript code on a MongoDB server so there is a type for it in the BSON specification. For Raku, there is a type called B<BSON::Javascript> to handle that type.

BSON is a specification for C-type data so the Raku types are converted into native types. Integers are only signed integers. There are B<int32> and B<int64> types. Internally this class will take the smallest size when encoding the integer. Int can handle very large numbers. When they are too big for an int64, they are truncated. Num is always converted to a B<double> and boolean values to int32 C<0> or C<1>.

The BSON::Document class has a role B<BSON::Ordered> to handle the types. This means that you can create your own class and do almost the same things. What it does not handle is the encoding and decoding and the initialization like done for BSON::Document. So, in short, only the assignments are supported.

  class A does BSON::Ordered { }
  my A $a .= new;
  $a<k1> = 10;
  my BSON::Document $d1 .= new($a);


=head2 Encoding and decoding

Encoding is needed when it is sent to a MongoDB server. This is done internally by the mongodb driver. Decoding happens when the server returns a document so the user gets the data in decoded form. However, If you want to use the encoded forms to store it somewhere else you must know how to do that. There are two classes; B<BSON::Encode> and B<BSON::Decode>.

  my BSON::Document $doc0 .= new: (…);
  my Buf $b = BSON::Encode.new.encode($doc0);
  …
  my BSON::Document $doc1 = BSON::Decode.new.decode($b);

Simple comme bonjour!

You can also give the Buf directly to the C<.new()> method of B<BSON::Document> and it will decode the buffer. As a convenience to other native byte arrays B<CArray[byte]> is also accepted. Furthermore, the method C<.encode()> and C<.decode()> are also available in the BSON::Document class.

  # Create a document and store it in a native array of bytes
  my BSON::Document $doc .= new: (…);
  my $bytes = CArray[byte].new($doc.encode);
<commit_msg>add pod doc<commit_after>#TL:1:BSON::Document:

use v6.d;

#-------------------------------------------------------------------------------
=begin pod

=head1 BSON::Document

BSON Encodable and Decodable document


=head1 Description

Document storage with Hash like behavior used mainly to communicate with a mongodb server. It can also be used as a serialized storage format. The main difference with the Hash is that this class keeps the input order of inserted key-value pairs which is important for the use with mongodb.


=head2 Raku type mapping

To get a proper mapping from Raku types to what BSON can handle, there are some rules and restrictions implemented.

At the top of BSON there is a type called document. (See also L<the bson spec|https://bsonspec.org/spec.html>). This is a representation of a series of keys with values. This could be represented by a B<Hash> but the problem is that the keys of a Hash are not ordered, that is, the order of keys is often not the same as on input while this is a requirement of BSON. So, the best option to represent a document, is a B<List> of B<Pair>s.

The values of a Pair can be simple types like B<Bool>, B<Str>, B<Num> and B<Int>. B<Rat> and B<FatRat> are converted to Num. It can not be another Pair!

An B<Array> may also be used as a value. Its elements have types like those of the values of a Pair. This also leads to the fact that Arrays may be nested.

Binary data can be provided as B<BSON::Binary>. There is support for several types of binary data. When a general type of binary data is offered, you can also give a B<Buf>. The Buf will then be converted into a BSON::Binary.

It is possible to run Javascript code on a MongoDB server so there is a type for it in the BSON specification. For Raku, there is a type called B<BSON::Javascript> to handle that type.

BSON is a specification for C-type data so the Raku types are converted into native types. Integers are only signed integers. There are B<int32> and B<int64> types. Internally this class will take the smallest size when encoding the integer. Int can handle very large numbers. When they are too big for an int64, they are truncated. Num is always converted to a B<double> and boolean values to int32 C<0> or C<1>.

The BSON::Document class has a role B<BSON::Ordered> to handle the types. This means that you can create your own class and do almost the same things. What it does not handle is the encoding and decoding and the initialization like done for BSON::Document. So, in short, only the assignments are supported.

  class A does BSON::Ordered { }
  my A $a .= new;
  $a<k1> = 10;
  my BSON::Document $d1 .= new($a);

=head2 Assignments of values to keys

Next to initialization, one can add or change the data using assignments to a BSON::Document. This is all defined by the BSON::Ordered role which the BSON::Document uses.

Some examples are;

  my BSON::Document $doc .= new;
  $doc<key1> = 10;
  $doc<key2> = [ 0, 'Foo', π, True, [ 'a', ( :x<a>, :y([-1, -2]) ) ], 20e3 ];
  $doc<key3> =  (:p<z>, :x(2e-2));
  $doc<key4><a><b> = 10;
  $doc<key5> = :a<c>;

The Pair used in the last example, is automatically converted to a List of Pair having one Pair.

To see what the document became so far, call C<.raku()> or C<.perl()>.

For the above examples it will show;

  BSON::Document.new: (
    key1 => 10,
    key2 => [
      0,
      'Foo',
      3.141592653589793,
      True,
      [
        'a',
         (
          x => 'a',
          y => [
            -1,
            -2,
          ],
        ),
      ],
      20000,
    ],
    key3 =>  (
      p => 'z',
      x => 0.02,
    ),
    key4 =>  (
      a =>  (
        b => 10,
      ),
    ),
    key5 =>  (
      a => 'c',
    ),
  );


=head2 Encoding and decoding

Encoding is needed when it is sent to a MongoDB server. This is done internally by the mongodb driver. Decoding happens when the server returns a document so the user gets the data in decoded form. However, If you want to use the encoded forms to store it somewhere else you must know how to do that. There are two classes; B<BSON::Encode> and B<BSON::Decode>.

  my BSON::Document $doc0 .= new: (…);
  my Buf $b = BSON::Encode.new.encode($doc0);
  …
  my BSON::Document $doc1 = BSON::Decode.new.decode($b);

Simple comme bonjour!

You can also give the Buf directly to the C<.new()> method of B<BSON::Document> and it will decode the buffer. As a convenience to other native byte arrays B<CArray[byte]> is also accepted. Furthermore, the method C<.encode()> and C<.decode()> are also available in the BSON::Document class.

  # Create a document and store it in a native array of bytes
  my BSON::Document $doc .= new: (…);
  my $bytes = CArray[byte].new($doc.encode);

{'fraction_non_alphanumeric': 0.09885572726200306, 'fraction_numerical': 0.011796626164916833, 'mean_word_length': 3.253888610135474, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 83, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1640429', 'n_tokens_mistral': 2513, 'n_tokens_neox': 2398, 'n_words': 1316}