<jupyter_start><jupyter_text>Example BSDF in Python<jupyter_code>import os
import bsdf
filename = os.path.expanduser('~/foo.bsdf')<jupyter_output><empty_output><jupyter_text>Simple useHere, we define some data, encode it, and decode back.<jupyter_code>data = ['hello', 3, 4, None, {'foo': [1, 2], 'bar': b'xxxxx'}]
bb = bsdf.encode(data)
bb
bsdf.decode(bb)<jupyter_output><empty_output><jupyter_text>The above encodes/decodes to/from bytes. We can also use file objects. `bsdf.save()` and `bsdf.load()` also support file names.<jupyter_code>bsdf.save(filename, data)
bsdf.load(filename)<jupyter_output><empty_output><jupyter_text>Custom extensionsExtensions are easy to write.<jupyter_code># For the sake of example, let's write an extension that encodes complex numbers.
# (Spoiler: this is the actual code for the standard complex extention)
class ComplexExtension(bsdf.Extension):

    name = 'c'
    cls = complex

    def encode(self, s, v):
        return (v.real, v.imag)

    def decode(self, s, v):
        return complex(v[0], v[1])
    
bb = bsdf.encode(3+4j, [ComplexExtension])
print(bb)
bsdf.decode(bb, [ComplexExtension])
# This would work too, because it is a standard extension
# bsdf.decode(bb)
# But this would not
# bsdf.decode(bb, [])<jupyter_output><empty_output><jupyter_text>Use of serializer classThe serializer class is recommended when performance matters. It also defines what options and extensions to use; using the `s` will use compression by default and supports complex numbers. The serializer class has methods `save()`, `load()`, `encode()`, `decode()`, and methods to add/remove extensions. Note that `save()` and `load()` need file objects (not file names).<jupyter_code>s = bsdf.BsdfSerializer([ComplexExtension], compression=1)
s
len(s.encode(b'x'*1000))<jupyter_output><empty_output><jupyter_text>StreamingBSDF support streamed wirting and reading of lists though the `ListStream` class.<jupyter_code>ls = bsdf.ListStream()
ls<jupyter_output><empty_output><jupyter_text>The stream object is used as an element in a BSDF structure. It must be the very last element. In this casewe simply use it as the root element.<jupyter_code>f = open(filename, 'wb')
bsdf.save(f, ls)
f.tell()  # only the BSDF header is written and the identifier for the streaming list<jupyter_output><empty_output><jupyter_text>Now we can write whatever we want (as long as its BSDF serializable)!<jupyter_code>for i in range(4, 9):
    ls.append('a' * i)
for i in range(4, 9):
    ls.append(i)
ls.append('the list can of course contain anything')
ls.append([False, None, 'the end'])<jupyter_output><empty_output><jupyter_text>That should be enough.<jupyter_code>f.close()
# ls.append(3) This raises IOError; cannot write to a close file<jupyter_output><empty_output><jupyter_text>We can load the written data at once:<jupyter_code>bsdf.load(filename)<jupyter_output><empty_output><jupyter_text>Or we can load in streaming mode:<jupyter_code>f = open(filename, 'rb')
ls = bsdf.load(f, load_streaming=True)
ls<jupyter_output><empty_output><jupyter_text>At this point, only the BSDF header and the list identifier have been read:<jupyter_code>f.tell()<jupyter_output><empty_output><jupyter_text>Data is read on demand:<jupyter_code>print(ls.next())
f.tell()
for item in ls:
    print(item)
f.close()<jupyter_output>aaaaa
aaaaaa
aaaaaaa
aaaaaaaa
4
5
6
7
8
the list can of course contain anything
[False, None, 'the end']<jupyter_text>Appending data to a streamSince the stream above is not closed, we can add data to it. The "simple" way is by appending bytes to the file, though this requires knowledge of the BSDF format.<jupyter_code>with open(filename, 'ab') as f:
    f.write(b'v')  # None (void)
    f.write(b'h*\x00')  # The number 42
bsdf.load(filename)<jupyter_output><empty_output><jupyter_text>But we can also use the BSDF API ...<jupyter_code>with open(filename, 'r+b') as f:  # Note the r+ mode
    ls = bsdf.load(f, load_streaming=True)
    for i in ls:  # This looks funny, but we need to move to the end of the stream
        pass
    ls.append([3, 4, 5])
    ls.append('that should be it')    
bsdf.load(filename)<jupyter_output><empty_output><jupyter_text>Modifying binary blobsStart by writing a file that contains bytes.<jupyter_code>bsdf.save(filename, [3, 4, b'xxyyzz'])<jupyter_output><empty_output><jupyter_text>Open in r+ mode to be able to modify the data.<jupyter_code>with open(filename, 'r+b') as f:
    blob = bsdf.load(f, lazy_blob=True)[2]
    blob.seek(0)  # we need to move to the start of the blob
    print(blob.read(2))  # xx
    # Change the last two bytes from zz to aa
    blob.seek(4)
    blob.write(b'aa')
    # Update the checksum, in case there is one
    blob.update_checksum()<jupyter_output>b'xx'<jupyter_text>And check the data ...<jupyter_code>bsdf.load(filename)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11711340206185566, 'fraction_numerical': 0.007422680412371134, 'mean_word_length': 5.520161290322581, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '535552', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1727, 'n_tokens_neox': 1668, 'n_words': 511, 'path': 'python/bsdf_example_python.ipynb', 'repo_name': 'sailfish009/bsdf', 'size': '4850'}