<commit_before># Explanation: HTTP(S) client for PicoLisp

This document provides a short walkthrough of the source code for the [PicoLisp-HTTPS](https://github.com/aw/picolisp-https.git) client.

I won't cover concepts which were covered in previous source code explanations. You can read them here:

* [Nanomsg Explanation](https://github.com/aw/picolisp-nanomsg/blob/master/EXPLAIN.md)
* [JSON Explanation](https://github.com/aw/picolisp-json/blob/master/EXPLAIN.md)

This document is split into a few sections:

1. [Loading and initialization](#loading-and-initialization): Loading files and performing initial work.
2. [Error handling](#error-handling): An idiom for handling errors.
3. [Internal functions](#internal-functions): Destructuring, native C callbacks, and memory management.
  * [making HTTPS requests](#making-https-requests)
  * [parsing HTTPS responses](#parsing-https-responses)
  * [cleaning up errors](#cleaning-up-errors)

Make sure you read the [README](README.md) to get an idea of what this library does.

# Loading and initialization

We've made some changes to how we load files across all libraries.

### Loading

[PicoLisp](http://picolisp.com) loads files from the _current working directory_ [pwd](http://software-lab.de/doc/refP.html#pwd), which is in relation to where you ran the command:

```lisp
alex@dev-box:~/picolisp-https$ pil +
: (pwd)
-> "/home/aw/picolisp-https"
```

So far so good, but what happens when the file you load also loads a file in a different directory? Depending if the path is relative or absolute, you will not necessarily get what you want.

To fix this, we use [file](http://software-lab.de/doc/refF.html#file):

```lisp
*Https              (pack (car (file)) "lib/libneon.so")
```

What this does is load the file `lib/libneon.so` relative to the file that's loading it. 

We use this technique further down as well:

```lisp
# ffi-bindings
(load (pack (car (file)) "ffi.l"))

# internal
(load (pack (car (file)) "internal.l"))
```

Perhaps there should be a `(cwd)` primitive for that? ;)

### Initialization

There is a concept of `constructors` in PicoLisp, but it's only used with classes and objects. We're trying to be functional here.

Our approach is simple: perform initialization tasks after loading all the necessary files.

```lisp
(when (=0 (ne-has-support *NE_FEATURE_SSL))
      (throw-error NIL "Missing support for SSL/TLS") )
```

What we've done here is try to ensure `SSL` is compiled into the shared library. If it's not, an error is thrown. [Error handling](#error-handling) is explained in the next section.

We also ensure to [seed](http://software-lab.de/doc/refS.html#seed) some random data from the system's PRNG:

```lisp
(seed (in "/dev/urandom" (rd 20)))
```

This tries to obtain `20 bytes` from `/dev/urandom` using [rd](http://software-lab.de/doc/refR.html#rd), a function for reading raw bytes from an input stream, and initializes the seed with it.

# Error handling

PicoLisp provides us with a few ways to handle errors, so why not use them to our advantage?

My idea was:

1. Throw errors in the library, but don't quit/exit disgracefully (a.k.a. be nice).
2. Provide a _type_ of error, and a brief message explaining what happened.
3. Allow the user to catch the errors outside the library.

```lisp
[de throw-error (Session Message)
  (throw 'InternalError (cons 'HttpsError (if Session
                                              (ne-get-error Session)
                                              Message ]
```

The **Neon C library** has a function `(ne-get-error)` which returns a string containing an error message (if any). Sometimes, we want to provide our own error message though.

In the `(throw-error)` function, we satisfy the first two requirements by using [throw](http://software-lab.de/doc/refT.html#throw) to send an `'InternalError`, along with a [cons](http://software-lab.de/doc/refC.html#cons) pair containing the `'HttpsError` _type_ in the [car](http://software-lab.de/doc/refC.html#car) and the error _message_ in the [cdr](http://software-lab.de/doc/refC.html#cdr).<commit_msg>Clear up confusion about PRNG (thnx <NAME>)<commit_after># Explanation: HTTP(S) client for PicoLisp

This document provides a short walkthrough of the source code for the [PicoLisp-HTTPS](https://github.com/aw/picolisp-https.git) client.

I won't cover concepts which were covered in previous source code explanations. You can read them here:

* [Nanomsg Explanation](https://github.com/aw/picolisp-nanomsg/blob/master/EXPLAIN.md)
* [JSON Explanation](https://github.com/aw/picolisp-json/blob/master/EXPLAIN.md)

This document is split into a few sections:

1. [Loading and initialization](#loading-and-initialization): Loading files and performing initial work.
2. [Error handling](#error-handling): An idiom for handling errors.
3. [Internal functions](#internal-functions): Destructuring, native C callbacks, and memory management.
  * [making HTTPS requests](#making-https-requests)
  * [parsing HTTPS responses](#parsing-https-responses)
  * [cleaning up errors](#cleaning-up-errors)

Make sure you read the [README](README.md) to get an idea of what this library does.

# Loading and initialization

We've made some changes to how we load files across all libraries.

### Loading

[PicoLisp](http://picolisp.com) loads files from the _current working directory_ [pwd](http://software-lab.de/doc/refP.html#pwd), which is in relation to where you ran the command:

```lisp
alex@dev-box:~/picolisp-https$ pil +
: (pwd)
-> "/home/aw/picolisp-https"
```

So far so good, but what happens when the file you load also loads a file in a different directory? Depending if the path is relative or absolute, you will not necessarily get what you want.

To fix this, we use [file](http://software-lab.de/doc/refF.html#file):

```lisp
*Https              (pack (car (file)) "lib/libneon.so")
```

What this does is load the file `lib/libneon.so` relative to the file that's loading it. 

We use this technique further down as well:

```lisp
# ffi-bindings
(load (pack (car (file)) "ffi.l"))

# internal
(load (pack (car (file)) "internal.l"))
```

Perhaps there should be a `(cwd)` primitive for that? ;)

### Initialization

There is a concept of `constructors` in PicoLisp, but it's only used with classes and objects. We're trying to be functional here.

Our approach is simple: perform initialization tasks after loading all the necessary files.

```lisp
(when (=0 (ne-has-support *NE_FEATURE_SSL))
      (throw-error NIL "Missing support for SSL/TLS") )
```

What we've done here is try to ensure `SSL` is compiled into the shared library. If it's not, an error is thrown. [Error handling](#error-handling) is explained in the next section.

We also ensure to [seed](http://software-lab.de/doc/refS.html#seed) some random data from the system's random pool:

```lisp
(seed (in "/dev/urandom" (rd 20)))
```

This tries to obtain `20 bytes` from `/dev/urandom` using [rd](http://software-lab.de/doc/refR.html#rd), a function for reading raw bytes from an input stream, and initializes the seed with it.

# Error handling

PicoLisp provides us with a few ways to handle errors, so why not use them to our advantage?

My idea was:

1. Throw errors in the library, but don't quit/exit disgracefully (a.k.a. be nice).
2. Provide a _type_ of error, and a brief message explaining what happened.
3. Allow the user to catch the errors outside the library.

```lisp
[de throw-error (Session Message)
  (throw 'InternalError (cons 'HttpsError (if Session
                                              (ne-get-error Session)
                                              Message ]
```

The **Neon C library** has a function `(ne-get-error)` which returns a string containing an error message (if any). Sometimes, we want to provide our own error message though.

In the `(throw-error)` function, we satisfy the first two requirements by using [throw](http://software-lab.de/doc/refT.html#throw) to send an `'InternalError`, along with a [cons](http://software-lab.de/doc/refC.html#cons) pair containing the `'HttpsError` _type_ in the [car](http://software-lab.de/doc/refC.html#car) and the error _message_ in the [cdr](http://software-lab.de/doc/refC.html#cdr).
{'fraction_non_alphanumeric': 0.11502603220728902, 'fraction_numerical': 0.002663760745853009, 'mean_word_length': 3.957983193277311, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1391628', 'n_tokens_mistral': 2570, 'n_tokens_neox': 2455, 'n_words': 992}