<commit_before>#lang scribble/doc
@(require "common.ss"
          scribble/bnf
          (for-label net/url
                     net/url-unit
                     net/url-sig
                     net/head
                     net/uri-codec))

@title{URLs and HTTP}

@defmodule[net/url]{The @schememodname[net/url] library provides
utilities to parse and manipulate URIs, as specified in RFC 2396
@cite["RFC2396"], and to use the HTTP protocol.}

To access the text of a document from the web, first obtain its URL as
a string. Convert the address into a @scheme[url] structure using
@scheme[string->url]. Then, open the document using
@scheme[get-pure-port] or @scheme[get-impure-port], depending on
whether or not you wish to examine its MIME headers.  At this point,
you have a regular input port with which to process the document, as with
any other file. 

Currently the only supported protocols are @scheme["http"] and
sometimes @scheme["file"]. See also the @schememodname[net/news]
library.

@section{URL Structure}

@declare-exporting[net/url-structs net/url]

@defmodule*/no-declare[(net/url-structs)]{The URL structure types are
provided by the @schememodname[net/url-structs] library, and
re-exported by @schememodname[net/url].}

@; ----------------------------------------


@defstruct[url ([scheme (or/c false/c string?)]
                [user (or/c false/c string?)]
                [host (or/c false/c string?)]
                [port (or/c false/c exact-nonnegative-integer?)]
                [path-absolute? boolean?]
                [path (listof path/param?)]
                [query (listof (cons/c symbol? (or/c false/c string?)))]
                [fragment (or/c false/c string?)])]{

The basic structure for all URLs, hich is explained in RFC 3986
@cite["RFC3986"]. The following diagram illustrates the parts:

@verbatim|{
  http://sky@www:801/cgi-bin/finger;xyz?name=shriram;host=nw#top
  {-1}   {2} {3} {4}{---5---------} {6} {----7-------------} {8}

  1 = scheme, 2 = user, 3 = host, 4 = port,
  5 = path (two elements),  6 = param (of second path element),
  7 = query, 8 = fragment
}|

The strings inside the @scheme[user], @scheme[path], @scheme[query],
and @scheme[fragment] fields are represented directly as Scheme
strings, without URL-syntax-specific quoting. The procedures
@scheme[string->url] and @scheme[url->string] translate encodings such
as @litchar{%20} into spaces and back again.

By default, query associations are parsed with either @litchar{;} or
@litchar{&} as a separator, and they are generated with @litchar{&} as
a separator. The @scheme[current-alist-separator-mode] parameter
adjusts the behavior.

An empty string at the end of the @scheme[path] list corresponds to a
URL that ends in a slash. For example, the result of
@scheme[(string->url "http://www.drscheme.org/a/")] has a
@scheme[path] field with strings @scheme["a"] and @scheme[""], while<commit_msg>add #:indent to verbatim<commit_after>#lang scribble/doc
@(require "common.ss"
          scribble/bnf
          (for-label net/url
                     net/url-unit
                     net/url-sig
                     net/head
                     net/uri-codec))

@title{URLs and HTTP}

@defmodule[net/url]{The @schememodname[net/url] library provides
utilities to parse and manipulate URIs, as specified in RFC 2396
@cite["RFC2396"], and to use the HTTP protocol.}

To access the text of a document from the web, first obtain its URL as
a string. Convert the address into a @scheme[url] structure using
@scheme[string->url]. Then, open the document using
@scheme[get-pure-port] or @scheme[get-impure-port], depending on
whether or not you wish to examine its MIME headers.  At this point,
you have a regular input port with which to process the document, as with
any other file. 

Currently the only supported protocols are @scheme["http"] and
sometimes @scheme["file"]. See also the @schememodname[net/news]
library.

@section{URL Structure}

@declare-exporting[net/url-structs net/url]

@defmodule*/no-declare[(net/url-structs)]{The URL structure types are
provided by the @schememodname[net/url-structs] library, and
re-exported by @schememodname[net/url].}

@; ----------------------------------------


@defstruct[url ([scheme (or/c false/c string?)]
                [user (or/c false/c string?)]
                [host (or/c false/c string?)]
                [port (or/c false/c exact-nonnegative-integer?)]
                [path-absolute? boolean?]
                [path (listof path/param?)]
                [query (listof (cons/c symbol? (or/c false/c string?)))]
                [fragment (or/c false/c string?)])]{

The basic structure for all URLs, hich is explained in RFC 3986
@cite["RFC3986"]. The following diagram illustrates the parts:

@verbatim[#:indent 2]|{
  http://sky@www:801/cgi-bin/finger;xyz?name=shriram;host=nw#top
  {-1}   {2} {3} {4}{---5---------} {6} {----7-------------} {8}

  1 = scheme, 2 = user, 3 = host, 4 = port,
  5 = path (two elements),  6 = param (of second path element),
  7 = query, 8 = fragment
}|

The strings inside the @scheme[user], @scheme[path], @scheme[query],
and @scheme[fragment] fields are represented directly as Scheme
strings, without URL-syntax-specific quoting. The procedures
@scheme[string->url] and @scheme[url->string] translate encodings such
as @litchar{%20} into spaces and back again.

By default, query associations are parsed with either @litchar{;} or
@litchar{&} as a separator, and they are generated with @litchar{&} as
a separator. The @scheme[current-alist-separator-mode] parameter
adjusts the behavior.

An empty string at the end of the @scheme[path] list corresponds to a
URL that ends in a slash. For example, the result of
@scheme[(string->url "http://www.drscheme.org/a/")] has a
@scheme[path] field with strings @scheme["a"] and @scheme[""], while
{'fraction_non_alphanumeric': 0.14214932513241074, 'fraction_numerical': 0.012813941568426449, 'mean_word_length': 3.017844886753603, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1060643', 'n_tokens_mistral': 1877, 'n_tokens_neox': 1749, 'n_words': 637}