<commit_before>import collections

__all__ = ["H11Connection"]

from .util import asciify
from .events import *
from .parser import HttpParser

# XX FIXME: sendfile support?
#   maybe switch data_to_send to returning an iterable of stuff-to-do, which
#     could be a mix of bytes-likes, sendfile objects, and CloseSocket
#   and Data could accept sendfile objects as a .data field

# XX FIXME: once we have the high-level state machine in place, using it to
# drive our own lowlevel parser would not be that hard... it already knows
# (better than libhttp_parser!) things like "next is a chunked-encoded body",
# and if we are allowed to buffer and have context then HTTP tokenization is
# pretty trivial I think? and everything above tokenization we are already
# handling. basically the primitive we need is length-bounded regexp matching:
# try to match regexp, if it fails then wait for more data to arrive in
# buffer, raise HttpParseError if the buffer is already longer than the max
# permitted length.

# XX FIXME: it would be nice to support sending Connection: keep-alive headers
# back to HTTP 1.0 clients who have requested this:
#   https://en.wikipedia.org/wiki/HTTP_persistent_connection#HTTP_1.0
# though I'm not 100% sure whether this actually does anything.

# XX FIXME: replace our RuntimeError's with some more specific "you are doing
# HTTP wrong" error like H2's ProtocolError.

# XX FIXME: we should error out if people try to pipeline as a client, since
# otherwise we will give silently subtly wrong behavior
#
# XX FIXME: better tracking for when one has really and truly processed a
# single request/response pair would be good.
#
# XX FIXME: might at that point make sense to split the client and server into
# two separate classes?

# headers to consider auto-supporting at the high-level:
# - Date: https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7231.html#header.date
#     MUST be sent by origin servers who know what time it is
#     (clients don't bother)
# - Server
# - automagic compression

# should let handlers control timeouts

################################################################
#
# Higher level stuff:
# - Timeouts: waiting for 100-continue, killing idle keepalive connections,
#     killing idle connections in general
#     basically just need a timeout when we block on read, and if it times out
#       then we close. should be settable in the APIs that block on read
#       (e.g. iterating over body).
# - Expect:
#     https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7231.html#rfc.section.5.1.1
#   This is tightly integrated with flow control, not a lot we can do, except
#   maybe provide a method to be called before blocking waiting for the
#   request body?
# - Sending an error when things go wrong (esp. 400 Bad Request)
#
# - Transfer-Encoding: compress, gzip
#   - but unfortunately, libhttp_parser doesn't support these at all (just
#     ignores the Transfer-Encoding field and doesn't even do chunked parsing,
#     so totally unfixable)
#       https://stackapps.com/questions/916/why-content-encoding-gzip-rather-than-transfer-encoding-gzip
#     So... this sucks, but I guess we don't support it either.

# rules for upgrade are:
# - when you get back an message-complete, you have to check for the upgrade
#   flag
# - if it's set, then there's also some trailing-data provided
# - if you continue doing HTTP on the same socket, then you have to
#   receive_data that trailing data again
# maybe we should make this an opt-in thing in the constructor -- you have to
# say whether you're prepared for upgrade handling?
#
# also, after sending a message-complete on the server you then have to
# immediately call receive_data even if there's no new bytes to pass, because
# more responses might have been pipelined up.

# server-side state machine:
# WAIT -> got request line + headers          ->            read body
#                             -> waiting for 100-continue ->
#                             -> response sent
#                                            -> finish response
#     need response sent + read body before can process the next request

# client-side state machine:
# send request line + headers
#    (-> wait for 100-continue ->)
#          send body
#    read response
# can't send new request until after have read response (crucial for current
# method tracking! and pipelining is not important to support)
#
# and NB 1xx responses don't count as having read the response (though
# libhttp_parser will parse them as a complete message-begin
# ... message-complete cycle) in fact it seems to have a bug where it doesn't
# know that 100 responses can't contain a body... probably harmless but eh.
#
# also note that client is allowed to just go ahead sending body even after
# saying Expect: 100-continue and seeing no response
# this is needed to handle ancient servers that don't know about Expect:
# 100-continue (maybe they don't exist anymore?)
# in fact even if gets a 4xx response still has to "send the rest" -- if using
# chunked it could send 0 bytes (but a common 4xx response is "your
# content-length is too big", so maybe you want to use content-length!), or
# could just close the connection (or server might just close the connection),
# but these are your options.
# - seeing a response when in wait-for-continue state just moves you to
# send-body state

# Connection shutdown is tricky. Quoth RFC 7230:
#
# "If a server performs an immediate close of a TCP connection, there is a
# significant risk that the client will not be able to read the last HTTP
# response. If the server receives additional data from the client on a fully
# closed connection, such as another request that was sent by the client
# before receiving the server's response, the server's TCP stack will send a
# reset packet to the client; unfortunately, the reset packet might erase the
# client's unacknowledged input buffers before they can be read and
# interpreted by the client's HTTP parser.
#
# "To avoid the TCP reset problem, servers typically close a connection in
# stages. First, the server performs a half-close by closing only the write
# side of the read/write connection. The server then continues to read from
# the connection until it receives a corresponding close by the client, or
# until the server is reasonably certain that its own TCP stack has received
# the client's acknowledgement of the packet(s) containing the server's last
# response. Finally, the server fully closes the connection."
#
# So this needs shutdown(2). This is what data_to_send's close means -- this
# complicated close dance.

# EndOfMessage is tricky:
# - upgrade trailing data handling<commit_msg>checkpoint: starting to work seriously on state tracking<commit_after>import collections

__all__ = ["H11Connection"]

from .util import asciify
from .events import *
from .parser import HttpParser

# XX FIXME: sendfile support?
#   maybe switch data_to_send to returning an iterable of stuff-to-do, which
#     could be a mix of bytes-likes, sendfile objects, and CloseSocket
#   and Data could accept sendfile objects as a .data field

# XX FIXME: once we have the high-level state machine in place, using it to
# drive our own lowlevel parser would not be that hard... it already knows
# (better than libhttp_parser!) things like "next is a chunked-encoded body",
# and if we are allowed to buffer and have context then HTTP tokenization is
# pretty trivial I think? and everything above tokenization we are already
# handling. basically the primitive we need is length-bounded regexp matching:
# try to match regexp, if it fails then wait for more data to arrive in
# buffer, raise HttpParseError if the buffer is already longer than the max
# permitted length.

# XX FIXME: it would be nice to support sending Connection: keep-alive headers
# back to HTTP 1.0 clients who have requested this:
#   https://en.wikipedia.org/wiki/HTTP_persistent_connection#HTTP_1.0
# though I'm not 100% sure whether this actually does anything.

# XX FIXME: replace our RuntimeError's with some more specific "you are doing
# HTTP wrong" error like H2's ProtocolError.

# XX FIXME: we should error out if people try to pipeline as a client, since
# otherwise we will give silently subtly wrong behavior
#
# XX FIXME: better tracking for when one has really and truly processed a
# single request/response pair would be good.
#
# XX FIXME: might at that point make sense to split the client and server into
# two separate classes?

# headers to consider auto-supporting at the high-level:
# - Date: https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7231.html#header.date
#     MUST be sent by origin servers who know what time it is
#     (clients don't bother)
# - Server
# - automagic compression

# should let handlers control timeouts

################################################################
#
# Higher level stuff:
# - Timeouts: waiting for 100-continue, killing idle keepalive connections,
#     killing idle connections in general
#     basically just need a timeout when we block on read, and if it times out
#       then we close. should be settable in the APIs that block on read
#       (e.g. iterating over body).
# - Expect:
#     https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7231.html#rfc.section.5.1.1
#   This is tightly integrated with flow control, not a lot we can do, except
#   maybe provide a method to be called before blocking waiting for the
#   request body?
# - Sending an error when things go wrong (esp. 400 Bad Request)
#
# - Transfer-Encoding: compress, gzip
#   - but unfortunately, libhttp_parser doesn't support these at all (just
#     ignores the Transfer-Encoding field and doesn't even do chunked parsing,
#     so totally unfixable)
#       https://stackapps.com/questions/916/why-content-encoding-gzip-rather-than-transfer-encoding-gzip
#     So... this sucks, but I guess we don't support it either.

# rules for upgrade are:
# - when you get back an message-complete, you have to check for the upgrade
#   flag
# - if it's set, then there's also some trailing-data provided
# - if you continue doing HTTP on the same socket, then you have to
#   receive_data that trailing data again
# maybe we should make this an opt-in thing in the constructor -- you have to
# say whether you're prepared for upgrade handling?
#
# also, after sending a message-complete on the server you then have to
# immediately call receive_data even if there's no new bytes to pass, because
# more responses might have been pipelined up.

# Client sends (regex):
#   Request Data* EndOfMessage
# Server sends (regex):
#   InformationalResponse* Response Data* EndOfMessage
# They are linked in two places:
# - client has wait-for-100-continue state (not shown) where the transition
#   out is receiving a InformationalResponse or Response (or timeout)
# - *both* EndOfMessage's have to arrive before *either* machine returns to
#   the start state.

# We model the joint state of the client and server as a pair of finite state
# automata: one for the client and one for the server. Transitions in each
# machine can be triggered by either local or remote events. (For example, the
# client sending a Request triggers both the client to move to SENDING-BODY
# and the server to move to SENDING-RESPONSE.)

# We reuse the Client and Server class objects as sentinel values for
# referring to the two parties.
class Client(Enum):
    IDLE = 1
    WAIT_FOR_100 = 2
    SENDING_BODY = 3
    DONE = 4
    CLOSED = 5

class Server(Enum):
    WAIT = 1
    SENDING_RESPONSE = 2
    SENDING_BODY = 3
    DONE = 4
    CLOSED = 5

def _get_next_client_state_for_request(request):
    # if this request has no body, go straight to DONE
    # if this request has Expect: 100-continue, go to WAIT_FOR_100
    # otherwise, go to SENDING_BODY

def _get_next_server_state_for_response(response):
    # if this response has no body, go straight to DONE
    # otherwise, go to SENDING_BODY

client_transitions = {
    Client.IDLE: {
        (Client, Request): _get_next_client_state_for_request,
    },
    Client.WAIT_FOR_100: {
        (Server, InformationalResponse): Client.SENDING_BODY,
        (Server, Response): Client.SENDING_BODY,
        (Client, Data): Client.SENDING_BODY,
        (Client, EndOfMessage): Client.DONE,
    }
    Client.SENDING_BODY: {
        (Client, Data): Client.SENDING_BODY,
        (Client, EndOfMessage): Client.DONE,
    },
}

server_transitions = {
    Server.WAIT_FOR_REQUEST: {
        (Client, Request): Server.SENDING_RESPONSE,
    },
    Server.SENDING_RESPONSE: {
        (Server, InformationalResponse): Server.SENDING_RESPONSE,
        (Server, Response): _get_next_server_state_for_response,
    },
    Server.SENDING_BODY: {
        (Server, Data): Server.SENDING_BODY,
        (Server, EndOfMessage): Server.DONE,
    },
}

class PartyMachine:
    def __init__(self, party, initial_state, transitions):
        self.party = party
        self.initial_state = initial_state
        self.transitions = transitions
        self.reset()

    def process_event(self, party, event):
        key = (party, type(event))
        new_state = self.transitions[self.state].get(key)
        if new_state is None:
            if party is not self.party:
                return
            else:
                raise RuntimeError(
                    "illegal event {} in state {}".format(key, self.state))
        if callable(new_state):
            new_state = new_state(event)
        self.state = new_state

    def reset(self):
        self.state = self.initial_state

class ConnectionState:
    def __init__(self):
        self._client_machine = PartyMachine(Client.IDLE, client_transitions)
        self._server_machine = PartyMachine(Server.WAIT, server_transitions)
        self.request = None
        self.response = None

    @property
    def client_state(self):
        return self._client_machine.state

    @property
    def server_state(self):
        return self._server_machine.state

    def process_event(self, party, event):
        self.client_machine.receive_event(party, event)
        self.server_machine.receive_event(party, event)
        if isinstance(event, Request):
            self.request = event
        if isinstance(event, Response):
            self.response = event
        if (self.client_state is Client.DONE
            and self.server_state is Server.DONE):
            # XX FIXME either move them to CLOSING state or reset them
            self._client_machine.reset()
            self._server_machine.reset()
            self.request = None
            self.response = None

# Connection shutdown is tricky. Quoth RFC 7230:
#
# "If a server performs an immediate close of a TCP connection, there is a
# significant risk that the client will not be able to read the last HTTP
# response. If the server receives additional data from the client on a fully
# closed connection, such as another request that was sent by the client
# before receiving the server's response, the server's TCP stack will send a
# reset packet to the client; unfortunately, the reset packet might erase the
# client's unacknowledged input buffers before they can be read and
# interpreted by the client's HTTP parser.
#
# "To avoid the TCP reset problem, servers typically close a connection in
# stages. First, the server performs a half-close by closing only the write
# side of the read/write connection. The server then continues to read from
# the connection until it receives a corresponding close by the client, or
# until the server is reasonably certain that its own TCP stack has received
# the client's acknowledgement of the packet(s) containing the server's last
# response. Finally, the server fully closes the connection."
#
# So this needs shutdown(2). This is what data_to_send's close means -- this
# complicated close dance.

# EndOfMessage is tricky:
# - upgrade trailing data handling
{'fraction_non_alphanumeric': 0.07525062656641604, 'fraction_numerical': 0.007142857142857143, 'mean_word_length': 3.4459610027855154, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 11, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3051342', 'n_tokens_mistral': 4415, 'n_tokens_neox': 4231, 'n_words': 2010}