<commit_before><HTML>
<HEAD>
<TITLE>PostgreSQL Backend Flowchart</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#FF0000" VLINK="#A00000" ALINK="#0000FF">
<H1 ALIGN=CENTER>
PostgreSQL Backend Flowchart
</H1>
<H2 ALIGN=CENTER>
by <NAME>
</H2 ALIGN=CENTER>
<P>
Queries come into the backend via data packets coming in through TCP/IP
and Unix Domain sockets.   They are loaded into a string, and passed to
the
<A HREF="../../backend/parser">parser,</A> where the lexical scanner,
<A HREF="../../backend/parser/scan.l">scan.l,</A>
breaks the query up into tokens(words).  The parser
uses
<A HREF="../../backend/parser/gram.y">gram.y</A> and the tokens to
identify the query type, and load the proper query-type-specific
structure, like
<A HREF="../../include/nodes/parsenodes.h">CreateStmt or SelectStmt.</A>
<P>
The query is then identified as a <I>Utility</I> function or a more
complex query.  <I>Utility</I> queries are processed by a
query-type-specific function in <A HREF="../../backend/commands">
commands.</A> Complex queries, like <I>SELECT, UPDATE,</I> and
<I>DELETE</I> require much more handling.  
<P>
The parser takes the complex queries, and creates a
<A HREF="../../include/nodes/parsenodes.h">Query</A> structure that
contains all the elements used by complex queries.  Query.qual holds the
WHERE clause qualification, which is filled in by

<A HREF="../../backend/parser/parse_clause.c">transformWhereClause().</A>
Each table is represented by a <A HREF="../../include/nodes/parsenodes.h">
RangeTableEntry,</A>
and they are linked together to form the <I>range table</I> for the
query, and is generated by <A HREF="../../backend/parser/parse_clause.c">
makeRangeTable().</A>  Query.rtable holds the queries range table.
<P>
Certain queries, like SELECT, return columns of data.  Other queries,
like INSERT and UPDATE, specify the columns modified by the query. 
These columns references are converted to <A
HREF="../../include/nodes/primnodes.h"> Resdom</A> entries, which are
linked together to make up the <I>target list</I> of the query. The
target list is stored in Query.targetList, and is generated by
<A HREF="../../backend/parser/parse_target.c">transformTargetList().</A>
<P>
Other query elements, like aggregates(SUM()), GROUP BY, ORDER BY are
also stored in their own fields.
<P>
The next step is for the Query to be modified by any VIEWS or RULES that
may apply to the query.  This is performed by the <A
HREF="../../backend/rewrite">rewrite</A> system.
<P>
The optimizer takes the Query structure, and generates an optimal
<A HREF="../..//include/nodes/plannodes.h">Plan</A> containing primitive
operations to be performed by the executor to complete the query.  The
<A HREF="../../backend/optimizer/path">path</A> module
determines the table join order and join type of each of the tables in
the RangeTable, using Query.qual(WHERE clause) to consider optimal index
usage.
<P>
The Plan is then passed to the <A
HREF="../../backend/executor">executor</A> for execution, and the result
is returned to the client.
<P>
There are many other modules that support this basic functionality. 
They can be accessed by clicking on the flowchart.
<P>
Another area of interest is the shared memory area, containing
table data/index blocks, locks, and backend information:
<UL> 
<LI>ShmemIndex - contains an index of all other shared memory
structures, allowing quick lookup of other structure locations in shared
memory
<LI>Buffer Descriptors - control header for shared memory buffer block
<LI>Buffer Blocks - block of table/index data shared by all backends
<LI>Shared Buf Lookup Table - lookup to see if a requested buffer
is already in the shared memory area
<LI>LockTable - lock table structure, specifiying table, lock types, and
backends holding or waiting on lock
<LI>LockTable (lock hash) - lookup of LockTable structures using
table name
<LI>LockTable (xid hash) - lookup of LockTable structures using
transaction id
<LI>Proc Header - information about each backend, including locks held/waiting, 
indexed by process id
</UL>
Each structure is created by calling <A
HREF="../../backend/storage/ipc/shmem.c"> ShmemInitStruct().</A>
<HR>
<CENTER>
<EM><BIG>
Click on an item to see more detail or click
<A HREF="backend_dirs.html">here</a> to see the full index.
</EM></BIG>
<BR>
<BR>
<IMG src="flow.jpg" usemap="#flowmap">
</CENTER>
<MAP name="flowmap">
<AREA COORDS="290,10,450,50" HREF="backend_dirs.html#main">
<AREA COORDS="550,10,710,50" HREF="backend_dirs.html#bootstrap">
<AREA COORDS="290,90,450,130," HREF="backend_dirs.html#postmaster">
<AREA COORDS="550,90,710,130," HREF="backend_dirs.html#libpq">
<AREA COORDS="290,170,450,210" HREF="backend_dirs.html#tcop">
<AREA COORDS="550,170,710,210" HREF="backend_dirs.html#tcop">
<AREA COORDS="290,270,450,310" HREF="backend_dirs.html#parser">
<AREA COORDS="290,350,450,390" HREF="backend_dirs.html#tcop">
<AREA COORDS="290,430,450,470" HREF="backend_dirs.html#optimizer"><commit_msg>Update flowchart and backend descriptions.<commit_after><HTML>
<HEAD>
<TITLE>PostgreSQL Backend Flowchart</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#FF0000" VLINK="#A00000" ALINK="#0000FF">
<H1 ALIGN=CENTER>
PostgreSQL Backend Flowchart
</H1>
<H2 ALIGN=CENTER>
by <NAME>
</H2 ALIGN=CENTER>
<P>
Queries come into the backend via data packets coming in through TCP/IP
and Unix Domain sockets.   They are loaded into a string, and passed to
the
<A HREF="../../backend/parser">parser,</A> where the lexical scanner,
<A HREF="../../backend/parser/scan.l">scan.l,</A>
breaks the query up into tokens(words).  The parser
uses
<A HREF="../../backend/parser/gram.y">gram.y</A> and the tokens to
identify the query type, and load the proper query-type-specific
structure, like
<A HREF="../../include/nodes/parsenodes.h">CreateStmt or SelectStmt.</A>
<P>
The query is then identified as a <I>Utility</I> function or a more
complex query.  <I>Utility</I> queries are processed by a
query-type-specific function in <A HREF="../../backend/commands">
commands.</A> Complex queries, like <I>SELECT, UPDATE,</I> and
<I>DELETE</I> require much more handling.  
<P>
The parser takes the complex queries, and creates a
<A HREF="../../include/nodes/parsenodes.h">Query</A> structure that
contains all the elements used by complex queries.  Query.qual holds the
WHERE clause qualification, which is filled in by
<A HREF="../../backend/parser/parse_clause.c">transformWhereClause().</A>
Each table referenced in the query is represented by a <A
HREF="../../include/nodes/parsenodes.h"> RangeTableEntry,</A> and they
are linked together to form the <I>range table</I> of the query, which is
generated by <A HREF="../../backend/parser/parse_clause.c">
makeRangeTable().</A>  Query.rtable holds the queries range table.
<P>
Certain queries, like SELECT, return columns of data.  Other queries,
like INSERT and UPDATE, specify the columns modified by the query. 
These column references are converted to <A
HREF="../../include/nodes/primnodes.h"> Resdom</A> entries, which are
linked together to make up the <I>target list</I> of the query. The
target list is stored in Query.targetList, which is generated by
<A HREF="../../backend/parser/parse_target.c">transformTargetList().</A>
<P>
Other query elements, like aggregates(SUM()), GROUP BY, ORDER BY are
also stored in their own Query fields.
<P>
The next step is for the Query to be modified by any VIEWS or RULES that
may apply to the query.  This is performed by the <A
HREF="../../backend/rewrite">rewrite</A> system.
<P>
The <A HREF="../../backend/optimizer">optimizer</A> takes the Query
structure, and generates an optimal
<A HREF="../..//include/nodes/plannodes.h">Plan</A> containing the
primitive operations to be performed by the executor to execute the
query.  The <A HREF="../../backend/optimizer/path">path</A> module
determines the best table join order and join type of each table in the
RangeTable, using Query.qual(WHERE clause) to consider optimal index
usage.
<P>
The Plan is then passed to the <A
HREF="../../backend/executor">executor</A> for execution, and the result
is returned to the client.
<P>
There are many other modules that support this basic functionality. 
They can be accessed by clicking on the flowchart.
<P>
Another area of interest is the shared memory area, containing
table data/index blocks, locks, and backend information:
<UL> 
<LI>ShmemIndex - contains an index of all other shared memory
structures, allowing quick lookup of other structure locations in shared
memory
<LI><A HREF="../../include/storage/buf_internals.h">Buffer
Descriptors</A> - control header for shared memory buffer block

<LI><A HREF="../../include/storage/buf_internals.h">Buffer Blocks</A>
- block of table/index data shared by all backends
<LI>Shared Buf Lookup Table - lookup to see if a requested buffer
is already in the shared memory area
<LI><A HREF="../../include/storage/lock.h">LockTable</A>
- lock table structure, specifiying table, lock types, and
backends holding or waiting on lock
<LI>LockTable (lock hash) - lookup of LockTable structures using
table name
<LI>LockTable (xid hash) - lookup of LockTable structures using
transaction id
<LI><A HREF="../../include/storage/proc.h">Proc Header</A> - information
about each backend, including locks held/waiting,  indexed by process id
</UL>
Each data structure is created by calling <A
HREF="../../backend/storage/ipc/shmem.c"> ShmemInitStruct(),</A> and
the lookup hashes are created by
<A HREF="../../backend/storage/ipc/shmem.c">ShmemInitHash().</A>
<HR>
<CENTER>
<EM><BIG>
Click on an item to see more detail or click
<A HREF="backend_dirs.html">here</a> to see the full index.
</EM></BIG>
<BR>
<BR>
<IMG src="flow.jpg" usemap="#flowmap">
</CENTER>
<MAP name="flowmap">
<AREA COORDS="290,10,450,50" HREF="backend_dirs.html#main">
<AREA COORDS="550,10,710,50" HREF="backend_dirs.html#bootstrap">
<AREA COORDS="290,90,450,130," HREF="backend_dirs.html#postmaster">
<AREA COORDS="550,90,710,130," HREF="backend_dirs.html#libpq">
<AREA COORDS="290,170,450,210" HREF="backend_dirs.html#tcop">
<AREA COORDS="550,170,710,210" HREF="backend_dirs.html#tcop">
<AREA COORDS="290,270,450,310" HREF="backend_dirs.html#parser">
<AREA COORDS="290,350,450,390" HREF="backend_dirs.html#tcop">
<AREA COORDS="290,430,450,470" HREF="backend_dirs.html#optimizer">
{'fraction_non_alphanumeric': 0.1364075792730085, 'fraction_numerical': 0.024168600154679042, 'mean_word_length': 4.901312036508842, 'pattern_counts': {'":': 0, '<': 209, '<?xml version=': 0, '>': 209, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2462128', 'n_tokens_mistral': 3443, 'n_tokens_neox': 3184, 'n_words': 1195}