<commit_before>[alpha_api]: javascript:alphaApi()
[alpha_implementation]: javascript:alphaImplementation()

Session / Unit of Work {@name=unitofwork}
============

### Overview {@name=overview}    

The concept behind Unit of Work is to track modifications to a field of objects, and then be able to flush those changes to the database in a single operation.  Theres a lot of advantages to this, including that your application doesn't need to worry about individual save operations on objects, nor about the required order for those operations, nor about excessive repeated calls to save operations that would be more efficiently aggregated into one step.  It also simplifies database transactions, providing a neat package with which to insert into the traditional database begin/commit phase.
    
SQLAlchemy's unit of work includes these functions:
    
* The ability to monitor scalar and list attributes on object instances, as well as object creates.  This is handled via the attributes package.
* The ability to maintain and process a list of modified objects, and based on the relationships set up by the mappers for those objects as well as the foreign key relationships of the underlying tables, figure out the proper order of operations so that referential integrity is maintained, and also so that on-the-fly values such as newly created primary keys can be propigated to dependent objects that need them before they are saved.  The central algorithm for this is the *topological sort*.
* The ability to define custom functionality that occurs within the unit-of-work flush phase, such as "before insert", "after insert", etc.  This is accomplished via MapperExtension.
* an Identity Map, which is a dictionary storing the one and only instance of an object for a particular table/primary key combination.  This allows many parts of an application to get a handle to a particular object without any chance of modifications going to two different places.
* The sole interface to the unit of work is provided via the `Session` object.  Transactional capability, which rides on top of the transactions provided by `Engine` objects, is provided by the `SessionTransaction` object.
* Thread-locally scoped Session behavior is available as an option, which allows new objects to be automatically added to the Session corresponding to by the *default Session context*.  Without a default Session context, an application must explicitly create a Session manually as well as add new objects to it.  The default Session context, disabled by default, can also be plugged in with other user-defined schemes, which may also take into account the specific class being dealt with for a particular operation.
* The Session object in SQLAlchemy 0.2 borrows conceptually from that of [Hibernate](http://www.hibernate.org), a leading ORM for Java that was a great influence on the creation of the [JSR-220](http://jcp.org/aboutJava/communityprocess/pfd/jsr220/index.html) specification.  SQLAlchemy, under no obligation to conform to EJB specifications, is in general very different from Hibernate, providing a different paradigm for producing queries, a SQL API that is useable independently of the ORM, and of course Pythonic configuration as opposed to XML; however, JSR-220/Hibernate makes some pretty good suggestions with regards to the mechanisms of persistence.

### Object States {@name=states}

When dealing with mapped instances with regards to Sessions, an instance may be *attached* or *unattached* to a particular Session.  An instance also may or may not correspond to an actual row in the database.  The product of these two binary conditions yields us four general states a particular instance can have within the perspective of the Session:

* *Transient* - a transient instance exists within memory only and is not associated with any Session.  It also has no database identity and does not have a corresponding record in the database.  When a new instance of a class is constructed, and no default session context exists with which to automatically attach the new instance, it is a transient instance.  The instance can then be saved to a particular session in which case it becomes a *pending* instance.  If a default session context exists, new instances are added to that Session by default and therefore become *pending* instances immediately.

* *Pending* - a pending instance is a Session-attached object that has not yet been assigned a database identity.  When the Session is flushed (i.e. changes are persisted to the database), a pending instance becomes persistent.

* *Persistent* - a persistent instance has a database identity and a corresponding record in the database, and is also associated with a particular Session.   By "database identity" we mean the object is associated with a table or relational concept in the database combined with a particular primary key in that table.  Objects that are loaded by SQLAlchemy in the context of a particular session are automatically considered persistent, as are formerly pending instances which have been subject to a session `flush()`.

* *Detached* - a detached instance is an instance which has a database identity and corresponding row in the database, but is not attached to any Session.  This occurs when an instance has been removed from a Session, either because the session itself was cleared or closed, or the instance was explicitly removed from the Session.  The object can be re-attached with a session again in which case it becomes Persistent again.  Detached instances are useful when an application needs to represent a long-running operation across multiple Sessions, needs to store an object in a serialized state and then restore it later (such as within an HTTP "session" object), or in some cases where code needs to load instances locally which will later be associated with some other Session.

### Acquiring a Session {@name=getting}

A new Session object is constructed via the `create_session()` function:

    {python}
    session = create_session()

A common option used with `create_session()` is to specify a specific `Engine` or `Connection` to be used for all operations performed by this Session:

    {python}
    # create an engine
    e = create_engine('postgres://some/url')
    
    # create a Session that will use this engine for all operations.
    # it will open and close Connections as needed.
    session = create_session(bind_to=e)
    
    # open a Connection
    conn = e.connect()
    
    # create a Session that will use this specific Connection for all operations
    session = create_session(bind_to=conn)
    

The session to which an object is attached can be acquired via the `object_session()` function, which returns the appropriate `Session` if the object is pending or persistent, or `None` if the object is transient or detached:

    {python}
    session = object_session(obj)

Session Facts:

 * the Session object is **not threadsafe**.  For thread-local management of Sessions, the recommended approch is to use the [plugins_sessioncontext](rel:plugins_sessioncontext) extension module.

We will now cover some of the key concepts used by Sessions and its underlying Unit of Work.

### Introduction to the Identity Map {@name=identitymap}    

A primary concept of the Session's underlying Unit of Work is that it is keeps track of all persistent instances; recall that a persistent instance has a database identity and is attached to a Session.  In particular, the Unit of Work must insure that only *one* copy of a particular persistent instance exists within the Session at any given time.   The UOW accomplishes this task using a dictionary known as an *Identity Map*.

When a `Query` is used to issue `select` or `get` requests to the database, it will in nearly all cases result in an actual SQL execution to the database, and a corresponding traversal of rows received from that execution.  However, when the underlying mapper actually *creates* objects corresponding to the result set rows it receives, it will check the session's identity map first before instantating a new object, and return the same instance already present in the identity map if it already exists, essentially *ignoring* the object state represented by that row.  There are several ways to override this behavior and truly refresh an already-loaded instance which are described later, but the main idea is that once your instance is loaded into a particular Session, it will *never change* its state without your explicit approval, regardless of what the database says about it.

For example; below, two separate calls to load an instance with database identity "15" are issued, and the results assigned to two separate variables.   However, since the same `Session` was used, the two instances are the same instance:

    {python}
    mymapper = mapper(MyClass, mytable)
    
    session = create_session()
    obj1 = session.query(MyClass).selectfirst(mytable.c.id==15)
    obj2 = session.query(MyClass).selectfirst(mytable.c.id==15)
    
    >>> obj1 is obj2
    True
    
The Identity Map is an instance of `weakref.WeakValueDictionary`, so that when an in-memory object falls out of scope, it will be removed automatically.  However, this may not be instant if there are circular references upon the object.  To guarantee that an instance is removed from the identity map before removing references to it, use the `expunge()` method, described later, to remove it.

The Session supports an iterator interface in order to see all objects in the identity map:

    {python}
    for obj in session:
        print obj

As well as `__contains__()`:

    {python}
    if obj in session:
        print "Object is present"
        
The identity map itself is accessible via the `identity_map` accessor:

    {python}
    >>> session.identity_map.values()
    [<__main__.User object at 0x712630>, <__main__.Address object at 0x712a70>]

The identity of each object instance is available via the `_instance_key` property attached to each object instance, and is a tuple consisting of the object's class and an additional tuple of primary key values, in the order that they appear within the table definition:

    {python}
    >>> obj._instance_key 
    (<class 'test.tables.User'>, (7,))
    
At the moment that an object is assigned this key within a `flush()` operation, it is also added to the session's identity map.

The `get()` method on `Query`, which retrieves an object based on primary key identity, also checks in the Session's identity map first to save a database round-trip if possible.  In the case of an object lazy-loading a single child object, the `get()` method is used as well, so scalar-based lazy loads may in some cases not query the database; this is particularly important for backreference relationships as it can save a lot of queries.

### Whats Changed ? {@name=changed}    

The next concept is that in addition to the `Session` storing a record of all objects loaded or saved, it also stores lists of all *newly created* (i.e. pending) objects and lists of all persistent objects that have been marked as *deleted*.  These lists are used when a `flush()` call is issued to save all changes.  During a flush operation, it also scans its list of persistent instances for changes which are marked as dirty.

These records are all tracked by a collection of `Set` objects (which are a SQLAlchemy-specific instance called a `HashSet`) that are also viewable off the `Session`:

    {python}
    # pending objects recently added to the Session
    session.new
    
    # persistent objects which currently have changes detected
    # (this Set is now created on the fly each time the property is called)
    session.dirty

    # persistent objects that have been marked as deleted via session.delete(obj)
    session.deleted
    
Unlike the identity map, the `new`, `dirty`, and `deleted` lists are *not weak referencing.*  This means if you abandon all references to new or modified objects within a session, *they are still present* and will be saved on the next flush operation, unless they are removed from the Session explicitly (more on that later).  The `new` list may change in a future release to be weak-referencing, however for the `deleted` list, one can see that its quite natural for a an object marked as deleted to have no references in the application, yet a DELETE operation is still required.

### The Session API {@name=api}

#### query() {@name=query}

The `query()` function takes a class or `Mapper` as an argument, along with an optional `entity_name` parameter, and returns a new `Query` object which will issue mapper queries within the context of this Session.  If a Mapper is passed, then the Query uses that mapper.  Otherwise, if a class is sent, it will locate the primary mapper for that class which is used to construct the Query.  

    {python}
    # query from a class
    session.query(User).select_by(name='ed')
    
    # query from a mapper
    query = session.query(usermapper)
    x = query.get(1)
    
    # query from a class mapped with entity name 'alt_users'
    q = session.query(User, entity_name='alt_users')<commit_msg>doc adjust for "dirty" list behavior<commit_after>[alpha_api]: javascript:alphaApi()
[alpha_implementation]: javascript:alphaImplementation()

Session / Unit of Work {@name=unitofwork}
============

### Overview {@name=overview}    

The concept behind Unit of Work is to track modifications to a field of objects, and then be able to flush those changes to the database in a single operation.  Theres a lot of advantages to this, including that your application doesn't need to worry about individual save operations on objects, nor about the required order for those operations, nor about excessive repeated calls to save operations that would be more efficiently aggregated into one step.  It also simplifies database transactions, providing a neat package with which to insert into the traditional database begin/commit phase.
    
SQLAlchemy's unit of work includes these functions:
    
* The ability to monitor scalar and list attributes on object instances, as well as object creates.  This is handled via the attributes package.
* The ability to maintain and process a list of modified objects, and based on the relationships set up by the mappers for those objects as well as the foreign key relationships of the underlying tables, figure out the proper order of operations so that referential integrity is maintained, and also so that on-the-fly values such as newly created primary keys can be propigated to dependent objects that need them before they are saved.  The central algorithm for this is the *topological sort*.
* The ability to define custom functionality that occurs within the unit-of-work flush phase, such as "before insert", "after insert", etc.  This is accomplished via MapperExtension.
* an Identity Map, which is a dictionary storing the one and only instance of an object for a particular table/primary key combination.  This allows many parts of an application to get a handle to a particular object without any chance of modifications going to two different places.
* The sole interface to the unit of work is provided via the `Session` object.  Transactional capability, which rides on top of the transactions provided by `Engine` objects, is provided by the `SessionTransaction` object.
* Thread-locally scoped Session behavior is available as an option, which allows new objects to be automatically added to the Session corresponding to by the *default Session context*.  Without a default Session context, an application must explicitly create a Session manually as well as add new objects to it.  The default Session context, disabled by default, can also be plugged in with other user-defined schemes, which may also take into account the specific class being dealt with for a particular operation.
* The Session object in SQLAlchemy 0.2 borrows conceptually from that of [Hibernate](http://www.hibernate.org), a leading ORM for Java that was a great influence on the creation of the [JSR-220](http://jcp.org/aboutJava/communityprocess/pfd/jsr220/index.html) specification.  SQLAlchemy, under no obligation to conform to EJB specifications, is in general very different from Hibernate, providing a different paradigm for producing queries, a SQL API that is useable independently of the ORM, and of course Pythonic configuration as opposed to XML; however, JSR-220/Hibernate makes some pretty good suggestions with regards to the mechanisms of persistence.

### Object States {@name=states}

When dealing with mapped instances with regards to Sessions, an instance may be *attached* or *unattached* to a particular Session.  An instance also may or may not correspond to an actual row in the database.  The product of these two binary conditions yields us four general states a particular instance can have within the perspective of the Session:

* *Transient* - a transient instance exists within memory only and is not associated with any Session.  It also has no database identity and does not have a corresponding record in the database.  When a new instance of a class is constructed, and no default session context exists with which to automatically attach the new instance, it is a transient instance.  The instance can then be saved to a particular session in which case it becomes a *pending* instance.  If a default session context exists, new instances are added to that Session by default and therefore become *pending* instances immediately.

* *Pending* - a pending instance is a Session-attached object that has not yet been assigned a database identity.  When the Session is flushed (i.e. changes are persisted to the database), a pending instance becomes persistent.

* *Persistent* - a persistent instance has a database identity and a corresponding record in the database, and is also associated with a particular Session.   By "database identity" we mean the object is associated with a table or relational concept in the database combined with a particular primary key in that table.  Objects that are loaded by SQLAlchemy in the context of a particular session are automatically considered persistent, as are formerly pending instances which have been subject to a session `flush()`.

* *Detached* - a detached instance is an instance which has a database identity and corresponding row in the database, but is not attached to any Session.  This occurs when an instance has been removed from a Session, either because the session itself was cleared or closed, or the instance was explicitly removed from the Session.  The object can be re-attached with a session again in which case it becomes Persistent again.  Detached instances are useful when an application needs to represent a long-running operation across multiple Sessions, needs to store an object in a serialized state and then restore it later (such as within an HTTP "session" object), or in some cases where code needs to load instances locally which will later be associated with some other Session.

### Acquiring a Session {@name=getting}

A new Session object is constructed via the `create_session()` function:

    {python}
    session = create_session()

A common option used with `create_session()` is to specify a specific `Engine` or `Connection` to be used for all operations performed by this Session:

    {python}
    # create an engine
    e = create_engine('postgres://some/url')
    
    # create a Session that will use this engine for all operations.
    # it will open and close Connections as needed.
    session = create_session(bind_to=e)
    
    # open a Connection
    conn = e.connect()
    
    # create a Session that will use this specific Connection for all operations
    session = create_session(bind_to=conn)
    

The session to which an object is attached can be acquired via the `object_session()` function, which returns the appropriate `Session` if the object is pending or persistent, or `None` if the object is transient or detached:

    {python}
    session = object_session(obj)

Session Facts:

 * the Session object is **not threadsafe**.  For thread-local management of Sessions, the recommended approch is to use the [plugins_sessioncontext](rel:plugins_sessioncontext) extension module.

We will now cover some of the key concepts used by Sessions and its underlying Unit of Work.

### Introduction to the Identity Map {@name=identitymap}    

A primary concept of the Session's underlying Unit of Work is that it is keeps track of all persistent instances; recall that a persistent instance has a database identity and is attached to a Session.  In particular, the Unit of Work must insure that only *one* copy of a particular persistent instance exists within the Session at any given time.   The UOW accomplishes this task using a dictionary known as an *Identity Map*.

When a `Query` is used to issue `select` or `get` requests to the database, it will in nearly all cases result in an actual SQL execution to the database, and a corresponding traversal of rows received from that execution.  However, when the underlying mapper actually *creates* objects corresponding to the result set rows it receives, it will check the session's identity map first before instantating a new object, and return the same instance already present in the identity map if it already exists, essentially *ignoring* the object state represented by that row.  There are several ways to override this behavior and truly refresh an already-loaded instance which are described later, but the main idea is that once your instance is loaded into a particular Session, it will *never change* its state without your explicit approval, regardless of what the database says about it.

For example; below, two separate calls to load an instance with database identity "15" are issued, and the results assigned to two separate variables.   However, since the same `Session` was used, the two instances are the same instance:

    {python}
    mymapper = mapper(MyClass, mytable)
    
    session = create_session()
    obj1 = session.query(MyClass).selectfirst(mytable.c.id==15)
    obj2 = session.query(MyClass).selectfirst(mytable.c.id==15)
    
    >>> obj1 is obj2
    True
    
The Identity Map is an instance of `weakref.WeakValueDictionary`, so that when an in-memory object falls out of scope, it will be removed automatically.  However, this may not be instant if there are circular references upon the object.  To guarantee that an instance is removed from the identity map before removing references to it, use the `expunge()` method, described later, to remove it.

The Session supports an iterator interface in order to see all objects in the identity map:

    {python}
    for obj in session:
        print obj

As well as `__contains__()`:

    {python}
    if obj in session:
        print "Object is present"
        
The identity map itself is accessible via the `identity_map` accessor:

    {python}
    >>> session.identity_map.values()
    [<__main__.User object at 0x712630>, <__main__.Address object at 0x712a70>]

The identity of each object instance is available via the `_instance_key` property attached to each object instance, and is a tuple consisting of the object's class and an additional tuple of primary key values, in the order that they appear within the table definition:

    {python}
    >>> obj._instance_key 
    (<class 'test.tables.User'>, (7,))
    
At the moment that an object is assigned this key within a `flush()` operation, it is also added to the session's identity map.

The `get()` method on `Query`, which retrieves an object based on primary key identity, also checks in the Session's identity map first to save a database round-trip if possible.  In the case of an object lazy-loading a single child object, the `get()` method is used as well, so scalar-based lazy loads may in some cases not query the database; this is particularly important for backreference relationships as it can save a lot of queries.

### Whats Changed ? {@name=changed}    

The next concept is that in addition to the `Session` storing a record of all objects loaded or saved, it also stores lists of all *newly created* (i.e. pending) objects and lists of all persistent objects that have been marked as *deleted*.  These lists are used when a `flush()` call is issued to save all changes.  During a flush operation, it also scans its list of persistent instances for changes which are marked as dirty.

These records are all tracked by collection functions that are also viewable off the `Session` as properties:

    {python}
    # pending objects recently added to the Session
    session.new
    
    # persistent objects which currently have changes detected
    # (this Set is now created on the fly each time the property is called)
    session.dirty

    # persistent objects that have been marked as deleted via session.delete(obj)
    session.deleted

As of the 0.3 series, the `new` and `deleted` lists are actual `Set` objects, and are therefore *not weak referencing*.  Elements that are in the `dirty` collection are, however, and have to be maintained in the current scope else they will be garbage collected(this is a slightly suboptimal behavior introduced in the 0.3 series which may be improved in a future release).  

As for objects inside of `new` and `deleted`, if you abandon all references to new or modified objects within a session, *they are still present* in either of those two lists, and will be saved on the next flush operation, unless they are removed from the Session explicitly (more on that later).  The `new` list may change in a future release to be weak-referencing, however for the `deleted` list, one can see that its quite natural for a an object marked as deleted to have no references in the application, yet a DELETE operation is still required.

### The Session API {@name=api}

#### query() {@name=query}

The `query()` function takes a class or `Mapper` as an argument, along with an optional `entity_name` parameter, and returns a new `Query` object which will issue mapper queries within the context of this Session.  If a Mapper is passed, then the Query uses that mapper.  Otherwise, if a class is sent, it will locate the primary mapper for that class which is used to construct the Query.  

    {python}
    # query from a class
    session.query(User).select_by(name='ed')
    
    # query from a mapper
    query = session.query(usermapper)
    x = query.get(1)
    
    # query from a class mapped with entity name 'alt_users'
    q = session.query(User, entity_name='alt_users')
{'fraction_non_alphanumeric': 0.0487284966342558, 'fraction_numerical': 0.002842183994016455, 'mean_word_length': 4.276440410418311, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4100552', 'n_tokens_mistral': 6245, 'n_tokens_neox': 6048, 'n_words': 3929}