<commit_before><?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="persistent-112"><title>Persistent</title><conbody><p id="112-i1">Forms deal with the boundary between the user and the application. Another boundary we
            need to deal with is between the application and the storage layer. Whether it be a SQL
            database, a YAML file, or a binary blob, odds are you have to work to get your storage
            layer to accept your application datatypes. Persistent is Yesod's answer to data
            storage- a type-safe, universal data store interface for Haskell. </p><p id="112-i1.5">Haskell has many different database bindings available. However, most of these have
            little knowledge of a schema and therefore do not provide useful static guarantees. They
            also force database-dependent APIs and data types on the programmer. Haskellers have
            attempted a more revolutionary route of creating Haskell specific data stores to get
            around these flaws that allow one to easily store any Haskell type. These options are
            great for certain use cases, but they constrain one to the storage techniques provided
            by the library, do not interface well with other languages, and the flexibility can also
            mean one must write reams of code for querying data. In contrast, Persistent allows us
            to choose among existing databases that are highly tuned for different data storage use
            cases, interoperate with other programming languages, and to use a safe and productive
            query interface. </p><p id="112-i2">Persistent follows the guiding principles of type safety and concise, declarative syntax. Some other nice features are:</p><ul id="112-x1"><li id="112-x2">Database-agnostic. There is first class support for PostgreSQL, SQLite and MongoDB, with
                experimental CouchDB and MySQL support in the works.</li><li id="112-x3">By being non-relational in nature, we simultaneously are able to support a wider number of storage layers and are not constrained by some of the performance bottlenecks incurred through joins.</li><li id="112-x4">A major source of frustration in dealing with SQL databases is changes to the schema. Persistent can automatically perform database migrations.</li></ul>
    <!-- still want to cover:
        
        Advanced: select enumerator interface (maybe)
        Join modules
    -->

</conbody><concept id="synopsis-113"><title>Synopsis</title><conbody>  <codeblock id="113-x1" outputclass="haskell">{-# LANGUAGE QuasiQuotes, TemplateHaskell, TypeFamilies, OverloadedStrings #-}
{-# LANGUAGE GADTs, FlexibleContexts #-}
import Database.Persist
import Database.Persist.Sqlite
import Database.Persist.TH
import Control.Monad.IO.Class (liftIO)

share [mkPersist sqlSettings, mkMigrate "migrateAll"] [persist|
Person
    name String
    age Int Maybe
    deriving Show
BlogPost
    title String
    authorId PersonId
    deriving Show
|]

main :: IO ()
main = withSqliteConn ":memory:" $ runSqlConn $ do
    runMigration migrateAll

    johnId &lt;- insert $ Person "<NAME>" $ Just 35
    janeId &lt;- insert $ Person "<NAME>" Nothing

    insert $ BlogPost "My fr1st p0st" johnId
    insert $ BlogPost "One more for good measure" johnId

    oneJohnPost &lt;- selectList [BlogPostAuthorId ==. johnId] [LimitTo 1]
    liftIO $ print (oneJohnPost :: [Entity BlogPost])

    john &lt;- get johnId
    liftIO $ print (john :: Maybe Person)

    delete janeId
    deleteWhere [BlogPostAuthorId ==. johnId]</codeblock>
 </conbody></concept><concept id="solving-the-boundary-issue-114"><title>Solving the boundary issue</title><conbody><p id="114-b1">Suppose you are storing information on people in a SQL database. Your table might look
            something like:</p><codeblock outputclass="sql" id="114-x1">CREATE TABLE Person(id SERIAL PRIMARY KEY, name VARCHAR NOT NULL, age INTEGER)</codeblock><p id="114-b2">And if you are using a database like PostgreSQL, you can be guaranteed that the database will never store some arbitrary text in your age field. (The same cannot be said of SQLite, but let's forget about that for now.) To mirror this database table, you would likely create a Haskell datatype that looks something like:</p><codeblock id="114-x2" outputclass="haskell">data Person = Person
    { personName :: Text
    , personAge :: Int
    }</codeblock><p id="114-b3">It looks like everything is type safe: the database schema matches our Haskell datatypes, the database ensures that invalid data can never make it into our data store, and everything is generally awesome. Well, until:</p><ul id="114-x3"><li id="114-x4">You want to pull data from the database, and the database layer gives you the data in an untyped format.</li><li id="114-x5">You want to find everyone older than 32, and you accidently write "thirtytwo" in your SQL statement. Guess what: that will compile just fine, and you won't find out you have a problem until runtime.</li><li id="114-x6">You decide you want to find the first 10 people alphabetically. No problem... until you
                make a typo in your SQL. Once again, you don't find out until runtime.</li></ul><p id="114-b4">In dynamic languages, the answers to these issues is unit testing. For everything that <i id="114-x7">can</i> go wrong, make sure you write a test case. But as I am sure you are aware by now, that doesn't jive well with the Yesod approach to things. We like to take advantage of Haskell's strong typing to save us wherever possible, and data storage is no exception.</p><p id="114-b5">So the question remains: how can we use Haskell's type system to save the day?</p></conbody><concept id="types-115"><title>Types</title><conbody><p id="115-ba1">Like routing, there is nothing intrinsically difficult about type-safe data access. It
            just requires a lot of monotonous, error prone, boiler plate code. As usual, this means
            we can use the type system to keep us honest. And to avoid some of the drudgery, we'll
            use a sprinkling of Template Haskell.</p>
<note id="115-x1">Earlier versions of Persistent made much heavier usage of Template Haskell. Starting
            with 0.6, there is a new architecture inspired by the <apiname id="115-x2">groundhog</apiname>
            package. This approach uses phantom types to carry a lot of the burden.</note>
<p id="115-ba2"><codeph id="115-x3">PersistValue</codeph> is the basic building block of Persistent. It is a
            sum type that can represent data that gets sent to and from a database. Its definition
            is:</p><codeblock id="115-x4" outputclass="haskell">data PersistValue = PersistText Text
                  | PersistByteString ByteString
                  | PersistInt64 Int64
                  | PersistDouble Double
                  | PersistBool Bool
                  | PersistDay Day
                  | PersistTimeOfDay TimeOfDay
                  | PersistUTCTime UTCTime
                  | PersistNull
                  | PersistList [PersistValue]
                  | PersistMap [(T.Text, PersistValue)]
                  | PersistForeignKey ByteString -- ^ intended especially for MongoDB backend</codeblock><p id="115-ba4">Each Persistent backend needs to know how to translate the relevant values into
            something the database can understand. However, it would be awkward do have to express
            all of our data simply in terms of these basic types. The next layer is the <codeph id="115-x5">PersistField</codeph> typeclass, which defines how an arbitrary Haskell
            datatype can be marshaled to and from a <codeph id="115-x6">PersistValue</codeph>. A
                <codeph id="115-x-3">PersistField</codeph> correlates to a column in a SQL database. In our
            person example above, name and age would be our <codeph id="115-x7">PersistField</codeph>s.</p><p id="115-ba5">To tie up the user side of the code, our last typeclass is <codeph id="115-x8">PersistEntity</codeph>. An instance of PersistEntity correlates with a table in a
            SQL database. This typeclass defines a number of functions and some associated types. To
            review, we have the following correspondence between Persistent and SQL:</p>
        <simpletable id="115-simpletable_hmz_kck_kd">
            <sthead id="115-x9">
                <stentry id="115-x10">SQL</stentry>
                <stentry id="115-x11">Persistent</stentry>
            </sthead>
            <strow id="115-x12">
                <stentry id="115-x13">Datatypes (VARCHAR, INTEGER, etc)</stentry>
                <stentry id="115-x14">PersistValue</stentry>
            </strow>
            <strow id="115-x18">
                <stentry id="115-x19">Column</stentry>
                <stentry id="115-x20">PersistField</stentry>
            </strow>
            <strow id="115-x15">
                <stentry id="115-x16">Table</stentry>
                <stentry id="115-x17">PersistEntity</stentry>
            </strow>
        </simpletable></conbody></concept><concept id="code-generation-116"><title>Code Generation</title><conbody><p id="116-bb1">In order to ensure that the PersistEntity instances match up properly with your Haskell datatypes, Persistent takes responsibility for both. This is also good from a DRY (Don't Repeat Yourslef) perspective: you only need to define your entities once. Let's see a quick example:</p><codeblock id="116-x1" outputclass="haskell">-- START
{-# LANGUAGE QuasiQuotes, TypeFamilies, GeneralizedNewtypeDeriving, TemplateHaskell, OverloadedStrings, GADTs #-}
import Database.Persist
import Database.Persist.TH
import Database.Persist.Sqlite
import Control.Monad.IO.Class (liftIO)

mkPersist sqlSettings [persist|
Person
    name String
    age Int
    deriving Show
|]
-- STOP
main = return ()
</codeblock><p id="116-bb2">We use a combination of Template Haskell and Quasi-Quotation (like when defining
            routes): <apiname id="116-x6">persistent-template:Database.Persist.TH:persist</apiname> is a
            quasi-quoter which converts a whitespace-sensitive syntax into a list of entity
            definitions. (You can also declare your entities in a separate file using
                <apiname id="116-x9">persistent-template:Database.Persist.TH:persistFile</apiname>.)
                <apiname id="116-x10">persistent-template:Database.Persist.TH:mkPersist</apiname> takes that list
            of entities and declares:</p><ul id="116-x2"><li id="116-x3">One Haskell datatype for each entity.</li><li id="116-x4">A <codeph id="116-x-3">PersistEntity</codeph> instance for each datatype defined.</li></ul>
        <p>The example above generates code that looks like the following:</p>
        <codeblock outputclass="haskell">{-# OPTIONS_GHC -Wwarn #-}
-- START
{-# LANGUAGE TypeFamilies, GeneralizedNewtypeDeriving, OverloadedStrings, GADTs #-}
import Database.Persist
import Database.Persist.Store<commit_msg>persistFile => persistFileWith (#5)<commit_after><?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
<concept id="persistent-112"><title>Persistent</title><conbody><p id="112-i1">Forms deal with the boundary between the user and the application. Another boundary we
            need to deal with is between the application and the storage layer. Whether it be a SQL
            database, a YAML file, or a binary blob, odds are you have to work to get your storage
            layer to accept your application datatypes. Persistent is Yesod's answer to data
            storage- a type-safe, universal data store interface for Haskell. </p><p id="112-i1.5">Haskell has many different database bindings available. However, most of these have
            little knowledge of a schema and therefore do not provide useful static guarantees. They
            also force database-dependent APIs and data types on the programmer. Haskellers have
            attempted a more revolutionary route of creating Haskell specific data stores to get
            around these flaws that allow one to easily store any Haskell type. These options are
            great for certain use cases, but they constrain one to the storage techniques provided
            by the library, do not interface well with other languages, and the flexibility can also
            mean one must write reams of code for querying data. In contrast, Persistent allows us
            to choose among existing databases that are highly tuned for different data storage use
            cases, interoperate with other programming languages, and to use a safe and productive
            query interface. </p><p id="112-i2">Persistent follows the guiding principles of type safety and concise, declarative syntax. Some other nice features are:</p><ul id="112-x1"><li id="112-x2">Database-agnostic. There is first class support for PostgreSQL, SQLite and MongoDB, with
                experimental CouchDB and MySQL support in the works.</li><li id="112-x3">By being non-relational in nature, we simultaneously are able to support a wider number of storage layers and are not constrained by some of the performance bottlenecks incurred through joins.</li><li id="112-x4">A major source of frustration in dealing with SQL databases is changes to the schema. Persistent can automatically perform database migrations.</li></ul>
    <!-- still want to cover:
        
        Advanced: select enumerator interface (maybe)
        Join modules
    -->

</conbody><concept id="synopsis-113"><title>Synopsis</title><conbody>  <codeblock id="113-x1" outputclass="haskell">{-# LANGUAGE QuasiQuotes, TemplateHaskell, TypeFamilies, OverloadedStrings #-}
{-# LANGUAGE GADTs, FlexibleContexts #-}
import Database.Persist
import Database.Persist.Sqlite
import Database.Persist.TH
import Control.Monad.IO.Class (liftIO)

share [mkPersist sqlSettings, mkMigrate "migrateAll"] [persist|
Person
    name String
    age Int Maybe
    deriving Show
BlogPost
    title String
    authorId PersonId
    deriving Show
|]

main :: IO ()
main = withSqliteConn ":memory:" $ runSqlConn $ do
    runMigration migrateAll

    johnId &lt;- insert $ Person "<NAME>" $ Just 35
    janeId &lt;- insert $ Person "<NAME>" Nothing

    insert $ BlogPost "My fr1st p0st" johnId
    insert $ BlogPost "One more for good measure" johnId

    oneJohnPost &lt;- selectList [BlogPostAuthorId ==. johnId] [LimitTo 1]
    liftIO $ print (oneJohnPost :: [Entity BlogPost])

    john &lt;- get johnId
    liftIO $ print (john :: Maybe Person)

    delete janeId
    deleteWhere [BlogPostAuthorId ==. johnId]</codeblock>
 </conbody></concept><concept id="solving-the-boundary-issue-114"><title>Solving the boundary issue</title><conbody><p id="114-b1">Suppose you are storing information on people in a SQL database. Your table might look
            something like:</p><codeblock outputclass="sql" id="114-x1">CREATE TABLE Person(id SERIAL PRIMARY KEY, name VARCHAR NOT NULL, age INTEGER)</codeblock><p id="114-b2">And if you are using a database like PostgreSQL, you can be guaranteed that the database will never store some arbitrary text in your age field. (The same cannot be said of SQLite, but let's forget about that for now.) To mirror this database table, you would likely create a Haskell datatype that looks something like:</p><codeblock id="114-x2" outputclass="haskell">data Person = Person
    { personName :: Text
    , personAge :: Int
    }</codeblock><p id="114-b3">It looks like everything is type safe: the database schema matches our Haskell datatypes, the database ensures that invalid data can never make it into our data store, and everything is generally awesome. Well, until:</p><ul id="114-x3"><li id="114-x4">You want to pull data from the database, and the database layer gives you the data in an untyped format.</li><li id="114-x5">You want to find everyone older than 32, and you accidently write "thirtytwo" in your SQL statement. Guess what: that will compile just fine, and you won't find out you have a problem until runtime.</li><li id="114-x6">You decide you want to find the first 10 people alphabetically. No problem... until you
                make a typo in your SQL. Once again, you don't find out until runtime.</li></ul><p id="114-b4">In dynamic languages, the answers to these issues is unit testing. For everything that <i id="114-x7">can</i> go wrong, make sure you write a test case. But as I am sure you are aware by now, that doesn't jive well with the Yesod approach to things. We like to take advantage of Haskell's strong typing to save us wherever possible, and data storage is no exception.</p><p id="114-b5">So the question remains: how can we use Haskell's type system to save the day?</p></conbody><concept id="types-115"><title>Types</title><conbody><p id="115-ba1">Like routing, there is nothing intrinsically difficult about type-safe data access. It
            just requires a lot of monotonous, error prone, boiler plate code. As usual, this means
            we can use the type system to keep us honest. And to avoid some of the drudgery, we'll
            use a sprinkling of Template Haskell.</p>
<note id="115-x1">Earlier versions of Persistent made much heavier usage of Template Haskell. Starting
            with 0.6, there is a new architecture inspired by the <apiname id="115-x2">groundhog</apiname>
            package. This approach uses phantom types to carry a lot of the burden.</note>
<p id="115-ba2"><codeph id="115-x3">PersistValue</codeph> is the basic building block of Persistent. It is a
            sum type that can represent data that gets sent to and from a database. Its definition
            is:</p><codeblock id="115-x4" outputclass="haskell">data PersistValue = PersistText Text
                  | PersistByteString ByteString
                  | PersistInt64 Int64
                  | PersistDouble Double
                  | PersistBool Bool
                  | PersistDay Day
                  | PersistTimeOfDay TimeOfDay
                  | PersistUTCTime UTCTime
                  | PersistNull
                  | PersistList [PersistValue]
                  | PersistMap [(T.Text, PersistValue)]
                  | PersistForeignKey ByteString -- ^ intended especially for MongoDB backend</codeblock><p id="115-ba4">Each Persistent backend needs to know how to translate the relevant values into
            something the database can understand. However, it would be awkward do have to express
            all of our data simply in terms of these basic types. The next layer is the <codeph id="115-x5">PersistField</codeph> typeclass, which defines how an arbitrary Haskell
            datatype can be marshaled to and from a <codeph id="115-x6">PersistValue</codeph>. A
                <codeph id="115-x-3">PersistField</codeph> correlates to a column in a SQL database. In our
            person example above, name and age would be our <codeph id="115-x7">PersistField</codeph>s.</p><p id="115-ba5">To tie up the user side of the code, our last typeclass is <codeph id="115-x8">PersistEntity</codeph>. An instance of PersistEntity correlates with a table in a
            SQL database. This typeclass defines a number of functions and some associated types. To
            review, we have the following correspondence between Persistent and SQL:</p>
        <simpletable id="115-simpletable_hmz_kck_kd">
            <sthead id="115-x9">
                <stentry id="115-x10">SQL</stentry>
                <stentry id="115-x11">Persistent</stentry>
            </sthead>
            <strow id="115-x12">
                <stentry id="115-x13">Datatypes (VARCHAR, INTEGER, etc)</stentry>
                <stentry id="115-x14">PersistValue</stentry>
            </strow>
            <strow id="115-x18">
                <stentry id="115-x19">Column</stentry>
                <stentry id="115-x20">PersistField</stentry>
            </strow>
            <strow id="115-x15">
                <stentry id="115-x16">Table</stentry>
                <stentry id="115-x17">PersistEntity</stentry>
            </strow>
        </simpletable></conbody></concept><concept id="code-generation-116"><title>Code Generation</title><conbody><p id="116-bb1">In order to ensure that the PersistEntity instances match up properly with your Haskell datatypes, Persistent takes responsibility for both. This is also good from a DRY (Don't Repeat Yourslef) perspective: you only need to define your entities once. Let's see a quick example:</p><codeblock id="116-x1" outputclass="haskell">-- START
{-# LANGUAGE QuasiQuotes, TypeFamilies, GeneralizedNewtypeDeriving, TemplateHaskell, OverloadedStrings, GADTs #-}
import Database.Persist
import Database.Persist.TH
import Database.Persist.Sqlite
import Control.Monad.IO.Class (liftIO)

mkPersist sqlSettings [persist|
Person
    name String
    age Int
    deriving Show
|]
-- STOP
main = return ()
</codeblock><p id="116-bb2">We use a combination of Template Haskell and Quasi-Quotation (like when defining
            routes): <apiname id="116-x6">persistent-template:Database.Persist.TH:persist</apiname> is a
            quasi-quoter which converts a whitespace-sensitive syntax into a list of entity
            definitions. (You can also declare your entities in a separate file using
                <apiname id="116-x9">persistent-template:Database.Persist.TH:persistFileWith</apiname>.)
                <apiname id="116-x10">persistent-template:Database.Persist.TH:mkPersist</apiname> takes that list
            of entities and declares:</p><ul id="116-x2"><li id="116-x3">One Haskell datatype for each entity.</li><li id="116-x4">A <codeph id="116-x-3">PersistEntity</codeph> instance for each datatype defined.</li></ul>
        <p>The example above generates code that looks like the following:</p>
        <codeblock outputclass="haskell">{-# OPTIONS_GHC -Wwarn #-}
-- START
{-# LANGUAGE TypeFamilies, GeneralizedNewtypeDeriving, OverloadedStrings, GADTs #-}
import Database.Persist
import Database.Persist.Store
{'fraction_non_alphanumeric': 0.09090500179920835, 'fraction_numerical': 0.02424433249370277, 'mean_word_length': 3.2854664610639936, 'pattern_counts': {'":': 2, '<': 295, '<?xml version=': 2, '>': 296, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '790331', 'n_tokens_mistral': 6489, 'n_tokens_neox': 5897, 'n_words': 2439}