<commit_before><html>

  <head>
    <script src="https://use.typekit.net/xdj4hmw.js"></script>
    <script type="text/javascript">try { Typekit.load({ async: true }); } catch(e) { }</script>

    <link href="https://fonts.googleapis.com/css?family=Source+Code+Pro:400|Roboto:400,300,500" rel="stylesheet" type="text/css">
    <link rel="stylesheet" media="all" href="../stylesheets/main.css" type="text/css" />
    <link rel="stylesheet" media="all" href="../stylesheets/rocco.css" />
    <link rel="stylesheet" media="all" href="../stylesheets/github-markdown.css" />

    <script src="../assets/highlight.pack.js"></script>
    <script type="text/javascript">
      hljs.initHighlightingOnLoad();
    </script>
  </head>

  <body>
  <div class="backdrop tk-proxima-nova">
    <div class="header">
      <div class="header-left"><a href="/">Syberia</a></div>
      <div class="header-right">
        <ul>
          <li><a href="/docs#example">Get Started</a></li>
          <li><a href="/docs/packages">Packages</a></li>
          <li><a href="/docs">Docs</a></li>
        </ul>
      </div>
    </div>
    <br /> <br /> <br />
  </div>
  <div class="colmask leftmenu tk-proxima-nova">
    <div class="colleft">
      <div class="col1">
        <h1>Introduction</h1>

        <p>Let's get you up and running with Syberia!</p>

        <p>If you already know what Syberia is and what you can do it with it, feel free to
        skip to the next <a href="#example">subsection</a>. Otherwise, hopefully this
        brief section will answer some questions.</p>

        <p>Will Syberia help me if I am trying to <b>productionize data science models</b>?
        Yes, but primarily if those models are written in R.
        Syberia is a framework designed specifically to productionize the R
        programming language with rapid iteration cycles, although hybrid models
        consisting of Python, Scala, and other tools will also be possible to deploy.</p>

        <p>Will Syberia help me if I am trying to <b>quantitatively understand a business question</b>?
        While not its primary design, the packages that compose Syberia can be
        used to augment interactive data analysis and introduce the benefits
        of solid developer hygiene such as testability and modularity.</p>

        <p>Will Syberia help me if I am <b>solving an academic problem</b>? 
        Syberia was designed in an industry context but is general enough to be
        useful for other domains, including academia. Although no out-of-the-box 
        Syberia engine currently exists for reproducible research and publication,
        this is a possible path to explore for the future.</p>

        <p>What if I need to solve a <b>big data problem</b> or <b>deploy low latency
          machine learning models</b>? Syberia is designed to act as a command
        center for all data science problems and eventually interact smoothly
        with high-performance and large-size data requirements. Currently, you
        should expect to be able to productionize any R code (10-1000ms latency SLAs on data
        problems with inputs of millions of rows and thousands of features).
        </p>

        <p>What about <b>dashboards</b>? <b>Unstructured or audiovisual data</b>? <b>Deep 
          learning</b>? The world is at your feet but you will have to do some
        custom <a href="https://en.wikipedia.org/wiki/Foreign_function_interface" target="_new">FFI</a> work.
        For the current alpha release, productionization of medium-size structured data
        problems is supported out-of-the-box.</p>

        <h2><a name="example">An example</a></h2>

        <p>Before we get started, let's take a look at what a classifier might
           look like using the Syberia modeling engine:</p>
        <div class="code">
          <pre>
            <code class="R"><span class="spacer">
list( # R's a LISP, so we represent the modeling process with a list. :)
  import = list(
    # For this simple example, we'll use Kaggle's example Titanic data set.
    url = "https://raw.githubusercontent.com/haven-jeon/introduction_to_most_usable_pkgs_in_project/master/bicdata/data/titanic.csv"
  ), # You can use a variety of data import adapters, like csv or AWS's S3.

  data = list(
    "Factor to character"  = list(column_transformation(as.character), is.factor)
    # One line per feature engineering step seems undue, but it's for a
    # good cause: commenting out to try with and without a step is trivial,
    # and as the parametrization grows longer it's a clear indication
    # you need a new mungebit!
  , "fare_title"           = list(multi_column_transformation(function(title, fare) { ave(fare, title, FUN = mean) }), c("title", "fare"), "title_fare")
  , "Name length variable" = list(new_variable, function(name) nchar(name), "name_length")
  , "Set factors"          = list(column_transformation(factor), c("sex", "embarked"))
  , "Restore levels"       = list(restore_categorical_variables)
  , "Rename dep_var"       = list(renamer, c("survived" = "dep_var"))
  ), # Giving a short summary of each step will come in handy later.

  model = list(
      "gbm" # The name of our classifier. This is defined elsewhere.
    , .id_var             = "X"
    , distribution        = "bernoulli" # And some parameters.
    , number_of_trees     = 3000        # Here we try a binary classification.
    , shrinkage_factor    = 0.005
    , depth               = 5
    , min_observations    = 6
    , train_fraction      = 1
    , bag_fraction        = 0.5
    , cv                  = TRUE
    , cv_folds            = 5
    , number_of_cores     = 4
    , perf_method         = "cv"
    , prediction_type     = "response"
  ),

  export = list( # Exporting the model is done in 1 line of code.
    R    = "titanic_model"  # Export the model object to a global variable.
# , s3   = "path/titanic"   # It's also possible to export to Amazon's S3. 
# , file = "~/path/titanic" # Or a file...
                            # Or any other custom export format you define.
  )
)
            </span></code>
          </pre>
        </div>

        <p>This simple example showcases the primary principle behind
           the Syberia approach to modeling. Rather than using collections of
           loosely-organized scripts, Syberia lets you prescribe what should
           happen in the model construction process, from importing raw production<commit_msg>Regenerate docs.<commit_after><html>

  <head>
    <script src="https://use.typekit.net/xdj4hmw.js"></script>
    <script type="text/javascript">try { Typekit.load({ async: true }); } catch(e) { }</script>

    <link href="https://fonts.googleapis.com/css?family=Source+Code+Pro:400|Roboto:400,300,500" rel="stylesheet" type="text/css">
    <link rel="stylesheet" media="all" href="../stylesheets/main.css" type="text/css" />
    <link rel="stylesheet" media="all" href="../stylesheets/rocco.css" />
    <link rel="stylesheet" media="all" href="../stylesheets/github-markdown.css" />

    <script src="../assets/highlight.pack.js"></script>
    <script type="text/javascript">
      hljs.initHighlightingOnLoad();
    </script>
  </head>

  <body>
  <div class="backdrop tk-proxima-nova">
    <div class="header">
      <div class="header-left"><a href="/">Syberia</a></div>
      <div class="header-right">
        <ul>
          <li><a href="/docs#example">Get Started</a></li>
          <li><a href="/docs/packages">Packages</a></li>
          <li><a href="/docs">Docs</a></li>
        </ul>
      </div>
    </div>
    <br /> <br /> <br />
  </div>
  <div class="colmask leftmenu tk-proxima-nova">
    <div class="colleft">
      <div class="col1">
        <h1>Introduction</h1>

        <p>Let's get you up and running with Syberia!</p>

        <p>If you already know what Syberia is and what you can do it with it, feel free to
        skip to the next <a href="#example">subsection</a>. Otherwise, hopefully this
        brief section will answer some questions.</p>

        <p>Will Syberia help me if I am trying to <b>productionize data science models</b>?
        Yes, but primarily if those models are written in R.
        Syberia is a framework designed specifically to productionize the R
        programming language with rapid iteration cycles, although hybrid models
        consisting of Python, Scala, and other tools will also be possible to deploy.</p>

        <p>Will Syberia help me if I am trying to <b>quantitatively understand a business question</b>?
        While not its primary design, the packages that compose Syberia can be
        used to augment interactive data analysis and introduce the benefits
        of solid developer hygiene such as testability and modularity.</p>

        <p>Will Syberia help me if I am <b>solving an academic problem</b>? 
        Syberia was designed in an industry context but is general enough to be
        useful for other domains, including academia. Although no out-of-the-box 
        Syberia engine currently exists for reproducible research and publication,
        this is a possible path to explore for the future.</p>

        <p>What if I need to solve a <b>big data problem</b> or <b>deploy low latency
          machine learning models</b>? Syberia is designed to act as a command
        center for all data science problems and eventually interact smoothly
        with high-performance and large-size data requirements. Currently, you
        should expect to be able to productionize any R code (10-1000ms latency SLAs on data
        problems with inputs of millions of rows and thousands of features).
        </p>

        <p>What about <b>dashboards</b>? <b>Unstructured or audiovisual data</b>? <b>Deep 
          learning</b>? The world is at your feet but you will have to do some
        custom <a href="https://en.wikipedia.org/wiki/Foreign_function_interface" target="_new">FFI</a> work.
        For the current alpha release, productionization of medium-size structured data
        problems is supported out-of-the-box.</p>

        <h2><a name="example">An example</a></h2>

        <p>Before we get started, let's take a look at what [a realistic example model](https://github.com/syberia/examples/blob/master/models/dev/titanic/titanic.R)
           looks like using the Syberia modeling engine:</p>
        <div class="code">
          <pre>
            <code class="R"><span class="spacer">
# Some of the constants used here live in the main file:
# https://github.com/syberia/examples/blob/master/models/dev/titanic/titanic.R
list( # R's a LISP, so we represent the modeling process with a list. :)
  import = list(
    # For this simple example, we'll use Kaggle's example Titanic data set
    # to build a relatively thorough logistic regression analysis.
    url = list(
      "https://raw.githubusercontent.com/haven-jeon/introduction_to_most_usable_pkgs_in_project/master/bicdata/data/titanic.csv",
      # Treat strings as strings, not factors.
      stringsAsFactors = FALSE
    )
  ), # You can use a variety of data import adapters, like csv or AWS's S3.

  data = list(
    "Factor to character"  = list(column_transformation(as.character), is.factor)
    # One line per feature engineering step seems undue, but it's for a
    # good cause: commenting out to try with and without a step is trivial,
    # and as the parametrization grows longer it's a clear indication
    # you need a new mungebit!
  , "fare_title"           = list(multi_column_transformation(function(title, fare) { ave(fare, title, FUN = mean) }), c("title", "fare"), "title_fare")
  , "Name length variable" = list(new_variable, function(name) nchar(name), "name_length")
  , "Set factors"          = list(column_transformation(factor), c("sex", "embarked"))
  , "Restore levels"       = list(restore_categorical_variables)
  , "Rename dep_var"       = list(renamer, c("survived" = "dep_var"))

    # Parse out a binary feature that indicates if their name has a parenthesis.
    "Has paren in name"       = list(multi_column_transformation(function(name) grepl("(", fixed = TRUE, name)), "name", "has_paren")
    # Turn string length of name on ticket into an integer feature.
   ,"Name length variable"    = list(new_variable, function(name) nchar(name), "name_length")
   # Parse out title feature: c("mr", "ms", "mrs", "master", "rev", "dr", "other")
   ,"Formal title"            = list(regex_factor, "name", "title", cases = titles, fixed = fixed_titles)
   # Parse out ticket type feature: c("pc", "sc", "ca", "sp", "w", "soc", "ston", "line", "paris", "other")
   ,"Ticket type"             = list(regex_factor, "ticket", "ticket_type", cases = tickets, fixed = fixed_tickets)
   # Make a new title_vare variable using stats::ave
   ,"title_fare variable"     = list(new_variable, function(title, fare) { stats::ave(fare, title, FUN = mean) }, "title_fare")
   # Use a helper from mungebits2 package to define a class_fare feature.
   ,"class_fare"              = list(multi_column_transformation(function(klass, fare) { stats::ave(fare, klass, FUN = mean) }), c("pclass", "fare"), "class_fare")
   # Equivalent to using base::transform to derive differences and quotients.
   ,"Some simple derivations" = list(atransform, alist(fare_diff = fare - title_fare, fare_pct = fare / title_fare, fare_diff_class = fare - class_fare, fare_pct_class = fare / class_fare))
   # Some more derivations -- see full file in syberia/examples repo.
   ,"Derived cabin variables" = list(atransform, cabin_derivations)
   ,"Cabin diff and pct"      = list(atransform, alist(fare_diff_cabin = fare - cabin_fare, fare_pct_cabin = fare / cabin_fare))
   # One last logical feature because first letter of cabin seems significant.
   ,"cabin_single_letter"     = list(new_variable, function(cabin_letter) factor(gsub("^(.).*$", "\\1", cabin_letter)), "cabin_single_letter")
   # Mark sex and embarked features as factors.
   ,"Set factors"             = list(!factor, c("sex", "embarked"))
   # Automatically convert our earlier binary feature to a factor.
   ,"Logical to factor"       = list(!as.factor, is.logical)
   # Drop all other character features: the juice has been squeezed!
   ,"Drop character vars"     = list(drop_variables, is.character)
   # A key idea: this stores the levels under the hood for later production use.
   ,"Restore levels"          = list(restore_categorical_variables, is.factor)
   # Finally, the convention is that the dependent is "dep_var".
   # We could also change this in the model stage below to take "survived" instead.
   ,"Rename dep_var"          = list(renamer, c("survived" = "dep_var"))
  ), # Giving a short summary of each step will come in handy later,
     # and is good documentation!

  # Now let's build an ML model.
  model = list(
      "gbm" # The name of our classifier. This is defined elsewhere.
    , .id_var             = "X"
    , distribution        = "bernoulli" # Here we try a binary classification.
    , number_of_trees     = 3000        # Pick some parameters.
    , shrinkage_factor    = 0.005
    , depth               = 5
    , min_observations    = 6
    , train_fraction      = 1
    , bag_fraction        = 0.5
    , cv                  = TRUE
    , cv_folds            = 5
    , number_of_cores     = 4
    , perf_method         = "cv"
    , prediction_type     = "response"
  ),

  export = list( # Exporting the model is done in 1 line of code.
    R    = "titanic_model"  # Export the model object to a global variable.
# , s3   = "path/titanic"   # It's also possible to export to Amazon's S3. 
# , file = "~/path/titanic" # Or a file...
                            # Or any other custom export format you define.
  )
) # And we're done! The model comes to life using the global `run()` helper.
            </span></code>
          </pre>
        </div>

        <p>This example showcases the primary principle behind
           the Syberia approach to modeling. Rather than using collections of
           loosely-organized scripts, Syberia lets you prescribe what should
           happen in the model construction process, from importing raw production
{'fraction_non_alphanumeric': 0.11267083049904149, 'fraction_numerical': 0.005441840331457547, 'mean_word_length': 2.886565729391973, 'pattern_counts': {'":': 0, '<': 197, '<?xml version=': 0, '>': 197, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1251332', 'n_tokens_mistral': 4850, 'n_tokens_neox': 4645, 'n_words': 1661}