<jupyter_start><jupyter_text>CmdStanPy Tutorial Workflow OutlineGiven a dataset and a model specification written as a Stan program, the CmdStanPy workflow is: Assemble input data as either:  + A Python `dict` object consisting of key-value pairs where the key corresponds to Stan data variables and the value is of the correct type and shape.  + An existing data file on disk in either JSON or Rdump format. Compile Stan program: function `compile_model`  + Takes as input a Stan program and produces the corresponding c++ executable.  + Returns `Model` object which wraps program, c++ executable filenames or raises exception.  + Both Stan program and c++ executable stored on disk. Run sampler: function `sample`  + Invokes Stan's NUTS-HMC sampler to condition model on input data, produces output csv files.  + Creates lightweight object `RunSet` which wraps sampler arguments, per-chain outcome, output files.  + Runs any number of chains - default is 4 chains.  + Validates that all chains run to completion, produce expected number of draws.  + Returns `RunSet` if sample is valid, else raises exception. Assemble fit in-memory: function `get_drawset`  + Sampler output consists of csv files  + `RunSet` method `assemble_sample` reads sampler output csv files into 3-D numpy ndarray (draws, chains, columns)  + Function `get_drawset` flattens chains, returns a pandas DataFrame one row per draw, one column per parameters.  + `get_drawset` argument `params` filters DataFrame columns by list of parameter names. Summarize posterior fit:  function `summary`  + Invokes CmdStan's `bin/stansummary` utility  + Returns a Pandas DataFrame with estimates of posterior means, standard deviations, Monte-Carlo standard error, effective sample size, and convergence diagnostic statistic for all parameters in the model. Check for problems: function `diagnose`  + Invokes CmdStan's `bin/diagnose` utility  + Prints output to console  + Checks for following problems:    + transitions that hit the maximum treedepth    + divergent transitions    + low E-BFMI values (sampler transitions HMC potential energy)    + low effective sample sizes    + high R-hat values  Installation* Install Python package from PyPI or directly from GitHub:  + `pip install --upgrade cmdstanpy`  + `pip install -e git+https://github.com/stan-dev/cmdstanpy`* CmdStanPy uses CmdStan directly to compile and run Stan programs, therefore CmdStan must be installed locally.  + if you have a working installation of CmdStan, set environment variable `CMDSTAN` to the full path to the top-level CmdStan directory.    + if you don't already have CmdStan installed, run Python script `install_cmdstan` which downloads and compiles the latest release from https://github.com/stan-dev/cmdstan/releases.  By default this installs the latest version of CmdStan in the location `~/.cmdstanpy`.  Flags -d and -v are used to specify the directory and version, respectively.  Example 1:  Compile and run CmdStan example model `bernoulli.stan`, data `bernoulli.data.json`Every CmdStan release has an `examples/bernoulli` directory which contains the Stan model and test data files.  In this example we compile the model and run the sampler on the model and data.  Import relevant classes and methods<jupyter_code>import os
import os.path
from cmdstanpy import cmdstan_path, compile_model, sample, get_drawset, summary, diagnose<jupyter_output><empty_output><jupyter_text>Compile model, specify data or data file The CmdStanPy directory `examples/bernoulli` contains the model and data files.<jupyter_code>bernoulli_path = os.path.join(cmdstan_path(), 'examples', 'bernoulli')
bernoulli_stan = os.path.join(bernoulli_path, 'bernoulli.stan')<jupyter_output><empty_output><jupyter_text>The `compile_model` functions compiles the Stan program and returns a `Model` object.<jupyter_code>bernoulli_model = compile_model(bernoulli_stan)
print(bernoulli_model)<jupyter_output>Model(name=bernoulli,  stan_file="/Users/mitzi/.cmdstanpy/cmdstan-2.19.1/examples/bernoulli/bernoulli.stan", exe_file="/Users/mitzi/.cmdstanpy/cmdstan-2.19.1/examples/bernoulli/bernoulli")<jupyter_text>Input data is either a Python `Dict` with entries corresponding to input data values, or it can be a file in JSON or Rdump format.<jupyter_code>bern_json = os.path.join(bernoulli_path, 'bernoulli.data.json')<jupyter_output><empty_output><jupyter_text>If a `Dict` is specified, CmdStanPy writes it to a temp file in JSON format.<jupyter_code>bern_data = { "N" : 10, "y" : [0,1,0,0,0,0,0,0,0,1] }<jupyter_output><empty_output><jupyter_text>Run the HMC-NUTS sampler on the model and data The `sample` function runs the NUTS-HMC sampler and returns a `RunSet` object.<jupyter_code>bern_fit = sample(bernoulli_model, data=bern_data)<jupyter_output>input data tempfile: /var/folders/sc/0f0wdc_11_xgjs2v52g20fvr0000gn/T/tmpggqfg6d4/tmpjvatu0sj.json
start chain 1.  
finish chain 1.  
start chain 2.  
finish chain 2.  
start chain 3.  
finish chain 3.  
start chain 4.  
finish chain 4.<jupyter_text>By default, the sample function runs 4 sampler chains.  The `chains` argument specifies the number of chains to run.  The `cores` argument specifies the number of processes to run in parallel.<jupyter_code>bern_fit = sample(bernoulli_model, chains=5, cores=3, data=bern_data)<jupyter_output>input data tempfile: /var/folders/sc/0f0wdc_11_xgjs2v52g20fvr0000gn/T/tmpggqfg6d4/tmp0f38xgj5.json
start chain 1.  start chain 2.  start chain 3.  


finish chain 3.  
start chain 4.  
finish chain 1.  
finish chain 2.  start chain 5.  

finish chain 4.  
finish chain 5.<jupyter_text>Summarize or save the results The `summary` function returns output of CmdStan bin/stansummary as pandas.DataFrame:<jupyter_code>summary(bern_fit)<jupyter_output><empty_output><jupyter_text>The `diagnose` function prints diagnostics to console:<jupyter_code>diagnose(bern_fit)<jupyter_output>No problems detected.<jupyter_text>The `get_drawset` function returns a pandas.DataFrame, one draw per row.<jupyter_code>bern_drawset = get_drawset(bern_fit)<jupyter_output><empty_output><jupyter_text>By default, `get_drawset` returns a DataFrame which contains all columns from the sampler's csv output file, i.e., it contains both the sampler state and the values for all parameter, transformed parameter, and generated quantities variables.<jupyter_code>bern_drawset.shape, bern_drawset.columns<jupyter_output><empty_output><jupyter_text>The `get_drawset` function argument `params` takes a list of parameter or column names:<jupyter_code>thetas = get_drawset(bern_fit, params=['theta'])
thetas.shape

thetas[0:3]
bern_drawset.theta.plot.density()<jupyter_output><empty_output><jupyter_text>Access to sampler output via `RunSet ` methods and attributes sampleThe `sample` property is a 3-D numpy ndarray which contains all draws across all chains.  This array is created only as needed; therefore the first time that this property is accessed CmdStanPy will read in the contents of the sampler's csv output files.  Because the csv output files also contain stepsize and metric information, the `stepsize` and `metric` arrays will also be created.The ndarray is stored column major format so that values for each parameter are stored contiguously in memory, likewise all draws from a chain are contiguous.  Thus the dimensions of the ndarray are arranged as follows:  (draws, chains, columns):<jupyter_code>bern_fit.sample
bern_fit.sample.shape<jupyter_output><empty_output><jupyter_text>Python's index slicing operations can be used to access the information by chain.For example, to select all draws and all output columns from the first chain,we specify the chain index (2nd index dimension).  As arrays indexing starts at 0,the index '0' corresponds to the first chain in the ``RunSet``.The following expression selects the first 3 draws from chain 1 for the parameter `theta`:<jupyter_code>bern_fit.column_names[7], bern_fit.sample[0:3,0,7]<jupyter_output><empty_output><jupyter_text>stepsizeThe `stepsize` property is a 1-D numpy ndarray which contains the stepsize used by the sampler for each chain.  This array is created at the same time as the `sample` and `metric` arrays are created.At the end of adaptation, the stepsize for the 4 chains in this example is:<jupyter_code>bern_fit.stepsize<jupyter_output><empty_output><jupyter_text>metric_type, metricThe `metric` property is an numpy ndarray which contains the metric used by the sampler for each chain.  This array is created at the same time as the `sample` and `stepsize` arrays are created.At the end of adaptation, the metric for the 4 chains in this example is:<jupyter_code>bern_fit.metric_type,  bern_fit.metric<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07157628676470588, 'fraction_numerical': 0.010914522058823529, 'mean_word_length': 5.675613496932515, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '444129', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2712, 'n_tokens_neox': 2578, 'n_words': 1001, 'path': 'cmdstanpy_tutorial.ipynb', 'repo_name': 'codeLovingYogi/cmdstanpy', 'size': '8704'}