<jupyter_start><jupyter_text>Wrapping dependent libraries**StructureAnalysis** is a set of libraries that include statistical models for the analysis of structured data (mainly sequences and tree-structured data):* **StatTool** is a library containing classes concerning parametric  modeling of univariate and multivariate data.* **SequenceAnalysis** is a library containing statistical functions and classes for markovian models (e.g., hidden variable-order Markov and hidden semi-Markov models) and multiple change-point models for sequences.  The **SequenceAnalysis** library depends on the **StatTool** library.These libraries have been extensively used for the identification and characterization of developmental patterns in plants from the tissular to the whole plant scale.Previously interfaced with *AML* (a home-made, domain-specific programming language), some work has been done to switch to *Python*.Nevertheless, the complexity of writing wrappers with **Boost.Python** limited the number of available components in *Python* in comparison to *AML*.One advantage of having a statistical library written in *C++* available in *Python* is that developers can benefit from all *Python* packages.As illustrated with the following figures this is particularly useful for providing visualizations for model quality assessment using -- for example -- the **Matplotlib** *Python* package.<jupyter_code>import matplotlib
%matplotlib nbagg<jupyter_output><empty_output><jupyter_text>We here aim at presenting how dependent libraries can be wrapped.First, we need:* to detect if the operating system (OS) is a Windows OS or a Unix OS.<jupyter_code>import platform
is_windows = any(platform.win32_ver())<jupyter_output><empty_output><jupyter_text>On Windows OSes, the visual studio version used to compile future wrappers must be given.<jupyter_code>if is_windows:
    kwargs = dict(msvc_version = '12.0')
else:
    kwargs = dict()<jupyter_output><empty_output><jupyter_text>* to detect the version of *Python* installed and to save it in the `PYTHON_VERSION` environment variable.<jupyter_code>import os
import sys
PYTHON_VERSION = str(sys.version_info.major) + '.' + str(sys.version_info.minor)<jupyter_output><empty_output><jupyter_text>* to import **AutoWIG**.<jupyter_code>import autowig<jupyter_output><empty_output><jupyter_text>* to import **subprocess**.<jupyter_code>import subprocess<jupyter_output><empty_output><jupyter_text>* to detect the **Git** repository root<jupyter_code>import os
GIT_ROOT = subprocess.check_output('git rev-parse --show-toplevel', shell=True).decode()
GIT_ROOT = GIT_ROOT.replace('/', os.sep).strip()<jupyter_output><empty_output><jupyter_text>The **StatTool** libraryTo compile and install the **StatTool** library, we use the available **Conda** recipes.<jupyter_code>subprocess.call('conda remove -y libstat_tool', shell=True)
CONDA_RECIPE = os.path.join(GIT_ROOT, 'share', 'git', 'StructureAnalysis', 'stat_tool', 'bin', 'conda',
                            'libstat_tool')
subprocess.check_call('conda build --python=' + PYTHON_VERSION + ' ' + CONDA_RECIPE + ' -c statiskit -c defaults --override-channels',
                      shell=True)
subprocess.check_call('conda install -y libstat_tool --use-local -c statiskit -c defaults --override-channels',
                      shell=True)<jupyter_output><empty_output><jupyter_text>Then, we can proceed to the actual generation of wrappers for the **StatTool** library.For this, we create an empty Abstract Semantic Graph (ASG).<jupyter_code>import autowig
asg = autowig.AbstractSemanticGraph()<jupyter_output><empty_output><jupyter_text>Then, we parse headers with relevant compilation flags.<jupyter_code>%%time

import sys
try:
    from path import Path
except:
    from path import path as Path
prefix = Path(sys.prefix).abspath()
if is_windows:
    headers = list((prefix/'include'/'stat_tool').walkfiles('*.h'))
else:
    headers = list((prefix/'include'/'stat_tool').walkfiles('*.h'))
    
flags = ['-x', 'c++', '-std=c++11', '-ferror-limit=0', '-Wno-switch', '-Wno-unknown-attributes']
if is_windows:
    flags.append('-I' + str((prefix/'Library'/'include').abspath()))
else:
    flags.append('-I' + str((prefix/'include').abspath()))
    
if autowig.parser.plugin == 'libclang':
    kwargs['silent'] = True
    
autowig.parser.plugin = 'clanglite'
asg = autowig.parser(asg, headers,
                          flags = flags,
                          bootstrap = 2,
                          **kwargs)<jupyter_output><empty_output><jupyter_text>Since most of **AutoWIG** guidelines are respected, the `default` `controller` implementation could be suitable.Nevertheless, some **AutoWIG** limitations (**AutoWIG** doesn't have a complete knowledge concerning copyable classes) and the requirement of classes defined in the standard *C++* library lead us to implement a new `controller`.<jupyter_code>def stat_tool_controller(asg):
    for noncopyable in ['class ::std::basic_streambuf< char, struct ::std::char_traits< char > >',
                        'class ::std::codecvt< char, char, __mbstate_t >',
                        'class ::std::locale::facet',
                        'class ::std::locale::id',
                        'class ::std::ctype< char >',
                        'class ::std::ios_base',
                        'class ::std::basic_istream< char, struct ::std::char_traits< char > >',
                        'class ::std::basic_ostream< char, struct ::std::char_traits< char > >',
                        'class ::std::basic_ostringstream< char, struct ::std::char_traits< char >, class ::std::allocator< char > >',
                        'class ::std::basic_ios< char, struct ::std::char_traits< char > >',
                        'class ::std::basic_stringbuf< char, struct ::std::char_traits< char >, class ::std::allocator< char > >']:
        if noncopyable in asg:
            asg[noncopyable].is_copyable = False
    for cls in asg.classes():
        for fld in cls.fields(access='public'):
            if fld.qualified_type.unqualified_type.globalname == 'class ::std::locale::id':
                fld.boost_python_export = False
    for template in ['class ::std::initializer_list',
                     'class ::std::reverse_iterator',
                     'class ::std::move_iterator',
                     'class ::std::allocator']:
        for specialization in asg[template].specializations():
            specialization.boost_python_export = False
    for template in ['class ::std::basic_ios',
                     'class ::std::basic_string']:
        for specialization in asg[template].specializations():
            for mtd in specialization.methods():
                mtd.boost_python_export = False
    asg['::std::ios_base::openmode'].qualified_type.boost_python_export = True
    return asg<jupyter_output><empty_output><jupyter_text>This `controller` is then dynamically registered and used on the ASG.<jupyter_code>%%time

autowig.controller['stat_tool'] = stat_tool_controller
autowig.controller.plugin = 'stat_tool'
asg = autowig.controller(asg)<jupyter_output><empty_output><jupyter_text>In order to wrap the library we need to select the `boost_python_internal` `generator` implementation.<jupyter_code>%%time

autowig.generator.plugin = 'boost_python_internal'
wrappers = autowig.generator(asg,
                             module = os.path.join(GIT_ROOT, 'share', "git", "StructureAnalysis",
                                                   "stat_tool", "src", "py", "wrapper", "_stat_tool.cpp"),
                             decorator = os.path.join(GIT_ROOT, 'share', "git", "StructureAnalysis",
                                                      "stat_tool", "src", "py", "stat_tool", "_stat_tool.py"),
                             prefix = 'wrapper_')<jupyter_output><empty_output><jupyter_text>The wrappers are only generated in-memory.It is therefore needed to write them on the disk to complete the process.<jupyter_code>%%time

wrappers.write()<jupyter_output><empty_output><jupyter_text>Here is the list of the generated wrappers (untracked files).<jupyter_code>REPO_DIR = os.path.join(GIT_ROOT, 'share', 'git', 'StructureAnalysis')
!git -C {REPO_DIR} status<jupyter_output><empty_output><jupyter_text>Here is an example of the generated wrappers.<jupyter_code>WRAPPER = os.path.join(GIT_ROOT, 'share', 'git', 'StructureAnalysis', 'stat_tool', 'src', 'py', 'wrapper',
                       '_stat_tool.cpp')
!pygmentize {WRAPPER}<jupyter_output><empty_output><jupyter_text>In order to wrap a *C++* library, that will be used as a dependency by other libraries, the user needs to save the ASG resulting from the wrapping process.We therefore use the **pickle** *Python* package for serializing the **StatTool** ASG in the `'StructureAnalysis/stat_tool/ASG.pkl'` file.<jupyter_code>import pickle
with open(os.path.join(GIT_ROOT, 'share', "git", "StructureAnalysis", "stat_tool", "ASG.pkl"), "wb") as filehandler:
    pickle.dump(asg, filehandler)<jupyter_output><empty_output><jupyter_text>Once the wrappers are written on disk, we need to compile and install the *Python* bindings.To do so, we use available **Conda** recipes.<jupyter_code>subprocess.call('conda remove -y python-stat_tool', shell=True)
CONDA_RECIPE = os.path.join(GIT_ROOT, 'share', 'git', 'StructureAnalysis', 'stat_tool', 'bin', 'conda',
                            'python-stat_tool')
subprocess.check_call('conda build --python=' + PYTHON_VERSION + ' ' + CONDA_RECIPE + ' -c statiskit -c defaults --override-channels',
                      shell=True)
subprocess.check_call('conda install -y python-stat_tool --use-local -c statiskit -c defaults --override-channels',
                      shell=True)<jupyter_output><empty_output><jupyter_text>Finally, we can hereafter use the *C++* library in the *Python* interpreter.<jupyter_code>import stat_tool
import os
%reload_ext stat_tool.mplotlib
%reload_ext stat_tool.aml
meri1 = stat_tool.Histogram(str(os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                         "stat_tool", "share", "data", "meri1.his")))
meri2 = stat_tool.Histogram(str(os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                         "stat_tool", "share", "data", "meri2.his")))
meri3 = stat_tool.Histogram(str(os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                         "stat_tool", "share", "data", "meri3.his")))
meri4 = stat_tool.Histogram(str(os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                         "stat_tool", "share", "data", "meri4.his")))
meri5 = stat_tool.Histogram(str(os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                         "stat_tool", "share", "data", "meri5.his")))
meri0 = stat_tool.Merge(meri1, meri2, meri3, meri4, meri5)
mixt = stat_tool.MixtureEstimation(meri0, 4, "BINOMIAL",
                                   display=True)
fig = mixt.plot()<jupyter_output><empty_output><jupyter_text>The **SequenceAnalysis** libraryOnce the wrapping of the **StatTool** library is performed, we need to compile and install the **SequenceAnalysis** library.For this, we use the available **Conda** recipes.<jupyter_code>subprocess.call('conda remove -y libsequence_analysis', shell=True)
CONDA_RECIPE = os.path.join(GIT_ROOT, 'share', 'git', 'StructureAnalysis', 'sequence_analysis', 'bin', 'conda',
                            'libsequence_analysis')
subprocess.check_call('conda build --python=' + PYTHON_VERSION + ' ' + CONDA_RECIPE + ' -c statiskit -c defaults --override-channels',
                      shell=True)
subprocess.check_call('conda install -y libsequence_analysis --use-local -c statiskit -c defaults --override-channels',
                      shell=True)<jupyter_output><empty_output><jupyter_text>Then, we can proceed to the actual generation of wrappers for the **SequenceAnalysis** *C++* library.In order to wrap a *C++* library that has dependencies, the user need to combine the ASGs resulting from the wrapping of its dependencies before performing its own wrapping.For this, we create an empty Abstract Semantic Graph (ASG).<jupyter_code>asg = autowig.AbstractSemanticGraph()<jupyter_output><empty_output><jupyter_text>Then, we use the **pickle** *Python* package for de-serializing the **StatTool** ASG and merge it in the current ASG.<jupyter_code>%%time

with open(os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis", "stat_tool", "ASG.pkl"), "rb") as filehandler:
    asg.merge(pickle.load(filehandler))<jupyter_output><empty_output><jupyter_text>Then, we parse headers with relevant compilation flags.<jupyter_code>%%time

if is_windows:
    headers = list((prefix/'include'/'sequence_analysis').walkfiles('*.h'))
else:
    headers = list((prefix/'include'/'sequence_analysis').walkfiles('*.h'))
    
flags = ['-x', 'c++', '-std=c++11', '-ferror-limit=0', '-Wno-switch', '-Wno-unknown-attributes']
if is_windows:
    flags.append('-I' + str((prefix/'Library'/'include').abspath()))
else:
    flags.append('-I' + str((prefix/'include').abspath()))
    
asg = autowig.parser(asg, headers,
                          flags = flags,
                          bootstrap = 2,
                          **kwargs)<jupyter_output><empty_output><jupyter_text>Since most of AutoWIG guidelines are respected, the default controller implementation could be suitable. Nevertheless, some errors induced by the `::sequence_analysis::Sequence::segment_profile_ascii_write` method lead us to implement a new controller. Since most of AutoWIG guidelines are respected, the default controller implementation is suitable.<jupyter_code>%%time

asg = autowig.controller(asg)<jupyter_output><empty_output><jupyter_text>In order to wrap the library we need to select the `boost_python_internal` `generator` implementation.<jupyter_code>%%time

autowig.generator.plugin = 'boost_python_internal'
wrappers = autowig.generator(asg,
                             module = os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                                   "sequence_analysis", "src", "py", "wrapper",
                                                   "_sequence_analysis.cpp"),
                             decorator = os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                                      "sequence_analysis", "src", "py", "sequence_analysis",
                                                      "_sequence_analysis.py"),
                             prefix = 'wrapper_')<jupyter_output><empty_output><jupyter_text>The wrappers are only generated in-memory.It is therefore needed to write them on the disk to complete the process.<jupyter_code>%%time

wrappers.write()<jupyter_output><empty_output><jupyter_text>Here is the list of the generated wrappers (untracked files).<jupyter_code>!git -C {REPO_DIR} status<jupyter_output><empty_output><jupyter_text>Here is an example of the generated wrappers.<jupyter_code>WRAPPER = os.path.join(GIT_ROOT, 'share', 'git', 'StructureAnalysis', 'sequence_analysis', 'src', 'py', 'wrapper',
                       '_sequence_analysis.cpp')
!pygmentize {WRAPPER}<jupyter_output><empty_output><jupyter_text>Once the wrappers are written on disk, we need to compile and install the *Python* bindings.To do so, we use available **Conda** recipes.<jupyter_code>subprocess.call('conda remove -y python-sequence_analysis', shell=True)
CONDA_RECIPE = os.path.join(GIT_ROOT, 'share', 'git', 'StructureAnalysis', 'sequence_analysis', 'bin', 'conda',
                            'python-sequence_analysis')
subprocess.check_call('conda build --python=' + PYTHON_VERSION + ' ' + CONDA_RECIPE + ' -c statiskit -c defaults --override-channels',
                      shell=True)
subprocess.check_call('conda install -y python-sequence_analysis --use-local -c statiskit -c defaults --override-channels',
                      shell=True)<jupyter_output><empty_output><jupyter_text>Finally, we can hereafter use the *C++* library in the *Python* interpreter.<jupyter_code>import sequence_analysis
%reload_ext sequence_analysis.mplotlib
%reload_ext sequence_analysis.aml
seq = sequence_analysis.Sequences(str(os.path.join(GIT_ROOT, "share", "git", "StructureAnalysis",
                                               "sequence_analysis", "share", "data",
                                                "well_log_filtered_indexed.seq")))<jupyter_output><empty_output><jupyter_text>This last cell is by default not executed since the algorithm used is computer intensive.To run this last cell, edit the metadata of the cell by switching the cell type from `Raw NBConvert` to `Code`.<jupyter_code>seq.segmentation(1, 80, "GAUSSIAN", "LIKELIHOOD_SLOPE", min_nb_segment=30)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1225565691268807, 'fraction_numerical': 0.0021324487619950243, 'mean_word_length': 3.6754361672666853, 'pattern_counts': {'":': 0, '<': 141, '<?xml version=': 0, '>': 141, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '419603', 'license': 'Apache-2.0', 'n_tokens_mistral': 4997, 'n_tokens_neox': 4874, 'n_words': 1294, 'path': 'share/jupyter/dependent.ipynb', 'repo_name': 'StatisKit/FP17', 'size': '16882'}