<commit_before># -*- org-adapt-indentation: nil; org-edit-src-content-indentation: 0; -*-
#+TITLE: Using SciGraph
#+OPTIONS: num:nil
#+STARTUP: showall

This readme consolidates a number of different processes for deploying
ontologies to SciGraph into a single workflow that avoids most of the
hidden pitfalls we have encountered over the past 4 years.

* Using this file
You can either [[https://orgmode.org/manual/Extracting-Source-Code.html][tangle]]
this file in emacs using =C-c C-v t= or you can tangle
the whole file from the command line using with the following.

#+name: tangle-this-file
#+header: :results none
#+begin_src bash :var THIS_FILE=(let ((home (expand-file-name "~")) (bfn (buffer-file-name))) (when (string-prefix-p home bfn) (concat "~" (substring bfn (length home) nil)))) :noweb yes
: ${THIS_FILE:="README.org"}
emacs --batch \
      --load org \
      --load ob-shell \
      --eval "(unless ((lambda (a b) (or (string> a b) (string= a b))) (org-version) \"9.3\") (message (format \"your org is too old! %s < 9.3 \" (org-version))) (kill-emacs 1))" \
      --eval "(org-babel-tangle-file \"${THIS_FILE}\")"
#+end_src
The core functionality is tangled to [[file:./bin/scigraph-functions.sh]]. \\
It can be sourced in a shell or from a script by running \\
=source ~/git/pyontutils/nifstd/scigraph/bin/scigraph-functions.sh= \\
to make the functions defined in this file available for use.
** Developing this file                                            :noexport:
You should also run =shellcheck= on the tangled file if you make any changes.
You don't usually have to worry about green issues, and there are a number of
false positives because we are dealing with some crazy local vs remote expansion.
#+name: run-shellcheck
#+begin_src bash :noweb yes :eval never
shellcheck "<<path-scigraph-functions()>>"
#+end_src
* Setup
:properties:
:visibility: folded
:end:
** Build host
*** Gentoo
:PROPERTIES:
:CUSTOM_ID: Setup:Build host:Gentoo
:END:
#+begin_src bash
layman -a tgbugs-overlay

emerge \
pyontutils \
rpmdevtools \
scigraph-bin \
yq
#+end_src
*** Ubuntu
:PROPERTIES:
:CUSTOM_ID: Setup:Build host:Ubuntu
:END:
#+begin_src bash
apt install \
rpm \
python3-dev \
python3-pip

pip install --user \
pyontutils \
yq
#+end_src
** Services host
*** Gentoo
:PROPERTIES:
:CUSTOM_ID: Setup:Services host:Gentoo
:END:
#+begin_src bash
layman -a tgbugs-overlay

emerge \
scigraph-bin \
yq
#+end_src
*** AWS Linux 2
:PROPERTIES:
:CREATED:  [2020-01-06 Mon 14:27]
:CUSTOM_ID: Setup:Services host:AWS Linux 2
:END:
Installing the scigraph RPM also handles the creation and configuration
of all relevant variables for operation on RHEL-like systems.
See the [[https://github.com/tgbugs/pyontutils/blob/master/nifstd/scigraph/scigraph.spec][spec file]]
for details.

#+name: dir-tramp-sudo
#+header: :noweb yes :results none
#+begin_src emacs-lisp :exports none
(defun :dir-tramp-sudo (ssh-config-hostname)
  (format "/ssh:%s|sudo:%s:" ssh-config-hostname ssh-config-hostname))
#+end_src

#+name: aws-block
#+begin_src bash :dir /ssh:hostname|sudo:hostname:
yum install -y \
python3 \
python3-pip

pip3 install \
yq  # yq is usually only needed on config build systems
    # however if something goes wrong and an in place change
    # needs to be made then editing the raw and regenerating
    # is the preferred course of action
#+end_src

# #+CALL: aws-block() :dir /ssh:aws-scigraph|sudo:aws-scigraph: :eval never
# #+CALL: aws-block() :dir (:dir-tramp-sudo "aws-scigraph") :eval never
* User setup
:properties:
:visibility: folded
:end:
This setup should be completed on the build host or on the ops host
that talks to both the build host and the services host. Currently the
build host and ops host are conflated.
** git repos
There are currently still a couple of repos that are needed for these workflows.
Most are needed configuration files, but pyontutils is needed for this README as well.
#+begin_src bash :eval never
pushd ~/git
git clone https://github.com/tgbugs/pyontutils.git
git clone https://github.com/tgbugs/tgbugs-overlay.git
git clone https://github.com/SciCrunch/sparc-curation.git
popd
#+end_src
** ssh config
Set =ip-aws-scigraph= here
#+name: ip-aws-scigraph
: 127.0.0.1
<commit_msg>scigraph readme add orgstrap until new version of emacs is ready<commit_after># -*- org-adapt-indentation: nil; org-edit-src-content-indentation: 0; -*-
#+TITLE: Using SciGraph
#+OPTIONS: num:nil
#+STARTUP: showall

This readme consolidates a number of different processes for deploying
ontologies to SciGraph into a single workflow that avoids most of the
hidden pitfalls we have encountered over the past 4 years.

* Using this file
Before you can use this file you will need to complete the [[#setup][Setup section]]. \\

You can either [[https://orgmode.org/manual/Extracting-Source-Code.html][tangle]]
this file in emacs using =C-c C-v t= or you can tangle
the whole file from the command line using with the following.

#+name: tangle-this-file
#+header: :results none
#+begin_src bash :var THIS_FILE=(let ((home (expand-file-name "~")) (bfn (buffer-file-name))) (when (string-prefix-p home bfn) (concat "~" (substring bfn (length home) nil)))) :noweb yes
: ${THIS_FILE:="README.org"}    # set to the local path to this file
emacs --batch \
      --load org \
      --load ob-shell \
      --eval "(unless ((lambda (a b) (or (string> a b) (string= a b))) (org-version) \"9.3\") (message (format \"your org is too old! %s < 9.3 \" (org-version))) (kill-emacs 1))" \
      --eval "(org-babel-tangle-file \"${THIS_FILE}\")"
#+end_src
The core functionality is tangled to [[file:./bin/scigraph-functions.sh]]. \\
It can be sourced in a shell or from a script by running \\
=source ~/git/pyontutils/nifstd/scigraph/bin/scigraph-functions.sh= \\
to make the functions defined in this file available for use.
** Developing this file                                            :noexport:
You should also run =shellcheck= on the tangled file if you make any changes.
You don't usually have to worry about green issues, and there are a number of
false positives because we are dealing with some crazy local vs remote expansion.
#+name: run-shellcheck
#+begin_src bash :noweb yes :eval never
shellcheck "<<path-scigraph-functions()>>"
#+end_src
* Setup
:properties:
:custom_id: setup
:visibility: folded
:end:
** Build host
*** Gentoo
:PROPERTIES:
:CUSTOM_ID: Setup:Build host:Gentoo
:END:
#+begin_src bash
layman -a tgbugs-overlay

emerge \
pyontutils \
rpmdevtools \
scigraph-bin \
yq
#+end_src
*** Ubuntu
:PROPERTIES:
:CUSTOM_ID: Setup:Build host:Ubuntu
:END:
#+begin_src bash
apt install \
rpm \
python3-dev \
python3-pip

pip install --user \
pyontutils \
yq
#+end_src
** Services host
*** Gentoo
:PROPERTIES:
:CUSTOM_ID: Setup:Services host:Gentoo
:END:
#+begin_src bash
layman -a tgbugs-overlay

emerge \
scigraph-bin \
yq
#+end_src
*** AWS Linux 2
:PROPERTIES:
:CREATED:  [2020-01-06 Mon 14:27]
:CUSTOM_ID: Setup:Services host:AWS Linux 2
:END:
Installing the scigraph RPM also handles the creation and configuration
of all relevant variables for operation on RHEL-like systems.
See the [[https://github.com/tgbugs/pyontutils/blob/master/nifstd/scigraph/scigraph.spec][spec file]]
for details.

#+name: dir-tramp-sudo
#+header: :noweb yes :results none
#+begin_src emacs-lisp :exports none
(defun :dir-tramp-sudo (ssh-config-hostname)
  (format "/ssh:%s|sudo:%s:" ssh-config-hostname ssh-config-hostname))
#+end_src

#+name: aws-block
#+begin_src bash :dir /ssh:hostname|sudo:hostname:
yum install -y \
python3 \
python3-pip

pip3 install \
yq  # yq is usually only needed on config build systems
    # however if something goes wrong and an in place change
    # needs to be made then editing the raw and regenerating
    # is the preferred course of action
#+end_src

# #+CALL: aws-block() :dir /ssh:aws-scigraph|sudo:aws-scigraph: :eval never
# #+CALL: aws-block() :dir (:dir-tramp-sudo "aws-scigraph") :eval never
* User setup
:properties:
:visibility: folded
:end:
This setup should be completed on the build host or on the ops host
that talks to both the build host and the services host. Currently the
build host and ops host are conflated.
** git repos
There are currently still a couple of repos that are needed for these workflows.
Most are needed configuration files, but pyontutils is needed for this README as well.
#+begin_src bash :eval never
pushd ~/git
git clone https://github.com/tgbugs/orgstrap.git
git clone https://github.com/tgbugs/pyontutils.git
git clone https://github.com/tgbugs/tgbugs-overlay.git
git clone https://github.com/SciCrunch/sparc-curation.git
popd
~/git/orgstrap/orgstrap --user  # needed to obtain newer version of org-mode
#+end_src
** ssh config
Set =ip-aws-scigraph= here
#+name: ip-aws-scigraph
: 127.0.0.1

{'fraction_non_alphanumeric': 0.12020489470688674, 'fraction_numerical': 0.007285145133750712, 'mean_word_length': 4.1140861466821885, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2197657', 'n_tokens_mistral': 3062, 'n_tokens_neox': 2882, 'n_words': 1050}