<commit_before>* pg-sphinx
  
  Pg-sphinx is a module for PostgreSQL which allows to integrate Sphinx search engine.

** Installation

*** Note for non-linux users

  GNU libtool is used to build this software. It's possible you need to define LIBTOOL variable.

  #+BEGIN_SRC sh
  export LIBTOOL=glibtool
  #+END_SRC
  
*** Compile

  #+BEGIN_SRC sh
  make
  #+END_SRC

*** Install
  
  #+BEGIN_SRC sh
  sudo make install
  #+END_SRC

*** Define sphinx functions in your database

  Superuser is required.

  #+BEGIN_SRC sh
  psql -U postgres mydatabase < declare.sql
  #+END_SRC

*** Uninstall

  #+BEGIN_SRC sh
  sudo make uninstall
  #+END_SRC

** Functions

*** Search query

  #+BEGIN_SRC sql
  sphinx_select(
      /*index*/     varchar,
      /*query*/     varchar,
      /*condition*/ varchar,
      /*order*/     varchar,
      /*offset*/    int,
      /*limit*/     int,
      /*options*/   varchar)
  #+END_SRC

  Returns pairs (id, weight).

*** Update data

  #+BEGIN_SRC sql
  sphinx_replace(
      /*index*/     varchar,
      /*id*/        int,
      /*data*/      varchar[])
  #+END_SRC

  Updates document with specified id. Data array must have following format:
  ARRAY['key1', 'value2', ...]

*** Delete data

  #+BEGIN_SRC sql
  sphinx_delete(
      /*index*/     varchar,
      /*id*/        int)
  #+END_SRC

  Removes specified document.
<commit_msg>example for snippet function<commit_after>* pg-sphinx
  
  Pg-sphinx is a module for PostgreSQL which allows to integrate Sphinx search engine.

** Installation

*** Note for non-linux users

  GNU libtool is used to build this software. It's possible you need to define LIBTOOL variable.

  #+BEGIN_SRC sh
  export LIBTOOL=glibtool
  #+END_SRC
  
*** Compile

  #+BEGIN_SRC sh
  make
  #+END_SRC

*** Install
  
  #+BEGIN_SRC sh
  sudo make install
  #+END_SRC

*** Define sphinx functions in your database

  Superuser is required.

  #+BEGIN_SRC sh
  psql -U postgres mydatabase < declare.sql
  #+END_SRC

*** Uninstall

  #+BEGIN_SRC sh
  sudo make uninstall
  #+END_SRC

** Functions

*** Search query

  #+BEGIN_SRC sql
  sphinx_select(
      /*index*/     varchar,
      /*query*/     varchar,
      /*condition*/ varchar,
      /*order*/     varchar,
      /*offset*/    int,
      /*limit*/     int,
      /*options*/   varchar)
  #+END_SRC

  Returns pairs (id, weight).

*** Update data

  #+BEGIN_SRC sql
  sphinx_replace(
      /*index*/     varchar,
      /*id*/        int,
      /*data*/      varchar[])
  #+END_SRC

  Updates document with specified id. Data array must have following format:
  ARRAY['key1', 'value2', ...]

*** Delete data

  #+BEGIN_SRC sql
  sphinx_delete(
      /*index*/     varchar,
      /*id*/        int)
  #+END_SRC

  Removes specified document.

*** Get snippet

  #+BEGIN_SRC sql
  sphinx_snippet(
      /*index*/     varchar,
      /*query*/     varchar,
      /*data*/      varchar,
      /*before*/    varchar,
      /*after*/     varchar)
  #+END_SRC

  Returns snippets for a given data and search query.

  Example:

  #+BEGIN_SRC sql
  SELECT sphinx_snippet('blog_posts', 'photo', 'There are photos from monday meeting', '<b>', '</b>')
  #+END_SRC

  This query will return following text:

  #+BEGIN_SRC sql
  'There are <b>photos</b> from monday meeting'
  #+END_SRC

{'fraction_non_alphanumeric': 0.11821764170960897, 'fraction_numerical': 0.0012124886329190664, 'mean_word_length': 2.4375, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7262632', 'n_tokens_mistral': 1180, 'n_tokens_neox': 1122, 'n_words': 338}