<commit_before>.. highlight:: none

Storm Reference - Pivot Operators
=================================

Pivot operators operate on the output of a **previous Storm query.**

The operators below can be used to pivot from one set of nodes to another set of nodes within a Synapse hypergraph based on a set of specified criteria.

All of these operators are defined in storm.py_.

See the `Storm Syntax Conventions`__ for an explanation of the usage format used below.

Where specific query examples are given, they are commonly provided in pairs using operator syntax followed by the equivalent macro syntax (if available).

pivot()
-------
Returns a set of nodes that share a specified property of the same type / valu as the original set of nodes.

Optional parameters:

* **Return limit:** specify the maximum number of nodes returned by the ``pivot()`` query.

  * ``limit=`` (operator syntax)

**Operator syntax:**

.. parsed-literal::

  **pivot(** [ *<srcprop>* **,** ] *<dstprop>* [ **, limit=** *<limit>* ] **)**

**Macro syntax:**

.. parsed-literal::

  [ *<srcprop>* ] **->** *<dstprop>*

**Examples:**

* Pivot from a set of domains (``inet:fqdn`` nodes) in the working set to the DNS A records for those domains:
  ::
    pivot( inet:fqdn, inet:dns:a:fqdn )
    
    pivot( inet:dns:a:fqdn )
    
    inet:fqdn  ->  inet:dns:a:fqdn
    
    -> inet:dns:a:fqdn

* Pivot from a set of domains in the working set to the DNS A records for those domains, but limit the results to 10 nodes:
  ::
    pivot( inet:fqdn, inet:dns:a:fqdn, limit=10 )
    
    pivot( inet:dns:a:fqdn, limit=10 )

* Pivot from a set of domains in the working set to the DNS A records for those domains, and from the IP addresses in the DNS A records to the associated set of IPv4 nodes (e.g., pivot from a set of domains to the set of IP addresses the domains resolved to).
  ::
    pivot( inet:fqdn, inet:dns:a:fqdn ) pivot( inet:dns:a:ipv4, inet:ipv4 )
      
    pivot( inet:dns:a:fqdn ) pivot( :ipv4, inet:ipv4 )
    
    inet:fqdn -> inet:dns:a:fqdn inet:dns:a:ipv4 
      -> inet:ipv4
    
    -> inet:dns:a:fqdn :ipv4 -> inet:ipv4

* Pivot from a set of domains in the working set to the set of subdomains for those domains:
  ::
    pivot( inet:fqdn, inet:fqdn:domain )
    
    pivot( inet:fqdn:domain )
    
    inet:fqdn -> inet:fqdn:domain
    
    -> inet:fqdn:domain

* Pivot from a set of email addresses in the working set to the set of domains registered to those email addresses.
  ::
    pivot( inet:email, inet:whois:regmail:email )
      pivot( inet:whois:regmail:fqdn, inet:fqdn )
    
    pivot( inet:whois:regmail:email ) pivot( :fqdn, inet:fqdn )
    
    inet:email -> inet:whois:regmail:email 
      inet:whois:regmail:fqdn -> inet:fqdn
    
    -> inet:whois:regmail:email :fqdn -> inet:fqdn

* Pivot from a set of email addresses in the working set to the set of whois records associated with those email addresses.
  ::
    pivot( inet:email, inet:whois:contact:email )
      pivot( inet:whois:contact:rec, inet:whois:rec )
    
    pivot( inet:whois:contact:email ) pivot( :rec, inet:whois:rec )
    
    inet:email -> inet:whois:contact:email inet:whois:contact:rec
      -> inet:whois:rec
    
    -> inet:whois:contact:email inet:whois:contact:rec -> inet:whois:rec

**Usage notes:**

* If the source property for the pivot is the primary property of the working set of nodes, the *<srcprop>* can be omitted from both Operator and Macro syntax.
* If the source property for the pivot is a secondary property of the working set of nodes, relative property syntax can be used to specify *<srcprop>* as the source properties are, by definition, properties from the working set of nodes.
* The ``limit=`` parameter can be provided as input to the ``pivot()`` operator itself when using Operator syntax. Alternately the ``limit()`` operator_ can be used after the ``pivot()`` operator (in either Operator or Macro syntax) to specify a limit on the number of nodes returned.

join()
------
Returns the current (working) set of nodes **and** the set of nodes that share a specified property of the same type / valu as the original set of nodes.

``join()`` can be thought of as a ``pivot()`` that retains the original set of nodes and combines (joins) them with the set of nodes that are pivoted to.

Optional parameters:

* **Return limit**: specify the maximum number of nodes returned by the ``join()`` query.
  
  * ``limit=`` (operator syntax)

**Operator syntax:**

.. parsed-literal::
  
  **join(** *<dstprop>* **,** *<srcprop>* [ **,** limit=** *<num>* ] **)**

**Macro syntax:**

N/A

**Examples:**

* Given a set of domains (``inet:fqdn``) in the working set, return the domains and their set of immediate subdomains:
  ::
    join( inet:fqdn:domain, inet:fqdn )

* Given a set of email addresses (``inet:email``) in the working set, return the set of domain / registrant email (``inet:whois:regmail``) records associated with those email addresses:
  ::
    join( inet:whois:regmail:email, inet:email )

**Usage notes:**

* ``join()`` takes its arguments in the order *<dstprop>*, *<srcprop>*, which is the opposite of ``pivot()``, which takes its arguments as *<srcprop>*, *<dstprop>*.
  
  * **Note:** ``join()`` may be modified in a future release so its syntax matches that of ``pivot()``.
* Both *<dstprop>* and *<srcprop>* must be specified.
* ``join()`` does not consume nodes by design.
* The ``limit=`` parameter can be provided as input to the ``join()`` operator itself when using Operator syntax. Alternately the ``limit()`` operator_ can be used after the ``join()`` operator (in either Operator or Macro syntax) to specify a limit on the number of nodes returned.
* Because ``join()`` does not consume nodes, this impacts the results returned by either the ``limit=`` parameter or the ``limit()`` operator.
  
  * The ``limit=`` parameter will return **all** of the original nodes, **plus** the specified number of results (if ``limit=10`` and the number of working nodes was eight, this will return 18 nodes).
  * The ``limit()`` operator will return a **total** number of nodes equal to the specified limit, first including the original working nodes and then including resulting nodes (if ``limit=10`` and the number of working nodes was eight, this will return 10 nodes: the original eight, plus two results).
<commit_msg>Update ug015_storm_ref_pivot.rst<commit_after>.. highlight:: none

Storm Reference - Pivot Operators
=================================

Pivot operators operate on the output of a **previous Storm query.**

The operators below can be used to pivot from one set of nodes to another set of nodes within a Synapse hypergraph based on a set of specified criteria.

All of these operators are defined in storm.py_.

See the `Storm Syntax Conventions`__ for an explanation of the usage format used below.

Where specific query examples are given, they are commonly provided in pairs using operator syntax followed by the equivalent macro syntax (if available).

pivot()
-------
Returns a set of nodes that share a specified property of the same type / valu as the original set of nodes.

Optional parameters:

* **Return limit:** specify the maximum number of nodes returned by the ``pivot()`` query.

  * ``limit=`` (operator syntax)

**Operator syntax:**

.. parsed-literal::

  **pivot(** [ *<srcprop>* **,** ] *<dstprop>* [ **, limit=** *<limit>* ] **)**

**Macro syntax:**

.. parsed-literal::

  [ *<srcprop>* ] **->** *<dstprop>*

**Examples:**

* Pivot from a set of domains (``inet:fqdn`` nodes) in the working set to the DNS A records for those domains:
  ::
    pivot( inet:fqdn, inet:dns:a:fqdn )
    
    pivot( inet:dns:a:fqdn )
    
    inet:fqdn  ->  inet:dns:a:fqdn
    
    -> inet:dns:a:fqdn

* Pivot from a set of domains in the working set to the DNS A records for those domains, but limit the results to 10 nodes:
  ::
    pivot( inet:fqdn, inet:dns:a:fqdn, limit=10 )
    
    pivot( inet:dns:a:fqdn, limit=10 )

* Pivot from a set of domains in the working set to the DNS A records for those domains, and from the IP addresses in the DNS A records to the associated set of IPv4 nodes (e.g., pivot from a set of domains to the set of IP addresses the domains resolved to).
  ::
    pivot( inet:fqdn, inet:dns:a:fqdn ) pivot( inet:dns:a:ipv4, inet:ipv4 )
      
    pivot( inet:dns:a:fqdn ) pivot( :ipv4, inet:ipv4 )
    
    inet:fqdn -> inet:dns:a:fqdn inet:dns:a:ipv4 
      -> inet:ipv4
    
    -> inet:dns:a:fqdn :ipv4 -> inet:ipv4

* Pivot from a set of domains in the working set to the set of subdomains for those domains:
  ::
    pivot( inet:fqdn, inet:fqdn:domain )
    
    pivot( inet:fqdn:domain )
    
    inet:fqdn -> inet:fqdn:domain
    
    -> inet:fqdn:domain

* Pivot from a set of email addresses in the working set to the set of domains registered to those email addresses.
  ::
    pivot( inet:email, inet:whois:regmail:email )
      pivot( inet:whois:regmail:fqdn, inet:fqdn )
    
    pivot( inet:whois:regmail:email ) pivot( :fqdn, inet:fqdn )
    
    inet:email -> inet:whois:regmail:email 
      inet:whois:regmail:fqdn -> inet:fqdn
    
    -> inet:whois:regmail:email :fqdn -> inet:fqdn

* Pivot from a set of email addresses in the working set to the set of whois records associated with those email addresses.
  ::
    pivot( inet:email, inet:whois:contact:email )
      pivot( inet:whois:contact:rec, inet:whois:rec )
    
    pivot( inet:whois:contact:email ) pivot( :rec, inet:whois:rec )
    
    inet:email -> inet:whois:contact:email inet:whois:contact:rec
      -> inet:whois:rec
    
    -> inet:whois:contact:email inet:whois:contact:rec -> inet:whois:rec

**Usage notes:**

* If the source property for the pivot is the primary property of the working set of nodes, *<srcprop>* can be omitted from both Operator and Macro syntax.
* If the source property for the pivot is a secondary property of the working set of nodes, relative property syntax can be used to specify *<srcprop>* as the source properties are, by definition, properties from the working set of nodes.
* The ``limit=`` parameter can be provided as input to the ``pivot()`` operator itself when using Operator syntax. Alternately the ``limit()`` operator_ can be used after the ``pivot()`` operator (in either Operator or Macro syntax) to specify a limit on the number of nodes returned.

join()
------
Returns the current (working) set of nodes **and** the set of nodes that share a specified property of the same type / valu as the original set of nodes.

``join()`` can be thought of as a ``pivot()`` that retains the original set of nodes and combines (joins) them with the set of nodes that are pivoted to.

Optional parameters:

* **Return limit**: specify the maximum number of nodes returned by the ``join()`` query.
  
  * ``limit=`` (operator syntax)

**Operator syntax:**

.. parsed-literal::
  
  **join(** [ *<srcprop>* **,** ] *<dstprop>* [ **, limit=** *<limit>* ] **)**

**Macro syntax:**

.. parsed-literal::
  
  [ *<srcprop>* ] **<-** *<dstprop>*

**Examples:**

* Given a set of domains (``inet:fqdn``) in the working set, return the domains and their set of immediate subdomains:
  ::
    join( inet:fqdn, inet:fqdn:domain )
    
    join( inet:fqdn:domain )
    
    inet:fqdn <- inet:fqdn:domain
    
    <- inet:fqdn:domain

* Given a set of email addresses (``inet:email``) in the working set, return the email addresses and the set of domain / registrant email (``inet:whois:regmail``) records associated with those email addresses:
  ::
    join( inet:email, inet:whois:regmail:email )
    
    join( inet:whois:regmail:email )
    
    inet:email <- inet:whois:regmail:email
    
    <- inet:whois:regmail:email

**Usage notes:**

* ``join()`` does not consume nodes by design.
* If the source property for the join is the primary property of the working set of nodes, *<srcprop>* can be omitted from both Operator and Macro syntax.
* If the source property for the join is a secondary property of the working set of nodes, relative property syntax can be used to specify *<srcprop>* as the source properties are, by definition, properties from the working set of nodes.
* The ``limit=`` parameter can be provided as input to the ``join()`` operator itself when using Operator syntax. Alternately the ``limit()`` operator_ can be used after the ``join()`` operator (in either Operator or Macro syntax) to specify a limit on the number of nodes returned.
* Because ``join()`` does not consume nodes, this impacts the results returned by either the ``limit=`` parameter or the ``limit()`` operator.
  
  * The ``limit=`` parameter will return **all** of the original nodes, **plus** the specified number of results (if ``limit=10`` and the number of working nodes was eight, this will return 18 nodes).
  * The ``limit()`` operator will return a **total** number of nodes equal to the specified limit, first including the original working nodes and then including resulting nodes (if ``limit=10`` and the number of working nodes was eight, this will return 10 nodes: the original eight, plus two results).

{'fraction_non_alphanumeric': 0.11927444554530904, 'fraction_numerical': 0.0037344714579681426, 'mean_word_length': 3.8798809966530308, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2973492', 'n_tokens_mistral': 4155, 'n_tokens_neox': 4048, 'n_words': 1694}