<commit_before>= Using Shims =

This tutorial illustrates how to use the shims library in your application, as well as how to create your own shims correctly.

----
[[TOC(inline)]]
----

[[BR]]
== What are shims ==
----

Blah blah blah


[[BR]]
== Using shims in your application ==
----

Suppose you're writing an echo server. This server encrypts all traffic on a per-connection basis, and it may be behind a Network Address Translator (NAT). As the application creator, you only need to focus on what is relevant to the core functionalities of the echo server. Encryptions and NAT rerouting are handled by the underlying shim library.

Here is a sample echo server. You need to tell the shim library what shims you would like to use. In this case, you want to encrypt the traffic. Since we're not sure if the network configuration changes, we insert a !NatDeciderShim. It automatically connects to a forwarder if the presence of a NAT is detected.

{{{
#!python
include ShimStackInterface.repy

server_shim = ShimStackInterface('(RSAShim)(NatDeciderShim)')

def server_callback(rip, rport, sock, th, lh):
  while True:
    msg = sock.recv(1024)
    sock.send(msg)

if callfunc == 'initialize':
  serverip = getmyip()
  server_shim.waitforconn(serverip, 12345, server_callback)
}}}

In writing the client, you don't need to specify what shims to use. The shim library automatically detects the shim stack on the server, and it subsequently inserts the correct shims into the client's stack to communicate with the server. This is known shim stack balancing.

Here is a simple echo client:

{{{
#!python
include ShimStackInterface.repy

client_shim = ShimStackInterface()

if callfunc == 'initialize':
  serverip = callargs[0]
  serverport = callargs[1]
  sock = client_shim.openconn(serverip, serverport)
  sock.send('Hello World')
}}}


In both cases involving the server and the client, you only need to initialize the !ShimStackInterface once for every type of shim stack you use. For instance, you decide to do a waitforconn on multiple ports and IPs. All these listeners use the same shim stack structure. You only need to initialize the shim stack once, say, at the top of the module or in the constructor. All subsequent operations can act on this shim stack instance.

In case you're wondering about the implementation details, the !ShimStackInterface actually creates a new shim stack for each operation. This, of course, has been abstracted away from you. It is one of our objectives to incur minimal changes if we decide to introduce shims to an existing application.

The public interface of the !ShimStackInterface is identical to the network system calls. The following public methods are defined. Their usage can be found in RepyLibrary.

 * `ShimStackInterface(shim_stack_string="")`: Constructor.
 * `waitforconn`
 * `openconn`
 * `stopcomm`
 * `recvmess`
 * `sendmess`

When you obtain a reference to a handle or socket, you can manipulate it as if it is a normal system object. In specific, the following operations are defined on the socket-like object. Again, their usage can be found in RepyLibrary.

 * `socket.send`
 * `socket.recv`
 * `socket.close`

[[BR]]
== Creating your own shims ==
----

Your own shim must be a subclass of !BaseShim, the super class of all shims. You don't need to include any special files.

{{{
#!python
class MyShim(BaseShim):
}}}

In your constructor, you should call the super class's constructor before the constructor returns. 

{{{
#!python
  def __init__(self, next_shim=None, optional_args=None):
    # I'm doing something here
    BaseShim.__init__(self, next_shim, optional_args)
    # I'm doing something else here
}}}

Always override the `copy()` and `get_advertisement_string()` method. The first defines how your shim can be duplicated. You need to specify how internal states, if any, are copied to the new instance of your shim. Here is an example.

{{{
#!python
  def copy(self):
    mycopy = MyShim()
    # If there are no internal states to copy, skip the next line
    mycopy._internalstate = self._internalstate
    return mycopy
}}}

The `get_advertisement_string()` method defines how your shim is advertised (i.e. published). The shim stack library publishes the contents of a server shim stack. The client subscribes to the contents, so it knows what shims to have in order to communicate with the server. When you publish your shim, always enclose the name of the shim within a pair of parentheses. If you have any additional arguments to publish, append them to the shim name, separated by commas. These arguments, when subscribed by the client, will become the optional_args in the constructor of the client shim. In the following example, we are publishing the shim along with some shared state.

{{{
#!python
  def get_advertisement_string(self):
    return '(MyShim,' + self._shim_args_str + ')' + \
            self.shim_stack.get_advertisement_string()
}}}

This call is recursive. Remember to append `self.shim_stack.get_advertisement_string()` at the end.

If, on the other hand, your shim is private, it should not be advertised or published. Then you simply return the advertisement string of the rest of the stack, as illustrated by the alternative version:

{{{
#!python
  def get_advertisement_string(self):
    return self.shim_stack.get_advertisement_string()
}}}

You may choose to override any of the following methods, if you decide to redefine the corresponding network operation. Otherwise, you do not need to include them in your shim. The super class will handle the appropriate calls.

{{{
#!python
  def _shim_waitforconn(self, host, port, callback):
    return self.shim_stack.waitforconn(host, port, callback)

  def _shim_openconn(self, host, port, localhost=None, localport=None, timeout=5):
    return self.shim_stack.openconn(host, port, localhost, localport, timeout)

  def _shim_recvmess(self, host, port, callback):
    return self.shim_stack.recvmess(host, port, callback)

  def _shim_sendmess(self, host, port, msg, localhost=None, localport=None):
    return self.shim_stack.sendmess(host, port, msg, localhost, localport)

  def _shim_stopcomm(self, handle):
    return self.shim_stack.stopcomm(handle)

  def _shim_socket_close(self, socket):
    return self.shim_stack.socket_close(socket)

  def _shim_socket_send(self, socket, msg):
    return self.shim_stack.socket_send(socket, msg)

  def _shim_socket_recv(self, socket, bytes): 
    return self.shim_stack.socket_recv(socket, bytes)
}}}

Let us put everything together so far, and write a simple encryption shim. It encrypts all TCP traffic.

{{{
#!python
class SimpleEncryptionShim(BaseShim):

  def __init__(self, next_shim=None, optional_args=None):
    BaseShim.__init__(self, next_shim, optional_args)
    self._encryption_key = int(optional_args[0])

  def copy(self):
    newshim = SimpleEncryptionShim(optional_args=[self._encryption_key])
    return newshim

  def get_advertisement_string(self):
    return '(SimpleEncryptionShim,' + str(self._encryption_key) + ')' + \
        self.shim_stack.get_advertisement_string()

  def _encrypt(self, clear_text):
    # do some magic
    return cypher

  def _decrypt(self, cypher):
    # do more magic
    return clear_text

  def _shim_socket_send(self, socket, msg):
    return self.shim_stack.socket_send(socket, self._encrypt(msg))

  def _shim_socket_recv(self, socket, bytes): 
    cypher = self.shim_stack.socket_recv(socket, bytes)
    return self._decrypt(cypher)
}}}

Finally, you need to register the shim with the shim library. Open `shimstackInterface.repy`. Include your shim at the top, and make sure you register it.

{{{
#!python
include MyShim.repy
register_shim('MyShim', MyShim)
}}}

And now, the application can use your shim!<commit_msg>Revision 15 of Outdated/UsingShims<commit_after>= Using Shims =

This tutorial illustrates how to use the shims library in your application, as well as how to create your own shims correctly.

----
[[TOC(inline)]]
----

[[BR]]
== What are shims ==
----

Blah blah blah


[[BR]]
== Using shims in your application ==
----

Suppose you're writing an echo server. This server encrypts all traffic on a per-connection basis, and it may be behind a Network Address Translator (NAT). As the application creator, you only need to focus on what is relevant to the core functionalities of the echo server. Encryptions and NAT rerouting are handled by the underlying shim library.

Here is a sample echo server. You need to tell the shim library what shims you would like to use. In this case, you want to encrypt the traffic. Since we're not sure if the network configuration changes, we insert a !NatDeciderShim. It automatically connects to a forwarder if the presence of a NAT is detected.

{{{
#!python
include ShimStackInterface.repy

server_shim = ShimStackInterface('(RSAShim)(NatDeciderShim)')

def server_callback(rip, rport, sock, th, lh):
  while True:
    msg = sock.recv(1024)
    sock.send(msg)

if callfunc == 'initialize':
  serverip = getmyip()
  server_shim.waitforconn(serverip, 12345, server_callback)
}}}

In writing the client, you don't need to specify what shims to use. The shim library automatically detects the shim stack on the server, and it subsequently inserts the correct shims into the client's stack to communicate with the server. This is known shim stack balancing.

Here is a simple echo client:

{{{
#!python
include ShimStackInterface.repy

client_shim = ShimStackInterface()

if callfunc == 'initialize':
  serverip = callargs[0]
  serverport = callargs[1]
  sock = client_shim.openconn(serverip, serverport)
  sock.send('Hello World')
}}}


In both cases involving the server and the client, you only need to initialize the !ShimStackInterface once for every type of shim stack you use. For instance, you decide to do a waitforconn on multiple ports and IPs. All these listeners use the same shim stack structure. You only need to initialize the shim stack once, say, at the top of the module or in the constructor. All subsequent operations can act on this shim stack instance.

In case you're wondering about the implementation details, the !ShimStackInterface actually creates a new shim stack for each operation. This, of course, has been abstracted away from you. It is one of our objectives to incur minimal changes if we decide to introduce shims to an existing application.

The public interface of the !ShimStackInterface is identical to the network system calls. The following public methods are defined. Their usage can be found in RepyLibrary.

 * `ShimStackInterface(shim_stack_string="")`: Constructor.
 * `waitforconn`
 * `openconn`
 * `stopcomm`
 * `recvmess`
 * `sendmess`

When you obtain a reference to a handle or socket, you can manipulate it as if it is a normal system object. In specific, the following operations are defined on the socket-like object. Again, their usage can be found in RepyLibrary.

 * `socket.send`
 * `socket.recv`
 * `socket.close`

[[BR]]
== Creating your own shims ==
----

Your own shim must be a subclass of !BaseShim, the super class of all shims. You don't need to include any special files.

{{{
#!python
class MyShim(BaseShim):
}}}

In your constructor, you should call the super class's constructor before the constructor returns. 

{{{
#!python
  def __init__(self, next_shim=None, optional_args=None):
    # I'm doing something here
    BaseShim.__init__(self, next_shim, optional_args)
    # I'm doing something else here
}}}

Always override the `copy()` and `get_advertisement_string()` method. The first defines how your shim can be duplicated. You need to specify how internal states, if any, are copied to the new instance of your shim. Here is an example.

{{{
#!python
  def copy(self):
    mycopy = MyShim()
    # If there are no internal states to copy, skip the next line
    mycopy._internalstate = self._internalstate
    return mycopy
}}}

The `get_advertisement_string()` method defines how your shim is advertised (i.e. published). The shim stack library publishes the contents of a server shim stack. The client subscribes to the contents, so it knows what shims to have in order to communicate with the server. When you publish your shim, always enclose the name of the shim within a pair of parentheses. If you have any additional arguments to publish, append them to the shim name, separated by commas. These arguments, when subscribed by the client, will become the optional_args in the constructor of the client shim. In the following example, we are publishing the shim along with some shared state.

{{{
#!python
  def get_advertisement_string(self):
    return '(MyShim,' + self._shim_args_str + ')' + \
            self.shim_stack.get_advertisement_string()
}}}

This call is recursive. Remember to append `self.shim_stack.get_advertisement_string()` at the end.

If, on the other hand, your shim is private, it should not be advertised or published. Then you simply return the advertisement string of the rest of the stack, as illustrated by the alternative version:

{{{
#!python
  def get_advertisement_string(self):
    return self.shim_stack.get_advertisement_string()
}}}

You may choose to override any of the following methods, if you decide to redefine the corresponding network operation. Otherwise, you do not need to include them in your shim. The super class will handle the appropriate calls.

{{{
#!python
  def _shim_waitforconn(self, host, port, callback):
    return self.shim_stack.waitforconn(host, port, callback)

  def _shim_openconn(self, host, port, localhost=None, localport=None, timeout=5):
    return self.shim_stack.openconn(host, port, localhost, localport, timeout)

  def _shim_recvmess(self, host, port, callback):
    return self.shim_stack.recvmess(host, port, callback)

  def _shim_sendmess(self, host, port, msg, localhost=None, localport=None):
    return self.shim_stack.sendmess(host, port, msg, localhost, localport)

  def _shim_stopcomm(self, handle):
    return self.shim_stack.stopcomm(handle)

  def _shim_socket_close(self, socket):
    return self.shim_stack.socket_close(socket)

  def _shim_socket_send(self, socket, msg):
    return self.shim_stack.socket_send(socket, msg)

  def _shim_socket_recv(self, socket, bytes): 
    return self.shim_stack.socket_recv(socket, bytes)
}}}

Let us put everything together so far, and write a simple encryption shim. It encrypts all TCP traffic.

{{{
#!python
class SimpleEncryptionShim(BaseShim):

  def __init__(self, next_shim=None, optional_args=None):
    BaseShim.__init__(self, next_shim, optional_args)
    self._encryption_key = int(optional_args[0])

  def copy(self):
    newshim = SimpleEncryptionShim(optional_args=[self._encryption_key])
    return newshim

  def get_advertisement_string(self):
    return '(SimpleEncryptionShim,' + str(self._encryption_key) + ')' + \
        self.shim_stack.get_advertisement_string()

  def _shim_socket_send(self, socket, msg):
    cypher = encrypt(msg, self._encryption_key)
    return self.shim_stack.socket_send(socket, cypher)

  def _shim_socket_recv(self, socket, bytes): 
    cypher = self.shim_stack.socket_recv(socket, bytes)
    return decrypt(cypher, self._encryption_key)
}}}

Finally, you need to register the shim with the shim library. Open `shimstackInterface.repy`. Include your shim at the top, and make sure you register it.

{{{
#!python
include MyShim.repy
register_shim('MyShim', MyShim)
}}}

And now, the application can use your shim!
{'fraction_non_alphanumeric': 0.07919004320629393, 'fraction_numerical': 0.001805636164312891, 'mean_word_length': 4.522792022792022, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2938345', 'n_tokens_mistral': 4642, 'n_tokens_neox': 4425, 'n_words': 1898}