<commit_before>---
layout: page
title: Signal Processing
redirect_from: "/2014/02/08/innards.html"
index: 3
---

{% include image.html path="/assets/signal_flow.png" caption="Overview of signal flow." %}

## Data Acquisition
***

There are two methods to get data from an rtl-sdr dongle, directly with librtlsdr and via tcp with the `rtl_tcp` spectrum server. Using librtlsdr requires the use of cgo which prevents cross-compilation; `rtl_tcp` is used instead. This has the added benefit of allowing the receiver to be somewhere other than the system running `rtlamr`.

## Demodulation
***

The ERT protocol is an on-off keyed manchester-coded signal transmitted at bit-rate of 32.768kbps. On-off keying is a type of amplitude shift keying. Individual symbols are represented as either a carrier of fixed amplitude or no transmission at all.

{% include image.html path="/assets/magnitude.png" caption="<strong>Top:</strong> Real component of complex signal. <strong>Bottom:</strong> Magnitude of complex signal. <strong>Note:</strong> Signal is truncated to show detail." %}

The signal is made up of interleaved in-phase and quadrature samples, 8-bits per component. The amplitude of each sample is:

<p>
\$\$\vert z\vert = \sqrt{\Re(z)^2 + \Im(z)^2}\$\$
</p>

To meet performance requirements the magnitude computation has two implementations. The first uses a pre-computed lookup table which maps all possible 8-bit values to their floating-point squares. Calculating the magnitude using the lookup table then only involves two lookups, one addition and one square-root.

The second implementation is an approximation known as Alpha-Max plus Beta-Min whose efficiency comes from eliminating the square root operation:

<p>
\$\$
	\begin{aligned}
		&\alpha = \frac{2\cos\frac{\pi}{8}}{1+cos\frac{\pi}{8}} \qquad \beta = \frac{2\sin\frac{\pi}{8}}{1+cos\frac{\pi}{8}} \\ \\
		\vert z\vert \approx \,&\alpha\cdot\max(\Re(z),\Im(z)) + \beta\cdot\min(\Re(z),\Im(z))
	\end{aligned}
\$\$
</p>

## Filtering
***

Filtering is required for later bit decision. The ideal filter kernel for a square-wave signal is known as a boxcar and is essentially a moving average. Due to the signal being manchester coded the ideal filter is a step function. Manchester coding mixes data with a clock signal to allow for synchronization and clock recovery while decoding as well as reducing any DC offset that would be present due to content of the data. The symbol length [[N]] is determined by the sampling rate [[F_s]] of the receiver and the data rate [[R]] of the signal.

<p>
\$\$N = \frac{F_s}{R}\$\$
</p>

The maximum sample rate without any sample loss is between 2.4 and 2.56 MHz. To simplify decoding we determine the sample rate from integral symbol lengths. From librtlsdr, the sample rate must satisfy one of two conditions:

<p>
\$\$
	\begin{aligned}
		225\text{kHz} \lt \, &F_s \le 300\text{kHz} \\
		900\text{kHz} \lt \, &F_s \le 3.2\text{MHz} \\
	\end{aligned}
\$\$
</p>

From this we can determine all of the valid symbol lengths and their corresponding sample rates. More info [here](https://github.com/bemasher/rtlamr/blob/master/help.md). Filtering can be implemented efficiently by computing the cumulative sum of each sample block and calculating the difference between a pair of subtractions:

<p>
\$\$
	\begin{aligned}
		\mathbf{M}_i &= \sum_{j=i}^{i+N} \mathbf{S}_j - \mathbf{S}_{j+N} \\ \\
		&= (\mathbf{C}_{i+N} - \mathbf{C}_i) - (\mathbf{C}_{i+2N} - \mathbf{C}_{i+N}) \\ \\
		&= 2\mathbf{C}_{i+N}-\mathbf{C}_{i+2 N}-\mathbf{C}_i
	\end{aligned}
\$\$
</p>

Where [[\mathbf{M}]] is the filtered signal, [[\mathbf{S}]] is the sample vector, [[N]] is the symbol length and [[\mathbf{C}]] is the cumulative or prefix sum of the signal.

{% include image.html path="/assets/filter.png" caption="<strong>Top:</strong> Ideal filter kernel for Manchester coded signals. <strong>Bottom:</strong> Filtered signal. <strong>Note:</strong> Signal is truncated to show detail." %}

## Bit Decision
***

The bit decision in this particular case is extremely simple. The symmetry of the filter kernel produces a signal with no DC offset, positive peaks represent a falling edge and negative peaks a rising edge. This applies at all points in the filtered signal so the bit value is only dependent on the sign of each sample.

{% include image.html path="/assets/quantized.png" caption="<strong>Red:</strong> Quantized filter signal. <strong>Blue:</strong> Filtered signal. <strong>Note:</strong> Signal is truncated to show detail." %}

## Preamble Search
***

Now that the signal has been quantized we need to search for the offset in the signal of the preamble, if there is one. The naive way to do this is simply to start at index zero and check it's value against the preamble, then if it matches look ahead one full symbol length and so on. The problem with using this method is that it has poor memory locality. For performance we interleave the signal so it can be searched linearly in memory.

## Decode
***

If a preamble is found the remaining bits following the preamble are passed to the appropriate decoder which performs a CRC to determine if the packet is valid.<commit_msg>Convert <p>'s to <span>'s...<commit_after>---
layout: page
title: Signal Processing
redirect_from: "/2014/02/08/innards.html"
index: 3
---

{% include image.html path="/assets/signal_flow.png" caption="Overview of signal flow." %}

## Data Acquisition
***

There are two methods to get data from an rtl-sdr dongle, directly with librtlsdr and via tcp with the `rtl_tcp` spectrum server. Using librtlsdr requires the use of cgo which prevents cross-compilation; `rtl_tcp` is used instead. This has the added benefit of allowing the receiver to be somewhere other than the system running `rtlamr`.

## Demodulation
***

The ERT protocol is an on-off keyed manchester-coded signal transmitted at bit-rate of 32.768kbps. On-off keying is a type of amplitude shift keying. Individual symbols are represented as either a carrier of fixed amplitude or no transmission at all.

{% include image.html path="/assets/magnitude.png" caption="<strong>Top:</strong> Real component of complex signal. <strong>Bottom:</strong> Magnitude of complex signal. <strong>Note:</strong> Signal is truncated to show detail." %}

The signal is made up of interleaved in-phase and quadrature samples, 8-bits per component. The amplitude of each sample is:

<span>
\$\$\vert z\vert = \sqrt{\Re(z)^2 + \Im(z)^2}\$\$
</span>

To meet performance requirements the magnitude computation has two implementations. The first uses a pre-computed lookup table which maps all possible 8-bit values to their floating-point squares. Calculating the magnitude using the lookup table then only involves two lookups, one addition and one square-root.

The second implementation is an approximation known as Alpha-Max plus Beta-Min whose efficiency comes from eliminating the square root operation:

<span>
\$\$
	\begin{aligned}
		&\alpha = \frac{2\cos\frac{\pi}{8}}{1+cos\frac{\pi}{8}} \qquad \beta = \frac{2\sin\frac{\pi}{8}}{1+cos\frac{\pi}{8}} \\ \\
		\vert z\vert \approx \,&\alpha\cdot\max(\Re(z),\Im(z)) + \beta\cdot\min(\Re(z),\Im(z))
	\end{aligned}
\$\$
</span>

## Filtering
***

Filtering is required for later bit decision. The ideal filter kernel for a square-wave signal is known as a boxcar and is essentially a moving average. Due to the signal being manchester coded the ideal filter is a step function. Manchester coding mixes data with a clock signal to allow for synchronization and clock recovery while decoding as well as reducing any DC offset that would be present due to content of the data. The symbol length [[N]] is determined by the sampling rate [[F_s]] of the receiver and the data rate [[R]] of the signal.

<span>
\$\$N = \frac{F_s}{R}\$\$
</span>

The maximum sample rate without any sample loss is between 2.4 and 2.56 MHz. To simplify decoding we determine the sample rate from integral symbol lengths. From librtlsdr, the sample rate must satisfy one of two conditions:

<span>
\$\$
	\begin{aligned}
		225\text{kHz} \lt \, &F_s \le 300\text{kHz} \\
		900\text{kHz} \lt \, &F_s \le 3.2\text{MHz} \\
	\end{aligned}
\$\$
</span>

From this we can determine all of the valid symbol lengths and their corresponding sample rates. More info [here](https://github.com/bemasher/rtlamr/blob/master/help.md). Filtering can be implemented efficiently by computing the cumulative sum of each sample block and calculating the difference between a pair of subtractions:

<span>
\$\$
	\begin{aligned}
		\mathbf{M}_i &= \sum_{j=i}^{i+N} \mathbf{S}_j - \mathbf{S}_{j+N} \\ \\
		&= (\mathbf{C}_{i+N} - \mathbf{C}_i) - (\mathbf{C}_{i+2N} - \mathbf{C}_{i+N}) \\ \\
		&= 2\mathbf{C}_{i+N}-\mathbf{C}_{i+2 N}-\mathbf{C}_i
	\end{aligned}
\$\$
</span>

Where [[\mathbf{M}]] is the filtered signal, [[\mathbf{S}]] is the sample vector, [[N]] is the symbol length and [[\mathbf{C}]] is the cumulative or prefix sum of the signal.

{% include image.html path="/assets/filter.png" caption="<strong>Top:</strong> Ideal filter kernel for Manchester coded signals. <strong>Bottom:</strong> Filtered signal. <strong>Note:</strong> Signal is truncated to show detail." %}

## Bit Decision
***

The bit decision in this particular case is extremely simple. The symmetry of the filter kernel produces a signal with no DC offset, positive peaks represent a falling edge and negative peaks a rising edge. This applies at all points in the filtered signal so the bit value is only dependent on the sign of each sample.

{% include image.html path="/assets/quantized.png" caption="<strong>Red:</strong> Quantized filter signal. <strong>Blue:</strong> Filtered signal. <strong>Note:</strong> Signal is truncated to show detail." %}

## Preamble Search
***

Now that the signal has been quantized we need to search for the offset in the signal of the preamble, if there is one. The naive way to do this is simply to start at index zero and check it's value against the preamble, then if it matches look ahead one full symbol length and so on. The problem with using this method is that it has poor memory locality. For performance we interleave the signal so it can be searched linearly in memory.

## Decode
***

If a preamble is found the remaining bits following the preamble are passed to the appropriate decoder which performs a CRC to determine if the packet is valid.
{'fraction_non_alphanumeric': 0.11196540124939933, 'fraction_numerical': 0.008649687650168188, 'mean_word_length': 4.210816224336504, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2075540', 'n_tokens_mistral': 3090, 'n_tokens_neox': 2915, 'n_words': 1372}