<commit_before>\documentclass[10pt]{article}
\usepackage{url}
\usepackage{a4}

\pagestyle{empty}

\setlength{\parskip}{4pt}

\author{<NAME> -- Tjaldur Software Governance Solutions}
\title{Binary Analysis Tool User and Developer Manual}

\begin{document}

\maketitle
\thispagestyle{empty}

\section{Introducing the Binary Analysis Tool}

The Binary Analysis Tool (BAT) is a framework that can help developers and
companies check binary files. Its primary application is for Open Source
software license compliance, with a special focus on supply chain management
in consumer electronics.

BAT consists of several scripts written in Python. The most important program
is the scanner for binary objects to detect the presence of Open Source
software inside binaries. There are also other programs to help with specific
tasks, such as verifying if configurations for a given BusyBox binary match
with the configuration in source code. Also included is a very experimental
program to help scan Linux kernel images and a script to derive a Linux kernel
configuration from a Linux kernel image.

\section{Installing the Binary Analysis Tool}

\subsection{Hardware requirements}

The tools in the Binary Analysis Tool can be quite resource intensive. Most of
them seem to be memorybound and I/O-bound, so it makes more sense to invest
in more memory and faster disks than in raw CPU power. Using more cores is
highly recommended, since most of the programs in the Binary Analysis Tool will
benefit from this.

\subsection{Software requirements}

To run BAT a recent Linux distribution is needed. Development is (currently)
done on Fedora 14, 15 and 16, so those platforms are likely to work best.
Other platforms (Debian 6, Ubuntu 10.10 and later) are tested with, but only to
build binary releases.

If the latest version from version control is used it is important to look at
the file \texttt{setup.cfg} to get a list of the dependencies that should be
met.

\subsubsection{Installing on Fedora}

To install on Fedora three packages are needed:
\texttt{bat-extratools}, \texttt{bat-extratools-java} and \texttt{bat}. These
can be downloaded from the website of the binary analysis tool in both prebuilt
versions and as a source RPM file. When installing the three files there should
be a list of dependencies that should be installed to let BAT work
successfully. Some of these packages are not in Fedora by default but need to
be installed through extra repositories, such as RPMfusion.

\subsubsection{Installing on Debian and Ubuntu}

To install on Debian and Ubuntu three packages are needed:
\texttt{bat-extratools}, \texttt{bat-extratools-java} and \texttt{bat}. These
can be downloaded from the website of the binary analysis tool in both prebuilt
versions. When installing the three files there should be a list of
dependencies that should be installed to let BAT work successfully. Some of
these packages are not in Debian by default but need to be installed by
enabling extra repositories such as Debian \texttt{non-free}.

The graphical interface might not work on Debian 6, because of a too old
version of \texttt{wxPython}.

\section{Analysing binaries with the Binary Analysis Tool}

BAT consists of several programs and a few helper scripts (not meant to be
used directly). The main purpose of the Binary Analysis tool is to analyse
arbitrary binaries and review results. There are two ways to do this: via the
commandline, or via a graphical user interface.

The commandline interface functionality can be found in \texttt{bruteforce.py}.
The graphical user interface is provided by \texttt{batgui} (although the
graphical user interface can do more than just scanning). Both programs are a
front end for the same functionality.

\subsection{Inner workings of the analysis process}

BAT uses a brute force approach for analysing a binary. It assumes no prior
knowledge of how a binary is constructed or what is inside the binary. Instead
it tries to determine what is inside by applying a wide range of methods, such
as looking for known identifiers of file systems and compressed files and
running external tools to find contents in the binary.

After successful identification it unpacks compressed files, media files and
file systems recursively. Finally several individual scans are applied to each
extracted file, ranging from determining dynamically linked libraries to an
elaborate ranking method to guess which sources were used to build a program.

In this section a short overview of the configuration and inner workings are
given. A more detailed description, including how to write scans, can be found
in the appendix.

\subsubsection{Analysis configuration}

The analysis process is highly configurable: methods can be simply enabled and
disabled, based on need: some methods can run for quite a long time, which
might be undesirable at times. Configuration is done via a simple configuration
file in Windows INI format.

Most sections are specific to scanning methods, except two sections.

The main configuration directive is \texttt{batconfig}. In this
section various global settings are defined, such as \texttt{multiprocessing}
which sets whether or not multiple CPUs (or cores) should be used during
scanning. The default configuration as shipped in the official BAT distribution
is to use a single CPU:

\begin{verbatim}
[batconfig]
multiprocessing = no
\end{verbatim}

By setting this setting to \texttt{yes} the program will start an extra process
per CPU that is available. In most cases it is completely safe to use
multiprocessing, but there is one important exception: if the ranking method
(described later) is used, without pregenerating the proper caching database
(or purchasing one) it is unsafe to use multiprocessing. If ranking is used and
there is a fully generated database it is completely safe and even strongly
advised to use multiprocessing, since it vastly speeds up running the program.

The other global section is \texttt{viewer}. This section is specific for the
graphical frontend and is currently not used.

\subsubsection{Enabling and disabling scans and methods}

The standard configuration file enables most of the scans and methods
implemented in BAT by default. Scans can be enabled and disabled by setting the
option \texttt{enabled} to \texttt{yes} and \texttt{no} respectively. Disabling
a scan will still allow you to enable it later in the graphical user interface.

To prevent a scan from being enabled in the graphical user interface completely
outcomment the entry in the configuration file (by starting the line with the
\texttt{\#} character), or remove the section from the configuration file.

\subsubsection{Marker search}

The first action performed is scanning a file for known markers of compressed
files and file systems. Which markers are searched for depends on the contents
of the configuration file (specifically the \texttt{magic} attribute). These
markers determine other methods, such as unpackers, are run.

The marker search cannot be enabled or disabled via the configuration file.

\subsubsection{Pre-run checks}

Before files are unpacked they are briefly inspected and if possible tagged.
Tags are used to pass hints to methods that are run later to reduce execution
time and false positives.

For example, files that only contain text are tagged as \texttt{text}, other
files are tagged as \texttt{binary} (this depends on the implementation of
Python. Python 2 only considers (by default) ASCII to be valid text). Methods
that only work on binaries can ignore anything that has been tagged with
\texttt{text}.

Other checks that are available are for valid XML, various Android formats,
ELF executables and libraries, graphics files, and so on.

\subsubsection{Unpackers}

Unpackers can be recognized in the configuration because their type is set
to \texttt{unpack}, for example:

\begin{verbatim}
[jffs2]
type        = unpack
module      = bat.fwunpack
method      = searchUnpackJffs2
priority    = 1
magic       = jffs2_le
noscan      = text:xml:graphics:pdf:compressed:audio:video
description = Unpack JFFS2 file systems
enabled     = yes
\end{verbatim}

In BAT 8.0 the following file systems, compressed files and media files can be
unpacked or extracted:

\begin{itemize}
\item file systems: cramfs, ext2/ext3/ext4, ISO9660, JFFS2, SquashFS (several
variants), romfs, YAFFS2 (specific variants)
\item compressed files and executable formats: 7z, ar, ARJ, BASE64, BZIP2,
compressed Flash, CAB, compress, CPIO, EXE (specific compression methods only),
GZIP, InstallShield (old versions), LRZIP, LZIP, LZMA, LZO, RAR, RPM, serialized
Java, TAR, UPX, XZ, ZIP
\item media files: GIF, ICO, PDF, PNG
\end{itemize}

Unpacking differs per file type. Most files use one or more identifiers that
can be searched for in a binary blob. Using this information it is possible
to carve out the right parts of a binary blob and verify if it indeed contains
a compressed file, media file or file system.

There is not always an identifier that can be searched for. The YAFFS2 file
system layout for example is dependent on the hardware specifics of the
underlying flash chip. Without knowing these specifics it is not possible to
unpack YAFFS2 file systems.

Other files (such as ARJ and ICO files) have a very generic identifier, so
there are a lot of false positives. This causes a big increase in runtime.
These two unpackers are therefore disabled by default.

If unpacking is successful a directory with unpacked files that can be
scanned further is returned, and, if available, some meta information to avoid
duplicate scanning (blacklisting information and tags).

\subsubsection{Leaf scans}

Leaf scans, also called program scans, are scans that are run on every single
file after unpacking, including files that contained files that were found
and extracted by unpackers.

Leaf/program scans can be recognized in the configuration because their type is
set to \texttt{program}, for example:

\begin{verbatim}
[iptables]
type        = program
module      = bat.checks
method      = searchIptables
noscan      = text:xml:graphics:pdf:compressed:audio:video
description = Determine presence of iptables
enabled     = yes
\end{verbatim}

The current program scans that are available in BAT are:

\begin{itemize}
\item fast string searches: dproxy, ez-ipupdate, iptables, iproute, libusb,
Linux kernel, loadlin, RedBoot, U-Boot, vsftpd, wireless-tools, wpa-supplicant
\item advanced search mode using ranking of strings and function names, using a
database
\item BusyBox version number
\item dynamic library dependencies (ELF files only)
\item Linux kernel module license (Linux kernel modules only)
\item Linux kernel version number, plus detection for several subsystems
\item PDF meta data extraction
\item architecture
\item presence of URLs indicating an open source license
\item presence of URLs indicating forges/collaborative software development
sites (SourceForge, GitHub, etcetera)
\end{itemize}

The fast string searches are meant for quick sweep scanning only. They have
their limits, can report false positives or miss finding a program. They should
only be used to signal that further inspection is necessary. For a thorough
investigation the advanced search mode should be used.

\subsubsection{Post-run methods}

Since BAT 6.0 there is the possibility to add methods that are run after all
the regular work has been performed, or ``post-run''. These methods should not
alter the scan results in any way, but just use the information from the
scanning process. A typical use case would be to present the data in a nicer
to use format than the standard report, to use more external data sources or
generate graphical representations of data.

The post-run methods have the type \texttt{postrun} in the configuration.

\begin{verbatim}
[hexdump]
type        = postrun
module      = bat.generatehexdump
method      = generateHexdump
noscan      = text:xml:graphics:pdf:audio:video
envvars     = BAT_REPORTDIR=/tmp/images:BAT_IMAGE_MAXFILESIZE=100000000
description = Create hexdump output of files
enabled     = no
\end{verbatim}


\subsubsection{Interpreting the results}

There are two formats in which \texttt{bruteforce.py} outputs its results:

\begin{enumerate}
\item dump file containing program state, complete unpacked directory tree
containing all unpacked data, plus possibly some extra generated data, such
as pictures and more reporting. These dumps are meant to be used by
\texttt{batgui}.
\item XML file (configurable)
\end{enumerate}

If configured \texttt{bruteforce.py} outputs its results in XML format on
standard output. After redirecting the output to a file it is possible to look
at this file with a commandline tool such as \texttt{xml\_pp} or a webbrowser
such as Mozilla Firefox.

This XML file is not meant for human consumption, but for use by for example
reporting tools.

The XML file starts with metadata, such as:

\begin{itemize}
\item date, plus time of the scan (local time of the computer, in UTC)
\item name of the file
\item SHA256 cryptographic checksum of the file, uniquely identifying it
\item size of the file
\item filetype as determined by \texttt{file} on a Linux system
\item relative path inside the unpacked system, plus the absolute path inside
the file system, which is useful for later analysis
\end{itemize}

If any of the scans were successful the results of the these scans can
be found in the element \texttt{scans}.

For each successful unpack action the following attributes are reported:

\begin{itemize}
\item name of the scan (corresponding to the name of the scan in the
configuration file)
\item offset in the parent file of the compressed file, file system or media
file
\end{itemize}

\subsubsection{Running \texttt{bruteforce.py}}

Invoking \texttt{bruteforce.py} is relatively simple. You will need to supply
three parameters:

\begin{enumerate}
\item \texttt{-c} : path to a configuration file
\item \texttt{-b} : path to the binary to be scanned
\item \texttt{-o} : path to an output file, where unpacked files, reports,
plus the final program state be written to
\end{enumerate}

A typical invocation looks like this:

\texttt{python bruteforce.py -c /path/to/configuration -b /path/to/binary -o
/path/to/outputfile}

XML output, if enabled in the configuration file, will be written to standard
output.

\subsection{\texttt{batgui}}

The \texttt{batgui} program is a graphical viewer for the results of
\texttt{bruteforce.py} as well as a front end for invoking the scanner itself.

\section{Additional programs in the Binary Analysis Tool}

\subsection{\texttt{busybox.py} and \texttt{busybox-compare-configs.py}}

Two other tools in BAT are \texttt{busybox-compare-configs.py} and
\texttt{busybox.py} (in the subdirectory \texttt{bat}). These two tools are
specifically used to analyze BusyBox binaries. BusyBox is in widespread use on
embedded devices and the license on BusyBox is actively enforced in court.

BusyBox binaries on embedded machines often have different configurations,
depending on the needs of the manufacturer. Since providing the correct
configuration is one of the requirements for license compliance it is important
to be able to determine the configuration of a BusyBox binary and verify that
there is a corresponding configuration file in the source code release.

The BusyBox processing tools in BAT try to extract the most likely
configuration from the binary and print it in the right format (which
depends on the BusyBox version).

\texttt{busybox.py} is used to extract the configuration from a binary.
Afterwards \texttt{busybox-compare-configs.py} can be used to compare the
extracted configuration with a vendor supplied configuration.

\subsubsection{Extracting a configuration from BusyBox}

Extracting a configuration from a BusyBox executable is done using
\texttt{busybox.py} which can be found in the \texttt{bat} directory. It needs
two commandline parameters: the path to the binary and the path to a directory
containing a directory \texttt{configs} which has files containing mappings
from BusyBox applet names to BusyBox configuration directives. By default this
value is hardcoded as \texttt{/etc/bat}, but this might change in the future.

\begin{verbatim}
python bat/busybox.py -b /path/to/busybox/binary -c
     /path/to/pre/extracted/configs > /path/to/saved/config
\end{verbatim}

This command will save the configuration to a file, which can be used as an
input to \texttt{busybox-compare-configs.py}.

\subsubsection{Comparing two BusyBox configurations}

After extracting the configuration the extracted configuration can be compared
to another configuration, for example a configuration as supplied by a vendor:

\begin{verbatim}
python busybox-compare-configs.py -e /path/to/saved/config
    -f /path/to/vendor/configuration -n $version
\end{verbatim}

\subsection{\texttt{extractkernelstrings.py}, \texttt{extractkernelconfig.py}
\\and \texttt{findkernelstrings.py}}

The three programs \texttt{extractkernelstrings.py},
\texttt{extractkernelconfig.py} and \texttt{findkernelstrings.py} are very
experimental tools to derive a configuration from a Linux kernel image. They
are far from mature and are not yet ready for production use.

Once they are ready for production use a description for the scripts will be
included in this manual. A description of the technical uses that need to be
solved, plus a motivation on why and how to solve them can be found in the
appendix.

%\subsubsection{\texttt{extractkernelstrings.py}}
%\subsubsection{\texttt{extractkernelconfig.py}}
%\subsubsection{\texttt{findkernelstrings.py}}


\section{Binary Analysis Tool extratools collection}

To help with unpacking non-standard file systems, or standard file systems for
which there are no tools readily available on Fedora or Ubuntu there is also
a collection of tools that can be used by BAT to unpack more file systems.
These tools are not part of the standard distribution, but have to be installed
separately. They are governed by different license conditions than the core BAT
distribution.

Currently the collection consists of:

\begin{itemize}
\item modified version of \texttt{cramfsck} that enables unpacking cramfs
file systems
\item unmodified version of \texttt{unyaffs} that enables unpacking for some
(but not all) YAFFS2 file systems.
\item various versions of \texttt{unsquashfs} that enable unpacking variants
of SquashFS. These versions have either been lifted from vendor SDKs, the
OpenWrt project, or upstream SquashFS project.
\item two Java projects: \texttt{jdeserialze} and \texttt{ddex}, to help
respectively with unpacking serialized Java files and scanning binary files
from the Dalvik VM (Android).
\end{itemize}

The collection is split in two packages: \texttt{bat-extratools-java} contains
the two Java packages, the \texttt{bat-extratools} package contains the rest.

\appendix

\section{Analyser internals}

The analyser was written with extensibility in mind: new file systems or
variants of old ones tend to appear regularly (for example: there are at
least 5 or more versions of SquashFS with LZMA compression out there).

\subsection{Code organisation}
<commit_msg>update list of bat-extratools programs<commit_after>\documentclass[10pt]{article}
\usepackage{url}
\usepackage{a4}

\pagestyle{empty}

\setlength{\parskip}{4pt}

\author{<NAME> -- Tjaldur Software Governance Solutions}
\title{Binary Analysis Tool User and Developer Manual}

\begin{document}

\maketitle
\thispagestyle{empty}

\section{Introducing the Binary Analysis Tool}

The Binary Analysis Tool (BAT) is a framework that can help developers and
companies check binary files. Its primary application is for Open Source
software license compliance, with a special focus on supply chain management
in consumer electronics.

BAT consists of several scripts written in Python. The most important program
is the scanner for binary objects to detect the presence of Open Source
software inside binaries. There are also other programs to help with specific
tasks, such as verifying if configurations for a given BusyBox binary match
with the configuration in source code. Also included is a very experimental
program to help scan Linux kernel images and a script to derive a Linux kernel
configuration from a Linux kernel image.

\section{Installing the Binary Analysis Tool}

\subsection{Hardware requirements}

The tools in the Binary Analysis Tool can be quite resource intensive. Most of
them seem to be memorybound and I/O-bound, so it makes more sense to invest
in more memory and faster disks than in raw CPU power. Using more cores is
highly recommended, since most of the programs in the Binary Analysis Tool will
benefit from this.

\subsection{Software requirements}

To run BAT a recent Linux distribution is needed. Development is (currently)
done on Fedora 14, 15 and 16, so those platforms are likely to work best.
Other platforms (Debian 6, Ubuntu 10.10 and later) are tested with, but only to
build binary releases.

If the latest version from version control is used it is important to look at
the file \texttt{setup.cfg} to get a list of the dependencies that should be
met.

\subsubsection{Installing on Fedora}

To install on Fedora three packages are needed:
\texttt{bat-extratools}, \texttt{bat-extratools-java} and \texttt{bat}. These
can be downloaded from the website of the binary analysis tool in both prebuilt
versions and as a source RPM file. When installing the three files there should
be a list of dependencies that should be installed to let BAT work
successfully. Some of these packages are not in Fedora by default but need to
be installed through extra repositories, such as RPMfusion.

\subsubsection{Installing on Debian and Ubuntu}

To install on Debian and Ubuntu three packages are needed:
\texttt{bat-extratools}, \texttt{bat-extratools-java} and \texttt{bat}. These
can be downloaded from the website of the binary analysis tool in both prebuilt
versions. When installing the three files there should be a list of
dependencies that should be installed to let BAT work successfully. Some of
these packages are not in Debian by default but need to be installed by
enabling extra repositories such as Debian \texttt{non-free}.

The graphical interface might not work on Debian 6, because of a too old
version of \texttt{wxPython}.

\section{Analysing binaries with the Binary Analysis Tool}

BAT consists of several programs and a few helper scripts (not meant to be
used directly). The main purpose of the Binary Analysis tool is to analyse
arbitrary binaries and review results. There are two ways to do this: via the
commandline, or via a graphical user interface.

The commandline interface functionality can be found in \texttt{bruteforce.py}.
The graphical user interface is provided by \texttt{batgui} (although the
graphical user interface can do more than just scanning). Both programs are a
front end for the same functionality.

\subsection{Inner workings of the analysis process}

BAT uses a brute force approach for analysing a binary. It assumes no prior
knowledge of how a binary is constructed or what is inside the binary. Instead
it tries to determine what is inside by applying a wide range of methods, such
as looking for known identifiers of file systems and compressed files and
running external tools to find contents in the binary.

After successful identification it unpacks compressed files, media files and
file systems recursively. Finally several individual scans are applied to each
extracted file, ranging from determining dynamically linked libraries to an
elaborate ranking method to guess which sources were used to build a program.

In this section a short overview of the configuration and inner workings are
given. A more detailed description, including how to write scans, can be found
in the appendix.

\subsubsection{Analysis configuration}

The analysis process is highly configurable: methods can be simply enabled and
disabled, based on need: some methods can run for quite a long time, which
might be undesirable at times. Configuration is done via a simple configuration
file in Windows INI format.

Most sections are specific to scanning methods, except two sections.

The main configuration directive is \texttt{batconfig}. In this
section various global settings are defined, such as \texttt{multiprocessing}
which sets whether or not multiple CPUs (or cores) should be used during
scanning. The default configuration as shipped in the official BAT distribution
is to use a single CPU:

\begin{verbatim}
[batconfig]
multiprocessing = no
\end{verbatim}

By setting this setting to \texttt{yes} the program will start an extra process
per CPU that is available. In most cases it is completely safe to use
multiprocessing, but there is one important exception: if the ranking method
(described later) is used, without pregenerating the proper caching database
(or purchasing one) it is unsafe to use multiprocessing. If ranking is used and
there is a fully generated database it is completely safe and even strongly
advised to use multiprocessing, since it vastly speeds up running the program.

The other global section is \texttt{viewer}. This section is specific for the
graphical frontend and is currently not used.

\subsubsection{Enabling and disabling scans and methods}

The standard configuration file enables most of the scans and methods
implemented in BAT by default. Scans can be enabled and disabled by setting the
option \texttt{enabled} to \texttt{yes} and \texttt{no} respectively. Disabling
a scan will still allow you to enable it later in the graphical user interface.

To prevent a scan from being enabled in the graphical user interface completely
outcomment the entry in the configuration file (by starting the line with the
\texttt{\#} character), or remove the section from the configuration file.

\subsubsection{Marker search}

The first action performed is scanning a file for known markers of compressed
files and file systems. Which markers are searched for depends on the contents
of the configuration file (specifically the \texttt{magic} attribute). These
markers determine other methods, such as unpackers, are run.

The marker search cannot be enabled or disabled via the configuration file.

\subsubsection{Pre-run checks}

Before files are unpacked they are briefly inspected and if possible tagged.
Tags are used to pass hints to methods that are run later to reduce execution
time and false positives.

For example, files that only contain text are tagged as \texttt{text}, other
files are tagged as \texttt{binary} (this depends on the implementation of
Python. Python 2 only considers (by default) ASCII to be valid text). Methods
that only work on binaries can ignore anything that has been tagged with
\texttt{text}.

Other checks that are available are for valid XML, various Android formats,
ELF executables and libraries, graphics files, and so on.

\subsubsection{Unpackers}

Unpackers can be recognized in the configuration because their type is set
to \texttt{unpack}, for example:

\begin{verbatim}
[jffs2]
type        = unpack
module      = bat.fwunpack
method      = searchUnpackJffs2
priority    = 1
magic       = jffs2_le
noscan      = text:xml:graphics:pdf:compressed:audio:video
description = Unpack JFFS2 file systems
enabled     = yes
\end{verbatim}

In BAT 8.0 the following file systems, compressed files and media files can be
unpacked or extracted:

\begin{itemize}
\item file systems: cramfs, ext2/ext3/ext4, ISO9660, JFFS2, SquashFS (several
variants), romfs, YAFFS2 (specific variants)
\item compressed files and executable formats: 7z, ar, ARJ, BASE64, BZIP2,
compressed Flash, CAB, compress, CPIO, EXE (specific compression methods only),
GZIP, InstallShield (old versions), LRZIP, LZIP, LZMA, LZO, RAR, RPM, serialized
Java, TAR, UPX, XZ, ZIP
\item media files: GIF, ICO, PDF, PNG
\end{itemize}

Unpacking differs per file type. Most files use one or more identifiers that
can be searched for in a binary blob. Using this information it is possible
to carve out the right parts of a binary blob and verify if it indeed contains
a compressed file, media file or file system.

There is not always an identifier that can be searched for. The YAFFS2 file
system layout for example is dependent on the hardware specifics of the
underlying flash chip. Without knowing these specifics it is not possible to
unpack YAFFS2 file systems.

Other files (such as ARJ and ICO files) have a very generic identifier, so
there are a lot of false positives. This causes a big increase in runtime.
These two unpackers are therefore disabled by default.

If unpacking is successful a directory with unpacked files that can be
scanned further is returned, and, if available, some meta information to avoid
duplicate scanning (blacklisting information and tags).

\subsubsection{Leaf scans}

Leaf scans, also called program scans, are scans that are run on every single
file after unpacking, including files that contained files that were found
and extracted by unpackers.

Leaf/program scans can be recognized in the configuration because their type is
set to \texttt{program}, for example:

\begin{verbatim}
[iptables]
type        = program
module      = bat.checks
method      = searchIptables
noscan      = text:xml:graphics:pdf:compressed:audio:video
description = Determine presence of iptables
enabled     = yes
\end{verbatim}

The current program scans that are available in BAT are:

\begin{itemize}
\item fast string searches: dproxy, ez-ipupdate, iptables, iproute, libusb,
Linux kernel, loadlin, RedBoot, U-Boot, vsftpd, wireless-tools, wpa-supplicant
\item advanced search mode using ranking of strings and function names, using a
database
\item BusyBox version number
\item dynamic library dependencies (ELF files only)
\item Linux kernel module license (Linux kernel modules only)
\item Linux kernel version number, plus detection for several subsystems
\item PDF meta data extraction
\item architecture
\item presence of URLs indicating an open source license
\item presence of URLs indicating forges/collaborative software development
sites (SourceForge, GitHub, etcetera)
\end{itemize}

The fast string searches are meant for quick sweep scanning only. They have
their limits, can report false positives or miss finding a program. They should
only be used to signal that further inspection is necessary. For a thorough
investigation the advanced search mode should be used.

\subsubsection{Post-run methods}

Since BAT 6.0 there is the possibility to add methods that are run after all
the regular work has been performed, or ``post-run''. These methods should not
alter the scan results in any way, but just use the information from the
scanning process. A typical use case would be to present the data in a nicer
to use format than the standard report, to use more external data sources or
generate graphical representations of data.

The post-run methods have the type \texttt{postrun} in the configuration.

\begin{verbatim}
[hexdump]
type        = postrun
module      = bat.generatehexdump
method      = generateHexdump
noscan      = text:xml:graphics:pdf:audio:video
envvars     = BAT_REPORTDIR=/tmp/images:BAT_IMAGE_MAXFILESIZE=100000000
description = Create hexdump output of files
enabled     = no
\end{verbatim}


\subsubsection{Interpreting the results}

There are two formats in which \texttt{bruteforce.py} outputs its results:

\begin{enumerate}
\item dump file containing program state, complete unpacked directory tree
containing all unpacked data, plus possibly some extra generated data, such
as pictures and more reporting. These dumps are meant to be used by
\texttt{batgui}.
\item XML file (configurable)
\end{enumerate}

If configured \texttt{bruteforce.py} outputs its results in XML format on
standard output. After redirecting the output to a file it is possible to look
at this file with a commandline tool such as \texttt{xml\_pp} or a webbrowser
such as Mozilla Firefox.

This XML file is not meant for human consumption, but for use by for example
reporting tools.

The XML file starts with metadata, such as:

\begin{itemize}
\item date, plus time of the scan (local time of the computer, in UTC)
\item name of the file
\item SHA256 cryptographic checksum of the file, uniquely identifying it
\item size of the file
\item filetype as determined by \texttt{file} on a Linux system
\item relative path inside the unpacked system, plus the absolute path inside
the file system, which is useful for later analysis
\end{itemize}

If any of the scans were successful the results of the these scans can
be found in the element \texttt{scans}.

For each successful unpack action the following attributes are reported:

\begin{itemize}
\item name of the scan (corresponding to the name of the scan in the
configuration file)
\item offset in the parent file of the compressed file, file system or media
file
\end{itemize}

\subsubsection{Running \texttt{bruteforce.py}}

Invoking \texttt{bruteforce.py} is relatively simple. You will need to supply
three parameters:

\begin{enumerate}
\item \texttt{-c} : path to a configuration file
\item \texttt{-b} : path to the binary to be scanned
\item \texttt{-o} : path to an output file, where unpacked files, reports,
plus the final program state be written to
\end{enumerate}

A typical invocation looks like this:

\texttt{python bruteforce.py -c /path/to/configuration -b /path/to/binary -o
/path/to/outputfile}

XML output, if enabled in the configuration file, will be written to standard
output.

\subsection{\texttt{batgui}}

The \texttt{batgui} program is a graphical viewer for the results of
\texttt{bruteforce.py} as well as a front end for invoking the scanner itself.

\section{Additional programs in the Binary Analysis Tool}

\subsection{\texttt{busybox.py} and \texttt{busybox-compare-configs.py}}

Two other tools in BAT are \texttt{busybox-compare-configs.py} and
\texttt{busybox.py} (in the subdirectory \texttt{bat}). These two tools are
specifically used to analyze BusyBox binaries. BusyBox is in widespread use on
embedded devices and the license on BusyBox is actively enforced in court.

BusyBox binaries on embedded machines often have different configurations,
depending on the needs of the manufacturer. Since providing the correct
configuration is one of the requirements for license compliance it is important
to be able to determine the configuration of a BusyBox binary and verify that
there is a corresponding configuration file in the source code release.

The BusyBox processing tools in BAT try to extract the most likely
configuration from the binary and print it in the right format (which
depends on the BusyBox version).

\texttt{busybox.py} is used to extract the configuration from a binary.
Afterwards \texttt{busybox-compare-configs.py} can be used to compare the
extracted configuration with a vendor supplied configuration.

\subsubsection{Extracting a configuration from BusyBox}

Extracting a configuration from a BusyBox executable is done using
\texttt{busybox.py} which can be found in the \texttt{bat} directory. It needs
two commandline parameters: the path to the binary and the path to a directory
containing a directory \texttt{configs} which has files containing mappings
from BusyBox applet names to BusyBox configuration directives. By default this
value is hardcoded as \texttt{/etc/bat}, but this might change in the future.

\begin{verbatim}
python bat/busybox.py -b /path/to/busybox/binary -c
     /path/to/pre/extracted/configs > /path/to/saved/config
\end{verbatim}

This command will save the configuration to a file, which can be used as an
input to \texttt{busybox-compare-configs.py}.

\subsubsection{Comparing two BusyBox configurations}

After extracting the configuration the extracted configuration can be compared
to another configuration, for example a configuration as supplied by a vendor:

\begin{verbatim}
python busybox-compare-configs.py -e /path/to/saved/config
    -f /path/to/vendor/configuration -n $version
\end{verbatim}

\subsection{\texttt{extractkernelstrings.py}, \texttt{extractkernelconfig.py}
\\and \texttt{findkernelstrings.py}}

The three programs \texttt{extractkernelstrings.py},
\texttt{extractkernelconfig.py} and \texttt{findkernelstrings.py} are very
experimental tools to derive a configuration from a Linux kernel image. They
are far from mature and are not yet ready for production use.

Once they are ready for production use a description for the scripts will be
included in this manual. A description of the technical uses that need to be
solved, plus a motivation on why and how to solve them can be found in the
appendix.

%\subsubsection{\texttt{extractkernelstrings.py}}
%\subsubsection{\texttt{extractkernelconfig.py}}
%\subsubsection{\texttt{findkernelstrings.py}}


\section{Binary Analysis Tool extratools collection}

To help with unpacking non-standard file systems, or standard file systems for
which there are no tools readily available on Fedora or Ubuntu there is also
a collection of tools that can be used by BAT to unpack more file systems.
These tools are not part of the standard distribution, but have to be installed
separately. They are governed by different license conditions than the core BAT
distribution.

Currently the collection consists of:

\begin{itemize}
\item modified version of \texttt{code2html} (which is unmaintained by the
upstream author) that adds support for various more programming languages.
\item unmodified version of \texttt{romfsck} needed for unpacking romfs file
systems.
\item modified version of \texttt{cramfsck} that enables unpacking cramfs
file systems.
\item unmodified version of \texttt{unyaffs} that enables unpacking for some
(but not all) YAFFS2 file systems.
\item various versions of \texttt{unsquashfs} that enable unpacking variants
of SquashFS. These versions have either been lifted from vendor SDKs, the
OpenWrt project, or upstream SquashFS project.
\item \texttt{bat-visualisation} containing a few custom tools to help generate
pictures. These might be removed in the future.
\item two Java projects: \texttt{jdeserialze} and \texttt{ddex}, to help
respectively with unpacking serialized Java files and scanning binary files
from the Dalvik VM (Android).
\end{itemize}

The collection is split in two packages: \texttt{bat-extratools-java} contains
the two Java packages, the \texttt{bat-extratools} package contains the rest.

\appendix

\section{Analyser internals}

The analyser was written with extensibility in mind: new file systems or
variants of old ones tend to appear regularly (for example: there are at
least 5 or more versions of SquashFS with LZMA compression out there).

\subsection{Code organisation}

{'fraction_non_alphanumeric': 0.04920692141312184, 'fraction_numerical': 0.002858172829333608, 'mean_word_length': 4.744268599319628, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3053344', 'n_tokens_mistral': 10206, 'n_tokens_neox': 9741, 'n_words': 5481}