<commit_before># python-idb

python-idb is a library for accessing the contents of [IDA Pro](https://www.hex-rays.com/products/ida/) databases (.idb files).
It provides read-only access to internal structures such as the B-tree (ID0 section), name address index (NAM section), and flags index (ID2 section).
The library also provides analysis of B-tree entries to expose logical structures such as functions, cross references, bytes, and disassembly (via [Capstone](http://www.capstone-engine.org/)).
An example use for python-idb might be to run IDA scripts in a pure-Python environment.

<NAME> (<mailto:<EMAIL>>) provided the initial research into the low-level structures in his projects [pyidbutil](https://github.com/nlitsme/pyidbutil) and [idbutil](https://github.com/nlitsme/idbutil).
Willem deserves substantial credit for reversing the .idb file format and publishing his results online.
This project heavily borrows from his knowledge, though there is little code overlap.


## example use:

### example: list functions names

In this example, we list the effective addresses and names of functions:

```
In [4]: import idb
   ...: with idb.from_file('./data/kernel32/kernel32.idb') as db:
   ...:     api = idb.IDAPython(db)
   ...:     for ea in api.idautils.Functions():
   ...:         print('%x: %s' % (ea, api.idc.GetFunctionName(ea)))

Out [4]: 68901010: GetStartupInfoA
   ....: 689011df: Sleep
   ....: 68901200: MulDiv
   ....: 68901320: SwitchToFiber
   ....: 6890142c: GetTickCount
   ....: 6890143a: ReleaseMutex
   ....: 68901445: WaitForSingleObject
   ....: 68901450: GetCurrentThreadId
        ...
```

Note that we create an emulated instance of the IDAPython scripting interface, and use
this to invoke `idc` and `idautils` routines to fetch data.


### example: run an existing IDAPython script

In this example, we run the [yara_fn.py](https://gist.github.com/williballenthin/3abc9577bede0aeef25526b201732246) IDAPython script to generate a [YARA](https://virustotal.github.io/yara/) rule for the function at effective address 0x68901695 in kernel32.idb:

[![asciicast](https://asciinema.org/a/14.png)](https://asciinema.org/a/14?theme=monokai)

The target script `yara_fn.py` has only been slightly modified:
  - to make it Python 3.x compatible, and
  - to use the modern IDAPython modules, such as `ida_bytes.GetManyBytes` rather than `idc.GetManyBytes`.


## what works

  - 50 unit tests that demonstrate functionality including file format, B-tree, analysis, and idaapi features.
  - read-only parsing of .idb files from IDA Pro v6.95
    - extraction of file sections
    - B-tree lookups and queries (ID0 section)
    - flag enumeration (ID1 section)
    - named address listing (NAM section)
  - analysis of artifacts that reconstructs logical elements, including:
    - root metadata
    - loader metadata
    - entry points
    - functions
    - structures<commit_msg>readme: lets use our asciinema link!<commit_after># python-idb

python-idb is a library for accessing the contents of [IDA Pro](https://www.hex-rays.com/products/ida/) databases (.idb files).
It provides read-only access to internal structures such as the B-tree (ID0 section), name address index (NAM section), and flags index (ID2 section).
The library also provides analysis of B-tree entries to expose logical structures such as functions, cross references, bytes, and disassembly (via [Capstone](http://www.capstone-engine.org/)).
An example use for python-idb might be to run IDA scripts in a pure-Python environment.

<NAME> (<mailto:<EMAIL>>) provided the initial research into the low-level structures in his projects [pyidbutil](https://github.com/nlitsme/pyidbutil) and [idbutil](https://github.com/nlitsme/idbutil).
Willem deserves substantial credit for reversing the .idb file format and publishing his results online.
This project heavily borrows from his knowledge, though there is little code overlap.


## example use:

### example: list functions names

In this example, we list the effective addresses and names of functions:

```
In [4]: import idb
   ...: with idb.from_file('./data/kernel32/kernel32.idb') as db:
   ...:     api = idb.IDAPython(db)
   ...:     for ea in api.idautils.Functions():
   ...:         print('%x: %s' % (ea, api.idc.GetFunctionName(ea)))

Out [4]: 68901010: GetStartupInfoA
   ....: 689011df: Sleep
   ....: 68901200: MulDiv
   ....: 68901320: SwitchToFiber
   ....: 6890142c: GetTickCount
   ....: 6890143a: ReleaseMutex
   ....: 68901445: WaitForSingleObject
   ....: 68901450: GetCurrentThreadId
        ...
```

Note that we create an emulated instance of the IDAPython scripting interface, and use
this to invoke `idc` and `idautils` routines to fetch data.


### example: run an existing IDAPython script

In this example, we run the [yara_fn.py](https://gist.github.com/williballenthin/3abc9577bede0aeef25526b201732246) IDAPython script to generate a [YARA](https://virustotal.github.io/yara/) rule for the function at effective address 0x68901695 in kernel32.idb:

[![asciicast](https://asciinema.org/a/9n8qxpChjBTrF1tYAbp7ABIFW.png)](https://asciinema.org/a/9n8qxpChjBTrF1tYAbp7ABIFW?theme=monokai)

The target script `yara_fn.py` has only been slightly modified:
  - to make it Python 3.x compatible, and
  - to use the modern IDAPython modules, such as `ida_bytes.GetManyBytes` rather than `idc.GetManyBytes`.


## what works

  - 50 unit tests that demonstrate functionality including file format, B-tree, analysis, and idaapi features.
  - read-only parsing of .idb files from IDA Pro v6.95
    - extraction of file sections
    - B-tree lookups and queries (ID0 section)
    - flag enumeration (ID1 section)
    - named address listing (NAM section)
  - analysis of artifacts that reconstructs logical elements, including:
    - root metadata
    - loader metadata
    - entry points
    - functions
    - structures
{'fraction_non_alphanumeric': 0.10728229502631133, 'fraction_numerical': 0.038363605499915125, 'mean_word_length': 3.766990291262136, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 14, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4344514', 'n_tokens_mistral': 1974, 'n_tokens_neox': 1809, 'n_words': 636}