<commit_before>% Abuse Contact datasets for IT security incident handling
% <NAME> <<EMAIL>>; <NAME> <<EMAIL>>; <NAME> <<EMAIL>>, 
% 2014/09/18

# Abstract

This document aims at listing the different datasets listing abuse contacts and CERTs. The goal is to give the audience an overview of datasets, show where this data can be found, who has access to the data and list known issues with the data sets.




# Terminology


| Term           | Explanation                                                            |
|:---------------|:-----------------------------------------------------------------------|
|CERT     		 | Computer Emergency Response Team. Synonymous with CSIRT |
|Constituency    | In this context, by constituency we mean the group of people, networks, ASNs, etc. a CERT addresses as their responsibility | 
|CSIRT    		 | Computer Security Incident Response Team  |
|FIRST			 | First.org - Forum for Incident Response and Security Teams |
|LIR      		 | Local Internet Registry; see (1) for more details  |
|Object			 | As defined by RIPE: a XXX RIPE: please define XXX |
|Network Resource | 	Or "Netresource". An ASN, netblock, IP address, domain name or hostname |
|Registrant 	 | 	A person registering a domain name for individual or business usage   |
|Registrar 		 | Often called domain registrar. An organisation handling the registration of domain names on behalf of a registrant  |
|RIPE NCC 		 | RIR serving Europe, Central Asia and the Middle East; see (2)  |
|RIR      		 | Regional Internet Registry; see (1) for more details  |
|RIS      		 | Routing Information Service  |
|TI       		 | Trusted Introducer  |

Table: Terminology


(1) [http://www.ripe.net/internet-coordination/internet-governance/internet-technical-community/the-rir-system](http://www.ripe.net/internet-coordination/internet-governance/internet-technical-community/the-rir-system)         
(2) [http://www.ripe.net](http://www.ripe.net)

# Intended Audience

CERTs and IT security incident abuse handlers as well as professionals working on automating IT security incident handling. 


# Problem description and motivation

CERTs - especially national CERTs - need to look up contact information very frequently for different resources on the Internet.  Examples of such lookups might be: given the IP address 1.2.3.4, give me the best matching abuse contact email address. Or given the domain www.example.com, what is the best contact for sending IT security incident notifications to?  

While the classical way to look up abuse contacts is to query some RIR whois service (such as RIPE's whois service) , there are some issues with this:
  * first of all, it might not be the data that's needed: for example the query might refer more to a domain instead of an IP address 
  * the contact registered for the specific network might be non-responsive or the wrong contact
  * bulk queries are usually not supported for whois
  * information in whois quickly gets outdated

As a consequence, a lot of different datasets for abuse contact lookups sprung up. They contain different data, have different access methods and -permissions.

This document aims at describing these different datasets in order to provide an overview. The document "abuse-lookups.mkd" deals with how to query these datasets.

This document is intended to be a request for input from the Internet community. The hope is to eventually define a standard for abuse contact lookups.



# General remarks and overview of datasets

For all potential datasets, there are two orthogonal approaches which must be considered: looking up abuse contacts for:

  * name based resources (e.g. domain names, host names)
  * number based resources (IP addresses, AS numbers)

This document assumes that a resource is either a name based resource or a number based resource. 


![Potential abuse contact lookups](./img/contact-lookup-flow.png)


Figure 1 describes possible paths in looking up abuse contact information based on the given resources and some of the datasets involved. 
Depending on the case and depending on the resource, there might be multiple paths for lookups. This will always be a case-by-case decision on behalf of the incident reporter. 

*Example:* the incident reporter finds a hacked webpage. Naturally, he will try to contact the domain owner (name based resource lookup) - the admin\_c and possibly also the tech\_c.  However, as time passes, the incident reporter notices more and more hacked webpages. Incidentally, they all turn out to be hosted on the very same IP address block.
After some investigations, the incident reporter decides that there must be something wrong with the web page hoster's servers. Hence, he decides to contact the owner of the IP block (the hoster) in order to hand him over a long CSV file of all web pages which are hacked and residing on the hosters' IP range.

In practice, often enough it is not clear, which path to choose. 

Sometimes, an incident report might want to contact a single point of contact (PoC) for a whole country. Typically this would be a national CERT which in turn will use some kind of contact lookup mechanism (possibly more accurate) to assign the incident report to a locally trusted and well known incident handler. For this case (sending incident reports to a national CERT), the report must find out the country a resource is assigned to. Since there is a well known dataset of national CERTs (see XXX), the problem thus can be reduced to finding the country code of the resource (for example: given an IP address 1.2.3.4, what is the country this IP address resides in?).
We will discuss this special case in section XXXX.

Whoever, we must remember that the problem of whom to contact is a choice which must be made on a case-by-case basis. Therefore this framework must be flexible enough to support both approaches.


# Existing datasets

%%% FIXME: document !## Perl's Net::Abuse

## Trusted Introducer

### Overview

Trusted Introducer
([http://trusted-introducer.org/](http://trusted-introducer.org/)) ("TI")
maintains a directory of IT security teams and TI members. While the database
is worldwide, there is a strong focus on Europe. TI assigns different maturity
levels to CERTs: listed, accredited and certified.  TI also offers a CSV Export
of its database to TI members (you need to be logged into the TI website to
access this CSV list).  The CSV list is searchable by netresources such as ASN,
country code or IP range.  However, at the time of writing this document, there
are still some issues with automating the querying of the TI data.

### Data in the dataset

The data in the TI database contains amongst other fields the following information:


  * Team name
  * TI Level (Listed, Accredited, ...)
  * URL of the team description on the TI webpage
  * Country
  * The constituency of the CERT:
    * ASN
    * netblocks
    * domains
  * Email Address(es) of the CERT
  * PGP key IDs of the CERT's team key
  * Telephone number
  * Postal address
  * Business hours
  * Time Zone
  * Team Representative
  * Website (URL) of the CERT

### Original or synthetic data?
This is an original dataset.

### Where to find it?

http://trusted-introducer.org/


### Who has access?

The public has access to a searchable list of teams via the website. 
TI members have access to the members view and can download a CSV file export of the database.

### APIs / automatic querying

Currently, a logged in user (TI member) can download a CSV file dump of the database in the internal section and query the (downloaded CSV) data automatically.

### Maintainers

Trusted Introducer.

### Liveliness

Trusted Intorducer confirms the data with each team representative every 12 months (XXX FIXME: check this XXX).


### Known issues






## FIRST.org

### Overview
FIRST (Forum for Incident Response and Security Teams) is one of the oldest IT security forums in the world. For a history, see http://first.org/about/history.
As such, FIRST maintains an extensive list of IT Security teams (FIRST's members) around the world. 



### Data in the dataset

The data in the FIRST directory contains amongst other fields the following information:

  * Team Info
    * Short team name
    * Team representative
    * Membership type
    * Date of establishment
    * Website (URL)
  * Constituency
    * Type of constituency
    * ASNs
    * Internet domain addresses
    * Country of constituency
  * Team contact information
    * Tel number
    * Email address
    * Postal address
    * Time zone
  * Business hours
    * Business hours
    * Reachability outside of business hours
  * Services
    * Textual description: reactive, proactive, ...
  * Cryptography
    * PGP key ID
    * PGP fingerprint
    * The full team PGP key (ascii armored)
  * Team members
    *  A list of names of team members


### Original or synthetic data?
Original data

### Where to find it?

http://www.first.org -> members

### Who has access?

Access is public 


### APIs / automatic querying

Currently at the time of this writing, there is no API and no way to automatically query data by netressources.

### Maintainers

FIRST.org Inc. maintains the database.

### Liveliness

FIRST members can update their data via web form.

### Known issues

The First directory is often lacking ASNs for FIRST team members.




## CERT.org's national CERT database

### Overview

CERT.org hosts a yearly national CERT (CSIRT) meeting([https://www.cert.org/incident-management/national-csirts/meeting/](https://www.cert.org/incident-management/national-csirts/meeting/). As part of these meetings, CERT.org also maintains a fairly accurate list of national CERTs ([https://www.cert.org/incident-management/national-csirts/national-csirts.cfm](https://www.cert.org/incident-management/national-csirts/national-csirts.cfm)).


Please note that the definition of "national CERT" is not 100% clear. 


### Data in the dataset

  * Organisation (i.e. the CERT)
  * Organisational Units
  * Groups
  * Personal data on CERT team members
  * Cryptography 
    * S/Mime or PGP or Kerberos infos
    * PGP key id
    *  
  

### Original or synthetic data?


Original 

### Where to find it?

https://nationalcsirts.cert.org/


### Who has access?

XXX FIXME: public? XXX

### APIs / automatic querying

XXX FIXME: CSV is public? XXX


### Maintainers


CERT.org 
CERT Coordination Center
Software Engineering Institute
Carnegie Mellon University
Office: +1 412 268 8876
CERT Hotline: +1 412 268 7090


### Liveliness

Somewhat up to date: the yearly national CSIRT conference helps in keeping the directory up to date.


### Known issues








## ENISAs CERT Directory

### Overview

ENISA collected a list of CERTs in Europe on their website. The list comes from TI as well as FIRST.

### Data in the dataset

XXX Cosmin, please add XXX

### Original or synthetic data?

Synthetic.

### Where to find it?


https://www.enisa.europa.eu/
and:
https://www.enisa.europa.eu/activities/cert/background/inv

(needs Javascript)

### Who has access?

Public.


### APIs / automatic querying

There is no API, but the data can be downloaded (XML file) at: XXX Cosmin, please add XXX

### Maintainers

ENISA 

### Liveliness

XXX Cosmin, please add XXX

### Known issues

XXX Cosmin, please add XXX








## CERT.at's national CERT database
CERT.at used this national CERT database and created an online lookup tool at [https://contacts.cert.at](https://contacts.cert.at).


### Overview

### Data in the dataset

### Original or synthetic data?

### Where to find it?

### Who has access?

### APIs / automatic querying

### Maintainers

### Liveliness

### Known issues








## RIPE database

( with a special focus on abuse-c, IRT object)

XXX Mirjam , please add/change/delete XXX

Following a proposal made by the RIPE community in XXX the RIPE policy ripe-563 made it mandatory for every resource object (inetnum, inet6num and aut-num) to have a dedicated abuse contact. This contact is usually referred to as "abuse-c" contact and it is the preferred way to report any form of abuse to.

The IRT (incident response team) object was created with a similar idea in mind but its deployment is even being around for years not very low and so is it is not widely used. Chances are high that this object will be dismissed. Abuse-c is considered to superceed the IRT object.


/XXX Mirjam , please add/change/delete XXX


### Overview

### Data in the dataset

### Original or synthetic data?

### Where to find it?

### Who has access?

### APIs / automatic querying

### Maintainers

### Liveliness

### Known issues






# Supporting databases

Often enough, there is a requirement to transform one network resource (e.g. an IP address) into a different network ressource (e.g. a country code) in order to look up the abuse contact (e.g. the national CERT) for the IP address. 
This section describes these supporting additional databases.



## Country Code lookups

As discussed in section 'General remarks on abuse contact lookups', some incident reports should simply go to the national CERT. For this task, it is important to find the country code of an IP address or a domain.

In the case of IP addresses, there are a couple of databases which will be discussed below.

### RIPE DB

Ripe stat offers a RESTful API  for XXX Christian / Mirjam, please extend XXX

### Maxmind

[Maxmind](https://www.maxmind.com/) is one of the most well known databases for IP to country code mappings. 
There are a couple of versions of Maxmind with varying degrees of accuracy and costs: 

* GeoIP City
* GeoIP Country
* GeoIP Lite
* GeoIP Legacy

We advise the reader to check Maxminds webpage for current offerings.

Using the GeoIP database in a wide variety of languages is very straightforward. In this document we give an example in Python (from [Readthedocs](http://geoip2.readthedocs.org/en/latest/)).

```
>>> import geoip2.database
>>> reader = geoip2.database.Reader('/path/to/GeoLite2-City.mmdb')
>>> response = reader.city('8.8.8.8')
>>> response.country.iso_code
'US'
```




### Team Cymru

[Team Cyrmu](https://www.team-cymru.org/Services/ip-to-asn.html) provides an IP address to ASN mapping service via whois ([RFC3912](https://tools.ietf.org/html/rfc3912)) as well as via netcat, http(s) or DNS based lookups. As part of this lookup mechanism, there are options to also get the country code of a given IP address. To the best of the authors knowledge, the IP to country mapping is done via the RIR databases.

*Example lookup*: 
```
$ whois -h whois.cymru.com  " -c   8.8.8.8"

AS      | IP               | CC | AS Name
15169   | 8.8.8.8          | US | GOOGLE - Google Inc.,US
```

or as a shell script:
```
#!/bin/bash

ip=$1
cmd=$(cat <<EOT;
begin
verbose
 -c
$ip
end
EOT)

echo "$cmd" | nc whois.cymru.com 43

```
Example output:

```
Bulk mode; whois.cymru.com [2014-05-12 19:30:14 +0000]
Error: no ASN or IP match on line 3.
15169   | 8.8.8.8          | 192.168.3.11/24          | US | arin     |            | GOOGLE - Google Inc.,US
```


## Shortcomings of existing IP to country lookups

RIPE country code lookups usually return the country of registration of the organisation which was assigned the specific netblocks. However, this is not necessarily the same country as the individual IP address. This is especially true with international organisations or Internet Service Providers.

On the other hand, Maxmind data might not find to correct country of an IP address or even return the special country codes 'A1' or 'A2' for satellite uplinks. 

For the purpose of delivering abuse reports, finding the country of registration might be  arguably a better approach in many cases.


# Shortcomings of existing databases



## RIPE NCC Data Sets 

RIPE NCC has a list of publicly available data sets that could be useful for CERTs:
  * Registration data:
    * RIPE DB (containing IP resource information and contact data)
    * Routing DB (containing routing policy)
    * Regional Internet Registry Statistics (which IP resources are allocated when and to whom)
  * Routing data collected by the 13 Routing Information Service (RIS) route collectors (https://www.ripe.net/data-tools/stats/ris/)
  * Active measurements done by the RIPE Atlas (https://atlas.ripe.net)
   

The following sections look at these data sets in more detail. All data sets described here 
are publicly avilable and are made accessible via various interfaces.

### RIPEstat (MIRJAM: Maybe move after Registration data to be consistent with list above?)

RIPEstat is a web-based interface that provides everything you ever wanted to know about 
IP address space, Autonomous System Numbers (ASNs), and related information for hostnames 
and countries in one place. It has multiple APIs ranging from a raw data centric Data API to a 
visualisation rich Widget API.
For documentation and a description of all the APIs:

https://stat.ripe.net/index/documentation/interfaces-apis

In the listing of the following data sets it will be mentioned if the data can be accessed
via RIPEstat.



### Registration Data

MIRJAM: Include a sentence describing what The RIPE registry means and entails and refer to ripe-508.

#### RIPE DB

Christian, 20014-05-08: Status = needs review

The RIPE Database contains registration information for networks in the RIPE NCC service region. It shows the organisations that hold the resources, the country the organisation is registered in, and contact details for the networks.
  
The Database can be searched by using the web interface (https://apps.db.ripe.net/search/query.html) or by directing your whois client to whois.ripe.net. A full-text search is also available 
https://apps.db.ripe.net/search/full-text.html.
  
FAQ: http://www.ripe.net/data-tools/db/faq
Documentation: http://www.ripe.net/data-tools/support/documentation 
  
This data is accessible via RIPEstat as a widget and as a data call.
Widget API: https://stat.ripe.net/widget/registry-browser
Data API: https://stat.ripe.net/docs/data\_api#RegistryBrowser
  
MIRJAM: I suggest to take out all the Q's and A's and format these sections with sub headings (the same for each of these sections). Might be better readable.

##### Source of the data:
- The data is created and maintained by the RIPE NCC and by the resource holders following a defined scheme for objects and attributes (MIRJAM: What does this mean: "scheme..."?).

##### Quality of the data and how often is the quality checked:
- The data quality in the RIPE registry is very important for network operations and it one of the main tasks of the RIPE NCC. The RIPE NCC is frequently cecking the quality of the data and is working with the maintainers of the data to keep it up to date.

##### Data set available in batch:
- RIPE Database APIs can be queried in batch but there is a limit if personal data is being fetched:
	- Number of personal data sets returned in queries from an IP address - 1000 per 24 hours (from "Accetable Use Policy" http://www.ripe.net/data-tools/support/documentation/aup)

This limit also applies for using RIPEstat but it is not per IP address per user account.

##### API for batch query interface:
- The command line interface (port 43) or RIPEstat can be used.

#### Historical data:
- The RIPE Database interfaces provide a historical lookup feature based on versions, see https://labs.ripe.net/Members/kranjbar/proposal-to-display-history-of-objects-in-ripe-database
for details.
In RIPEstat the data call for the Registry-Browser returns historical information for each queried object but this requires the user to be logged in as a RIPE NCC member.

Notes:
  --> interesting for CERTs: whowas service
    --> should exist in stat.ripe.net 

Christian: 
	 Whowas service I could only find for ARIN. For the RIPE DB this is basically
	 the solution with versions (mentioned previous).
	 I'm trying to find out if other RIRs provide similar services. Test!
:
    


#### Routing Registry

The RIPE Routing Registry is a sub-set of the RIPE Database and holds information about routing on the Internet. The routing information is stored in Autonomous System (AS) objects. The information in these AS objects show how a particular network is routed on the internet. 

The RIPE Routing Registry is part of the Internet Routing Registry (http://www.ripe.net/data-tools/db/the-internet-routing-registry-history-and-purpose), a collection of databases that mirror each other.

Sample output
```
route:           192.0.2.0/24
descr:           TEST-NET-1
origin:          ASXXXX
mnt-by:          ASXXXX-MNT
mnt-by:          YYYYY-LIR-MNT
source:          RIPE # Filtered
```

##### Data set available in batch

Same terms and conditions apply as for RIPE Database.

##### API for batch query interface

Same terms and conditions apply as for RIPE Database.

##### Historical data

See RIPE DB section (to be checked if history is available for route objects as well) 




#### Regional Internet Registry (RIR) statistics 

The RIR statistics provide an authoritative source of the status of all IP resources. It is maintained collectively by all RIRs. They are intended to provide a snapshot of the status of Internet number resources, without any transactional or historical details. (MIRJAM: Christian, I am not sure we need to sepcify this here. The most important fact is that this is the authoritative source for all IP resources, but that it doesn't provide a lot of details).

The statistics are primarily published as a text file by each RIR once a day. The statistics are usually available before 23:59:59 in the local time zone the RIR is in.

The published statistics follow a standardised format which contains a header and a body.
The header consists of a version line and a summary line.

The format of the version line is:

`version|registry|serial|records|startdate|enddate|UTCoffset`

| Field      | Description                                                               |
|:-----------|---------------------------------------------------------------------------| 
| version    | format version number of this file, currently 2                           |
| registry   | name of the producing RIR                                                 |
| serial     | serial number of this file (within the creating RIR series)               |
| records    | number of records in file, excluding blank lines, summary lines,          |
|            | the version line and comments                                             |
| startdate  | start date of time period yyyymmdd format                                 |
| enddate    | end date of period in yyyymmdd format                                     |
| UTCoffset  | offset from UTC (+/- hours) of local RIR producing file                   |


The format of the summary line is:

`registry|*|type|*|count|summary`

| Field      | Description 
|:-----------|---------------------------------------------------------------------------|
| registry   | name of the producing RIR                                                 |
| *          | an ASCII \* (unused field, retained for spreadsheet purposes)             |
| type       | as defined in the format used for a record in the body                    |
| count      | sum of the number of record lines of this type in the file                |
| summary    | the ASCII string 'summary' (to distinguish a record line in the body)     |

A record in the body follows this format:


`registry|cc|type|start|value|date|status[|extenstions...]`

| Field       | Description                                                              |
|:------------|--------------------------------------------------------------------------|
| registry    | name of the producing RIR                                                |
| cc          | ISO 3166 2-letter code of the organization to which the allocation or    |
|             | assignment was made                                                      |
| type        | Type of Internet number resource represented in this record. One value   | 
|             | from the set of defined strings:                                         |
|             |   `{asn, ipv4, ipv6}`                                                    |
| start       | In the case of records of type 'ipv4' or 'ipv6' this is the IPv4 or IPv6 |
|             | 'first address' of the range.                                            |
|             | In the case of an 16 bit AS number the format is the integer value in    |
|             | the range 0 to 65535, in the case of a 32 bit ASN the value is in the    |
|             | range 0 to 4294967296. No distinction is drawn between 16 and 32 bit ASN |
|             | values in the range 0 to 65535.                                          | 
| value       | In the case of IPv4 address the count of hosts for this range. This      |
|             | count does not have to represent a CIDR range.                           |
|             | In the case of an IPv6 address the value will be the CIDR prefix length  |
|             | from the 'first address' value of \<start\>.                             |
|             | In the case of records of type 'asn' the number is the count of AS from  |
|             | this start value.                                                        |
| date        | Date on this allocation/assignment was made by the RIR in the format     |
|             | yyyymmdd.                                                                |
|             | Where the allocation or assignment has been transferred from another     |
|             | registry, this date represents the date of first assignment or           |
|             | allocation as received in from the original RIR.                         |
|             | It is noted that where records do not show a date of first assignment,   |
|             | this can take the 0000/00/00 value.                                      |
| status      | Type of allocation from the set:                                         |
|             |  `{allocated, assigned}`                                                 |
|             | This is the allocation or assignment made by the registry producing the  |
|             | file and not any sub-assignment by other agencies.                       |
| extensions  | Any extra data on a line is undefined, but should conform to use of the  |
|             | field separator used above.                                              |


The files are primarily accessible via each RIR FTP site:

| RIR         | FTP site                                                                 |
|:------------|--------------------------------------------------------------------------|
| AFRINIC     | ftp://ftp.afrinic.net/pub/stats/afrinic/                                 |
| APNIC       | ftp://ftp.apnic.net/pub/stats/apnic/                                     |
| ARIN        | ftp://ftp.arin.net/pub/stats/arin/                                       |
| LACNIC      | ftp://ftp.lacnic.net/pub/stats/lacnic/                                   |           
| RIPE NCC    | ftp://ftp.ripe.net/pub/stats/ripencc/                                    |

The general file name has this format:

`delegated-<registry>-yyyymmdd`

Data compression is optional. If compressed, the normal file suffix is used to denote the compression algorithm (.gz, .bz2, .zip etc).
The most recent file must be available in a non-compressed form and has this file name:

`delegated-<registry>-latest`

There is not obligation on any registry to retain previous files, once a new file is available for public access but RIR usually keep an archive of previous files. 

##### Note on the data quality and accuracy

The RIR statistics are based on registration data that is maintained by the RIR itself which usually guarantees high data quality.
The country code that is stated with the allocation/assignment is based on the location of the organisation holding the resource at the time the request was issued. It is not guaranteed that this location is kept up-to-date.
Keep in mind that the nature of this data is registration related and not usage related. For usage related location information other providers are more suitable e.g. MaxMind GeoIP database.


##### RIPEstat

The RIR statistics are not directly retrievable via RIPEstat but there is a data call that provides a list of Internet number resources (IPv4, IPv6 and ASNs) per country:

https://stat.ripe.net/docs/data_api#CountryResourceList

E.g.: 
Internet number resources for Germany
`https://stat.ripe.net/data/country-resource-list/data.json?resource=de`

### Routing Data 

#### What is it?

Internet routing data is significantly different from the registration data as the routing data shows the state of routing tables. It has to be noted that this is not a single routing table as each point from where routing data gets collected adds its distinct vantage point.

The RIPE NCC collects and stores Internet routing data since 2001 using RIS which is short for Routing Information Service. The RIS system consists of a network of route collectors (RRC) which are running a customised routing software (Quagga) to passively collect routing information from other networks that are connected to the RRC (peering).

RIS: http://www.ripe.net/data-tools/stats/ris/

#### Fetching The Data

The RIS routing data can be accessed through the raw data files or via APIs the RIPE NCC provides.

##### Raw Data

The raw data files are a direct copy of the data that is being sent by the RRCs to a central storage location. These files come in two flavour, dumps and updates. 
Dumps contain the entire routing table for the time the dump was created which is 3 times a day (every 8 hours) starting from midnight (00:00, 08:00 and 16:00 UTC).
Updates contain BGP updates (announcements and withdrawals) aggregated over 5 minutes. 

The links to the data files can be found on http://www.ripe.net/data-tools/stats/ris/ris-raw-data. The URL follows a simple pattern:

`http://data.ris.ripe.net/<rrc_#>/` (e.g. http://data.ris.ripe.net/rrc01/ to access dumps and updates for RRC01)

Dumps as well as updates are stored in MRT format which can be read by libbgdump, a library written in C by <NAME>. More details on this library and where to download can be found on the page mentioned above.
On the same page there is also a Python library state, PyBGPdump. Unfortunately this library has been reported to fail for some dump files. If Python support is necessary mrtparse (https://github.com/YoshiyukiYamauchi/mrtparse) might be an alternative.


##### APIs

RIPEstat offers a couple of data calls which are based on the RIS routing data. This form of access is usually more convenient than having to extract the information from the raw data files.


* 	Resource-Overview (Prefix-Overview, As-Overview)
	This data call gives a summary of the given prefix, including whether and by whom it is announced.
	
	Documentation: 
	+ 	https://stat.ripe.net/docs/data_api#PrefixOverview, 
	+ 	https://stat.ripe.net/docs/data_api#AsOverview


* 	Routing-Status
	This data call returns a summary of the current BGP routing state of a given IP prefix or ASN, as observed by the RIS route collectors.
	
	Documentation: 
	+ 	https://stat.ripe.net/docs/data_api#RoutingStatus


* 	Routing-History
	This data call shows the history of announcements for prefixes, including the origin ASN and the first hop.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#RoutingHistory

	
* 	Asn-Neighbours
	This data call shows information on the network neighbours for a given ASN. This includes statistical information as well as the list of observed ASN neighbours.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#AsnNeighbours

	
* 	Asn-Neighbours-History
	This data call returns information about neigbouring ASNs of a queried ASN extended with history.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#AsnNeighboursHistory


* 	BGPlay
	This data call represents the scenario of what occurred to the BGP routes of a resource over a period of time.
	It includes data that defines the initial BGP state at the start time of the query, and all the BGP updates observed from then until the end time, as well as a description of all the AS nodes, and RIS BGP peers involved in the result.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#BGPlay
	

* 	Country-Routing-Statistics
	This data call returns statistics on Internet resources for a country - this includes:

	+	number of ASNs seen in routing data and registration data
	+	number of prefixes in routing data and registration data (split into IPv4 and IPv6)
	+	amount of IPv4 space seen in routing data as well as registration data
	The results can be restricted to a specific time period as well the granularity is variable but can be set explicitly.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#CountryResourceStats

For a general introduction on how to use the RIPEstat API please refer to https://stat.ripe.net/docs/data_api.

The query model that is used at the moment on RIPEstat supports queries for single resource at a time which could make it less convenient for a larger number of resource to be looked up. An already requested feature is to have also support for bulk request which will be added in the upcoming development sprints for RIPEstat.

A somehow similar but a bit more limited service is provided by RISwhois, http://www.ripe.net/data-tools/stats/ris/riswhois. The benefit of RISwhois is its performance.


#### 

### External Data Sets Served Via RIPEstat

Then there are other data sets that we provide via RIPEstat but they don't originate from the RIPE NCC, 
like MaxMind's GeoIP data set, some blacklists... the benefit can be a consistent data interface.


RIPEstat?

  --> geolocation. Quelle? Bitte genau genau dokumentieren! stats file ist die
  org, an die das vergeben wurde.  As CERTs we often have the problem that we
  need to deliver reports to national CERTs. So the question of proper
  geolocation comes immediately.  Maxmind seems to be the de-facto standard.
  However, RIPE data is used as a basis for cymru IP-to-ASN.

  We should document which geolocation solutions exist, how precise they are
  and when to use which.  Sometimes an IP address is in a country A, the RIPE
  entry for it points to an organisation in country B, maxmind will say it is
  country C.

  !!! super idee!!! geolocation history DB "wherewas?"
  organisational country vs. ip address is in use in country X

  --> google or sky hook wireless geolocation? Is it possible to get that data?
  maybe Apple as well?
  XXX insert CCC talk XXX

  traceroute history of ATLAS? did something change?

  Vorschlag: 
   1) TI DB wird aufgeraeumt: 
     a) ASN Felder, netblock, domains are being fetched from DBs (RIPE,  ccTLDs,...)
      make a new column "as-we-think-your-constituency-is"
     b) fix TI DB constituency fields syntax checks!
     c) mail "as-we-think-your-constituency-is" field to CERTs, ask them to verify
     d) share relevant data with RIPE, let them add that to their DB.
	  add field: "this is the relevant CERT" or use the IRT object (link to it
      in stats.ripe.net?)

 
  !! Make document use-cases how CERTs , pentester, ... use contact lookups 
   which data sources exist?

  stat.ripe.net - can we (CERTs) have bulk access??




### active measurements done by the Atlas probes
 - build-in measurements (every probe does it):
   - its own network configuration information
   - current uptime, total uptime and uptime history
   - RTT (round trip time) measurements (on IPv4) to the first and second hops
     (think about the first two lines in your outgoing traceroutes)
   - ping measurements to a number of predetermined destinations
   - traceroute measurements to a number of predetermined destinations
   - DNS queries to root DNS servers (others to come)
   - SSL queries to a number of predetermined destinations

--> all these are very interesting! However, I still need to discover different
use cases for it. For example http://sniffmap.telcomap.org/# would be a nice
use-case for tracerout measurments or SSL queries can give us data such as how
good are the SSL certificates in a country?  BCP38 monitoring (T.Schreck idea)

Interesting: monitoring: are things getting better overall? Or: alarms if
something gets bad

But none of these topics have anything to do with contact lookup.


Concerning BCP38 measurement: there are concerns regarding measuring spoofed
sourc IPs. Question: what happens if this falls into wrong hands?  Do we have a
mechanism to clean this up? Should we measure this in the first place ?


 - user-defined measurement
   - ping/ping6
   - traceroute/traceroute6
   - DNS/DNS6
   - SSLCert/SSLCert6

--> same. 

There might be some measurements (build-in as well as user-defined) that are not public!


<commit_msg>abstract<commit_after>% Abuse Contact datasets for IT security incident handling
% L. <NAME> <<EMAIL>>; <NAME> <<EMAIL>>; <NAME> <<EMAIL>>, 
% 2014/09/18

# Abstract

This document aims at listing the different datasets listing abuse contacts and CERTs. The goal is to give the audience an overview of datasets and show where this data can be found, who has access to the data and list known issues with the data sets.

The authors accept git pull requests, please help and enrich this document.




# Terminology


| Term           | Explanation                                                            |
|:---------------|:-----------------------------------------------------------------------|
|CERT     		 | Computer Emergency Response Team. Synonymous with CSIRT |
|Constituency    | In this context, by constituency we mean the group of people, networks, ASNs, etc. a CERT addresses as their responsibility | 
|CSIRT    		 | Computer Security Incident Response Team  |
|FIRST			 | First.org - Forum for Incident Response and Security Teams |
|LIR      		 | Local Internet Registry; see (1) for more details  |
|Object			 | As defined by RIPE: a XXX RIPE: please define XXX |
|Network Resource | 	Or "Netresource". An ASN, netblock, IP address, domain name or hostname |
|Registrant 	 | 	A person registering a domain name for individual or business usage   |
|Registrar 		 | Often called domain registrar. An organisation handling the registration of domain names on behalf of a registrant  |
|RIPE NCC 		 | RIR serving Europe, Central Asia and the Middle East; see (2)  |
|RIR      		 | Regional Internet Registry; see (1) for more details  |
|RIS      		 | Routing Information Service  |
|TI       		 | Trusted Introducer  |

Table: Terminology


(1) [http://www.ripe.net/internet-coordination/internet-governance/internet-technical-community/the-rir-system](http://www.ripe.net/internet-coordination/internet-governance/internet-technical-community/the-rir-system)         
(2) [http://www.ripe.net](http://www.ripe.net)

# Intended Audience

CERTs and IT security incident abuse handlers as well as professionals working on automating IT security incident handling. 


# Problem description and motivation

CERTs - especially national CERTs - need to look up contact information very frequently for different resources on the Internet.  Examples of such lookups might be: given the IP address 1.2.3.4, give me the best matching abuse contact email address. Or given the domain www.example.com, what is the best contact for sending IT security incident notifications to?  

While the classical way to look up abuse contacts is to query some RIR whois service (such as RIPE's whois service) , there are some issues with this:
  * first of all, it might not be the data that's needed: for example the query might refer more to a domain instead of an IP address 
  * the contact registered for the specific network might be non-responsive or the wrong contact
  * bulk queries are usually not supported for whois
  * information in whois quickly gets outdated

As a consequence, a lot of different datasets for abuse contact lookups sprung up. They contain different data, have different access methods and -permissions.

This document aims at describing these different datasets in order to provide an overview. The document "abuse-lookups.mkd" deals with how to query these datasets.

This document is intended to be a request for input from the Internet community. The hope is to eventually define a standard for abuse contact lookups.



# General remarks and overview of datasets

For all potential datasets, there are two orthogonal approaches which must be considered: looking up abuse contacts for:

  * name based resources (e.g. domain names, host names)
  * number based resources (IP addresses, AS numbers)

This document assumes that a resource is either a name based resource or a number based resource. 


![Potential abuse contact lookups](./img/contact-lookup-flow.png)


Figure 1 describes possible paths in looking up abuse contact information based on the given resources and some of the datasets involved. 
Depending on the case and depending on the resource, there might be multiple paths for lookups. This will always be a case-by-case decision on behalf of the incident reporter. 

*Example:* the incident reporter finds a hacked webpage. Naturally, he will try to contact the domain owner (name based resource lookup) - the admin\_c and possibly also the tech\_c.  However, as time passes, the incident reporter notices more and more hacked webpages. Incidentally, they all turn out to be hosted on the very same IP address block.
After some investigations, the incident reporter decides that there must be something wrong with the web page hoster's servers. Hence, he decides to contact the owner of the IP block (the hoster) in order to hand him over a long CSV file of all web pages which are hacked and residing on the hosters' IP range.

In practice, often enough it is not clear, which path to choose. 

Sometimes, an incident report might want to contact a single point of contact (PoC) for a whole country. Typically this would be a national CERT which in turn will use some kind of contact lookup mechanism (possibly more accurate) to assign the incident report to a locally trusted and well known incident handler. For this case (sending incident reports to a national CERT), the report must find out the country a resource is assigned to. Since there is a well known dataset of national CERTs (see XXX), the problem thus can be reduced to finding the country code of the resource (for example: given an IP address 1.2.3.4, what is the country this IP address resides in?).
We will discuss this special case in section XXXX.

Whoever, we must remember that the problem of whom to contact is a choice which must be made on a case-by-case basis. Therefore this framework must be flexible enough to support both approaches.


# Existing datasets

%%% FIXME: document !## Perl's Net::Abuse

## Trusted Introducer

### Overview

Trusted Introducer
([http://trusted-introducer.org/](http://trusted-introducer.org/)) ("TI")
maintains a directory of IT security teams and TI members. While the database
is worldwide, there is a strong focus on Europe. TI assigns different maturity
levels to CERTs: listed, accredited and certified.  TI also offers a CSV Export
of its database to TI members (you need to be logged into the TI website to
access this CSV list).  The CSV list is searchable by netresources such as ASN,
country code or IP range.  However, at the time of writing this document, there
are still some issues with automating the querying of the TI data.

### Data in the dataset

The data in the TI database contains amongst other fields the following information:


  * Team name
  * TI Level (Listed, Accredited, ...)
  * URL of the team description on the TI webpage
  * Country
  * The constituency of the CERT:
    * ASN
    * netblocks
    * domains
  * Email Address(es) of the CERT
  * PGP key IDs of the CERT's team key
  * Telephone number
  * Postal address
  * Business hours
  * Time Zone
  * Team Representative
  * Website (URL) of the CERT

### Original or synthetic data?
This is an original dataset.

### Where to find it?

http://trusted-introducer.org/


### Who has access?

The public has access to a searchable list of teams via the website. 
TI members have access to the members view and can download a CSV file export of the database.

### APIs / automatic querying

Currently, a logged in user (TI member) can download a CSV file dump of the database in the internal section and query the (downloaded CSV) data automatically.

### Maintainers

Trusted Introducer.

### Liveliness

Trusted Intorducer confirms the data with each team representative every 12 months (XXX FIXME: check this XXX).


### Known issues






## FIRST.org

### Overview
FIRST (Forum for Incident Response and Security Teams) is one of the oldest IT security forums in the world. For a history, see http://first.org/about/history.
As such, FIRST maintains an extensive list of IT Security teams (FIRST's members) around the world. 



### Data in the dataset

The data in the FIRST directory contains amongst other fields the following information:

  * Team Info
    * Short team name
    * Team representative
    * Membership type
    * Date of establishment
    * Website (URL)
  * Constituency
    * Type of constituency
    * ASNs
    * Internet domain addresses
    * Country of constituency
  * Team contact information
    * Tel number
    * Email address
    * Postal address
    * Time zone
  * Business hours
    * Business hours
    * Reachability outside of business hours
  * Services
    * Textual description: reactive, proactive, ...
  * Cryptography
    * PGP key ID
    * PGP fingerprint
    * The full team PGP key (ascii armored)
  * Team members
    *  A list of names of team members


### Original or synthetic data?
Original data

### Where to find it?

http://www.first.org -> members

### Who has access?

Access is public 


### APIs / automatic querying

Currently at the time of this writing, there is no API and no way to automatically query data by netressources.

### Maintainers

FIRST.org Inc. maintains the database.

### Liveliness

FIRST members can update their data via web form.

### Known issues

The First directory is often lacking ASNs for FIRST team members.




## CERT.org's national CERT database

### Overview

CERT.org hosts a yearly national CERT (CSIRT) meeting([https://www.cert.org/incident-management/national-csirts/meeting/](https://www.cert.org/incident-management/national-csirts/meeting/). As part of these meetings, CERT.org also maintains a fairly accurate list of national CERTs ([https://www.cert.org/incident-management/national-csirts/national-csirts.cfm](https://www.cert.org/incident-management/national-csirts/national-csirts.cfm)).


Please note that the definition of "national CERT" is not 100% clear. 


### Data in the dataset

  * Organisation (i.e. the CERT)
  * Organisational Units
  * Groups
  * Personal data on CERT team members
  * Cryptography 
    * S/Mime or PGP or Kerberos infos
    * PGP key id
    *  
  

### Original or synthetic data?


Original 

### Where to find it?

https://nationalcsirts.cert.org/


### Who has access?

XXX FIXME: public? XXX

### APIs / automatic querying

XXX FIXME: CSV is public? XXX


### Maintainers


CERT.org 
CERT Coordination Center
Software Engineering Institute
Carnegie Mellon University
Office: +1 412 268 8876
CERT Hotline: +1 412 268 7090


### Liveliness

Somewhat up to date: the yearly national CSIRT conference helps in keeping the directory up to date.


### Known issues








## ENISAs CERT Directory

### Overview

ENISA collected a list of CERTs in Europe on their website. The list comes from TI as well as FIRST.

### Data in the dataset

XXX Cosmin, please add XXX

### Original or synthetic data?

Synthetic.

### Where to find it?


https://www.enisa.europa.eu/
and:
https://www.enisa.europa.eu/activities/cert/background/inv

(needs Javascript)

### Who has access?

Public.


### APIs / automatic querying

There is no API, but the data can be downloaded (XML file) at: XXX Cosmin, please add XXX

### Maintainers

ENISA 

### Liveliness

XXX Cosmin, please add XXX

### Known issues

XXX Cosmin, please add XXX








## CERT.at's national CERT database
CERT.at used this national CERT database and created an online lookup tool at [https://contacts.cert.at](https://contacts.cert.at).


### Overview

### Data in the dataset

### Original or synthetic data?

### Where to find it?

### Who has access?

### APIs / automatic querying

### Maintainers

### Liveliness

### Known issues








## RIPE database

( with a special focus on abuse-c, IRT object)

XXX Mirjam , please add/change/delete XXX

Following a proposal made by the RIPE community in XXX the RIPE policy ripe-563 made it mandatory for every resource object (inetnum, inet6num and aut-num) to have a dedicated abuse contact. This contact is usually referred to as "abuse-c" contact and it is the preferred way to report any form of abuse to.

The IRT (incident response team) object was created with a similar idea in mind but its deployment is even being around for years not very low and so is it is not widely used. Chances are high that this object will be dismissed. Abuse-c is considered to superceed the IRT object.


/XXX Mirjam , please add/change/delete XXX


### Overview

### Data in the dataset

### Original or synthetic data?

### Where to find it?

### Who has access?

### APIs / automatic querying

### Maintainers

### Liveliness

### Known issues






# Supporting databases

Often enough, there is a requirement to transform one network resource (e.g. an IP address) into a different network ressource (e.g. a country code) in order to look up the abuse contact (e.g. the national CERT) for the IP address. 
This section describes these supporting additional databases.



## Country Code lookups

As discussed in section 'General remarks on abuse contact lookups', some incident reports should simply go to the national CERT. For this task, it is important to find the country code of an IP address or a domain.

In the case of IP addresses, there are a couple of databases which will be discussed below.

### RIPE DB

Ripe stat offers a RESTful API  for XXX Christian / Mirjam, please extend XXX

### Maxmind

[Maxmind](https://www.maxmind.com/) is one of the most well known databases for IP to country code mappings. 
There are a couple of versions of Maxmind with varying degrees of accuracy and costs: 

* GeoIP City
* GeoIP Country
* GeoIP Lite
* GeoIP Legacy

We advise the reader to check Maxminds webpage for current offerings.

Using the GeoIP database in a wide variety of languages is very straightforward. In this document we give an example in Python (from [Readthedocs](http://geoip2.readthedocs.org/en/latest/)).

```
>>> import geoip2.database
>>> reader = geoip2.database.Reader('/path/to/GeoLite2-City.mmdb')
>>> response = reader.city('8.8.8.8')
>>> response.country.iso_code
'US'
```




### Team Cymru

[Team Cyrmu](https://www.team-cymru.org/Services/ip-to-asn.html) provides an IP address to ASN mapping service via whois ([RFC3912](https://tools.ietf.org/html/rfc3912)) as well as via netcat, http(s) or DNS based lookups. As part of this lookup mechanism, there are options to also get the country code of a given IP address. To the best of the authors knowledge, the IP to country mapping is done via the RIR databases.

*Example lookup*: 
```
$ whois -h whois.cymru.com  " -c   8.8.8.8"

AS      | IP               | CC | AS Name
15169   | 8.8.8.8          | US | GOOGLE - Google Inc.,US
```

or as a shell script:
```
#!/bin/bash

ip=$1
cmd=$(cat <<EOT;
begin
verbose
 -c
$ip
end
EOT)

echo "$cmd" | nc whois.cymru.com 43

```
Example output:

```
Bulk mode; whois.cymru.com [2014-05-12 19:30:14 +0000]
Error: no ASN or IP match on line 3.
15169   | 8.8.8.8          | 8.8.8.0/24          | US | arin     |            | GOOGLE - Google Inc.,US
```


## Shortcomings of existing IP to country lookups

RIPE country code lookups usually return the country of registration of the organisation which was assigned the specific netblocks. However, this is not necessarily the same country as the individual IP address. This is especially true with international organisations or Internet Service Providers.

On the other hand, Maxmind data might not find to correct country of an IP address or even return the special country codes 'A1' or 'A2' for satellite uplinks. 

For the purpose of delivering abuse reports, finding the country of registration might be  arguably a better approach in many cases.


# Shortcomings of existing databases



## RIPE NCC Data Sets 

RIPE NCC has a list of publicly available data sets that could be useful for CERTs:
  * Registration data:
    * RIPE DB (containing IP resource information and contact data)
    * Routing DB (containing routing policy)
    * Regional Internet Registry Statistics (which IP resources are allocated when and to whom)
  * Routing data collected by the 13 Routing Information Service (RIS) route collectors (https://www.ripe.net/data-tools/stats/ris/)
  * Active measurements done by the RIPE Atlas (https://atlas.ripe.net)
   

The following sections look at these data sets in more detail. All data sets described here 
are publicly avilable and are made accessible via various interfaces.

### RIPEstat (MIRJAM: Maybe move after Registration data to be consistent with list above?)

RIPEstat is a web-based interface that provides everything you ever wanted to know about 
IP address space, Autonomous System Numbers (ASNs), and related information for hostnames 
and countries in one place. It has multiple APIs ranging from a raw data centric Data API to a 
visualisation rich Widget API.
For documentation and a description of all the APIs:

https://stat.ripe.net/index/documentation/interfaces-apis

In the listing of the following data sets it will be mentioned if the data can be accessed
via RIPEstat.



### Registration Data

MIRJAM: Include a sentence describing what The RIPE registry means and entails and refer to ripe-508.

#### RIPE DB

Christian, 20014-05-08: Status = needs review

The RIPE Database contains registration information for networks in the RIPE NCC service region. It shows the organisations that hold the resources, the country the organisation is registered in, and contact details for the networks.
  
The Database can be searched by using the web interface (https://apps.db.ripe.net/search/query.html) or by directing your whois client to whois.ripe.net. A full-text search is also available 
https://apps.db.ripe.net/search/full-text.html.
  
FAQ: http://www.ripe.net/data-tools/db/faq
Documentation: http://www.ripe.net/data-tools/support/documentation 
  
This data is accessible via RIPEstat as a widget and as a data call.
Widget API: https://stat.ripe.net/widget/registry-browser
Data API: https://stat.ripe.net/docs/data\_api#RegistryBrowser
  
MIRJAM: I suggest to take out all the Q's and A's and format these sections with sub headings (the same for each of these sections). Might be better readable.

##### Source of the data:
- The data is created and maintained by the RIPE NCC and by the resource holders following a defined scheme for objects and attributes (MIRJAM: What does this mean: "scheme..."?).

##### Quality of the data and how often is the quality checked:
- The data quality in the RIPE registry is very important for network operations and it one of the main tasks of the RIPE NCC. The RIPE NCC is frequently cecking the quality of the data and is working with the maintainers of the data to keep it up to date.

##### Data set available in batch:
- RIPE Database APIs can be queried in batch but there is a limit if personal data is being fetched:
	- Number of personal data sets returned in queries from an IP address - 1000 per 24 hours (from "Accetable Use Policy" http://www.ripe.net/data-tools/support/documentation/aup)

This limit also applies for using RIPEstat but it is not per IP address per user account.

##### API for batch query interface:
- The command line interface (port 43) or RIPEstat can be used.

#### Historical data:
- The RIPE Database interfaces provide a historical lookup feature based on versions, see https://labs.ripe.net/Members/kranjbar/proposal-to-display-history-of-objects-in-ripe-database
for details.
In RIPEstat the data call for the Registry-Browser returns historical information for each queried object but this requires the user to be logged in as a RIPE NCC member.

Notes:
  --> interesting for CERTs: whowas service
    --> should exist in stat.ripe.net 

Christian: 
	 Whowas service I could only find for ARIN. For the RIPE DB this is basically
	 the solution with versions (mentioned previous).
	 I'm trying to find out if other RIRs provide similar services. Test!
:
    


#### Routing Registry

The RIPE Routing Registry is a sub-set of the RIPE Database and holds information about routing on the Internet. The routing information is stored in Autonomous System (AS) objects. The information in these AS objects show how a particular network is routed on the internet. 

The RIPE Routing Registry is part of the Internet Routing Registry (http://www.ripe.net/data-tools/db/the-internet-routing-registry-history-and-purpose), a collection of databases that mirror each other.

Sample output
```
route:           192.0.2.0/24
descr:           TEST-NET-1
origin:          ASXXXX
mnt-by:          ASXXXX-MNT
mnt-by:          YYYYY-LIR-MNT
source:          RIPE # Filtered
```

##### Data set available in batch

Same terms and conditions apply as for RIPE Database.

##### API for batch query interface

Same terms and conditions apply as for RIPE Database.

##### Historical data

See RIPE DB section (to be checked if history is available for route objects as well) 




#### Regional Internet Registry (RIR) statistics 

The RIR statistics provide an authoritative source of the status of all IP resources. It is maintained collectively by all RIRs. They are intended to provide a snapshot of the status of Internet number resources, without any transactional or historical details. (MIRJAM: Christian, I am not sure we need to sepcify this here. The most important fact is that this is the authoritative source for all IP resources, but that it doesn't provide a lot of details).

The statistics are primarily published as a text file by each RIR once a day. The statistics are usually available before 23:59:59 in the local time zone the RIR is in.

The published statistics follow a standardised format which contains a header and a body.
The header consists of a version line and a summary line.

The format of the version line is:

`version|registry|serial|records|startdate|enddate|UTCoffset`

| Field      | Description                                                               |
|:-----------|---------------------------------------------------------------------------| 
| version    | format version number of this file, currently 2                           |
| registry   | name of the producing RIR                                                 |
| serial     | serial number of this file (within the creating RIR series)               |
| records    | number of records in file, excluding blank lines, summary lines,          |
|            | the version line and comments                                             |
| startdate  | start date of time period yyyymmdd format                                 |
| enddate    | end date of period in yyyymmdd format                                     |
| UTCoffset  | offset from UTC (+/- hours) of local RIR producing file                   |


The format of the summary line is:

`registry|*|type|*|count|summary`

| Field      | Description 
|:-----------|---------------------------------------------------------------------------|
| registry   | name of the producing RIR                                                 |
| *          | an ASCII \* (unused field, retained for spreadsheet purposes)             |
| type       | as defined in the format used for a record in the body                    |
| count      | sum of the number of record lines of this type in the file                |
| summary    | the ASCII string 'summary' (to distinguish a record line in the body)     |

A record in the body follows this format:


`registry|cc|type|start|value|date|status[|extenstions...]`

| Field       | Description                                                              |
|:------------|--------------------------------------------------------------------------|
| registry    | name of the producing RIR                                                |
| cc          | ISO 3166 2-letter code of the organization to which the allocation or    |
|             | assignment was made                                                      |
| type        | Type of Internet number resource represented in this record. One value   | 
|             | from the set of defined strings:                                         |
|             |   `{asn, ipv4, ipv6}`                                                    |
| start       | In the case of records of type 'ipv4' or 'ipv6' this is the IPv4 or IPv6 |
|             | 'first address' of the range.                                            |
|             | In the case of an 16 bit AS number the format is the integer value in    |
|             | the range 0 to 65535, in the case of a 32 bit ASN the value is in the    |
|             | range 0 to 4294967296. No distinction is drawn between 16 and 32 bit ASN |
|             | values in the range 0 to 65535.                                          | 
| value       | In the case of IPv4 address the count of hosts for this range. This      |
|             | count does not have to represent a CIDR range.                           |
|             | In the case of an IPv6 address the value will be the CIDR prefix length  |
|             | from the 'first address' value of \<start\>.                             |
|             | In the case of records of type 'asn' the number is the count of AS from  |
|             | this start value.                                                        |
| date        | Date on this allocation/assignment was made by the RIR in the format     |
|             | yyyymmdd.                                                                |
|             | Where the allocation or assignment has been transferred from another     |
|             | registry, this date represents the date of first assignment or           |
|             | allocation as received in from the original RIR.                         |
|             | It is noted that where records do not show a date of first assignment,   |
|             | this can take the 0000/00/00 value.                                      |
| status      | Type of allocation from the set:                                         |
|             |  `{allocated, assigned}`                                                 |
|             | This is the allocation or assignment made by the registry producing the  |
|             | file and not any sub-assignment by other agencies.                       |
| extensions  | Any extra data on a line is undefined, but should conform to use of the  |
|             | field separator used above.                                              |


The files are primarily accessible via each RIR FTP site:

| RIR         | FTP site                                                                 |
|:------------|--------------------------------------------------------------------------|
| AFRINIC     | ftp://ftp.afrinic.net/pub/stats/afrinic/                                 |
| APNIC       | ftp://ftp.apnic.net/pub/stats/apnic/                                     |
| ARIN        | ftp://ftp.arin.net/pub/stats/arin/                                       |
| LACNIC      | ftp://ftp.lacnic.net/pub/stats/lacnic/                                   |           
| RIPE NCC    | ftp://ftp.ripe.net/pub/stats/ripencc/                                    |

The general file name has this format:

`delegated-<registry>-yyyymmdd`

Data compression is optional. If compressed, the normal file suffix is used to denote the compression algorithm (.gz, .bz2, .zip etc).
The most recent file must be available in a non-compressed form and has this file name:

`delegated-<registry>-latest`

There is not obligation on any registry to retain previous files, once a new file is available for public access but RIR usually keep an archive of previous files. 

##### Note on the data quality and accuracy

The RIR statistics are based on registration data that is maintained by the RIR itself which usually guarantees high data quality.
The country code that is stated with the allocation/assignment is based on the location of the organisation holding the resource at the time the request was issued. It is not guaranteed that this location is kept up-to-date.
Keep in mind that the nature of this data is registration related and not usage related. For usage related location information other providers are more suitable e.g. MaxMind GeoIP database.


##### RIPEstat

The RIR statistics are not directly retrievable via RIPEstat but there is a data call that provides a list of Internet number resources (IPv4, IPv6 and ASNs) per country:

https://stat.ripe.net/docs/data_api#CountryResourceList

E.g.: 
Internet number resources for Germany
`https://stat.ripe.net/data/country-resource-list/data.json?resource=de`

### Routing Data 

#### What is it?

Internet routing data is significantly different from the registration data as the routing data shows the state of routing tables. It has to be noted that this is not a single routing table as each point from where routing data gets collected adds its distinct vantage point.

The RIPE NCC collects and stores Internet routing data since 2001 using RIS which is short for Routing Information Service. The RIS system consists of a network of route collectors (RRC) which are running a customised routing software (Quagga) to passively collect routing information from other networks that are connected to the RRC (peering).

RIS: http://www.ripe.net/data-tools/stats/ris/

#### Fetching The Data

The RIS routing data can be accessed through the raw data files or via APIs the RIPE NCC provides.

##### Raw Data

The raw data files are a direct copy of the data that is being sent by the RRCs to a central storage location. These files come in two flavour, dumps and updates. 
Dumps contain the entire routing table for the time the dump was created which is 3 times a day (every 8 hours) starting from midnight (00:00, 08:00 and 16:00 UTC).
Updates contain BGP updates (announcements and withdrawals) aggregated over 5 minutes. 

The links to the data files can be found on http://www.ripe.net/data-tools/stats/ris/ris-raw-data. The URL follows a simple pattern:

`http://data.ris.ripe.net/<rrc_#>/` (e.g. http://data.ris.ripe.net/rrc01/ to access dumps and updates for RRC01)

Dumps as well as updates are stored in MRT format which can be read by libbgdump, a library written in C by <NAME>. More details on this library and where to download can be found on the page mentioned above.
On the same page there is also a Python library state, PyBGPdump. Unfortunately this library has been reported to fail for some dump files. If Python support is necessary mrtparse (https://github.com/YoshiyukiYamauchi/mrtparse) might be an alternative.


##### APIs

RIPEstat offers a couple of data calls which are based on the RIS routing data. This form of access is usually more convenient than having to extract the information from the raw data files.


* 	Resource-Overview (Prefix-Overview, As-Overview)
	This data call gives a summary of the given prefix, including whether and by whom it is announced.
	
	Documentation: 
	+ 	https://stat.ripe.net/docs/data_api#PrefixOverview, 
	+ 	https://stat.ripe.net/docs/data_api#AsOverview


* 	Routing-Status
	This data call returns a summary of the current BGP routing state of a given IP prefix or ASN, as observed by the RIS route collectors.
	
	Documentation: 
	+ 	https://stat.ripe.net/docs/data_api#RoutingStatus


* 	Routing-History
	This data call shows the history of announcements for prefixes, including the origin ASN and the first hop.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#RoutingHistory

	
* 	Asn-Neighbours
	This data call shows information on the network neighbours for a given ASN. This includes statistical information as well as the list of observed ASN neighbours.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#AsnNeighbours

	
* 	Asn-Neighbours-History
	This data call returns information about neigbouring ASNs of a queried ASN extended with history.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#AsnNeighboursHistory


* 	BGPlay
	This data call represents the scenario of what occurred to the BGP routes of a resource over a period of time.
	It includes data that defines the initial BGP state at the start time of the query, and all the BGP updates observed from then until the end time, as well as a description of all the AS nodes, and RIS BGP peers involved in the result.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#BGPlay
	

* 	Country-Routing-Statistics
	This data call returns statistics on Internet resources for a country - this includes:

	+	number of ASNs seen in routing data and registration data
	+	number of prefixes in routing data and registration data (split into IPv4 and IPv6)
	+	amount of IPv4 space seen in routing data as well as registration data
	The results can be restricted to a specific time period as well the granularity is variable but can be set explicitly.
	
	Documentation:
	+ 	https://stat.ripe.net/docs/data_api#CountryResourceStats

For a general introduction on how to use the RIPEstat API please refer to https://stat.ripe.net/docs/data_api.

The query model that is used at the moment on RIPEstat supports queries for single resource at a time which could make it less convenient for a larger number of resource to be looked up. An already requested feature is to have also support for bulk request which will be added in the upcoming development sprints for RIPEstat.

A somehow similar but a bit more limited service is provided by RISwhois, http://www.ripe.net/data-tools/stats/ris/riswhois. The benefit of RISwhois is its performance.


#### 

### External Data Sets Served Via RIPEstat

Then there are other data sets that we provide via RIPEstat but they don't originate from the RIPE NCC, 
like MaxMind's GeoIP data set, some blacklists... the benefit can be a consistent data interface.


RIPEstat?

  --> geolocation. Quelle? Bitte genau genau dokumentieren! stats file ist die
  org, an die das vergeben wurde.  As CERTs we often have the problem that we
  need to deliver reports to national CERTs. So the question of proper
  geolocation comes immediately.  Maxmind seems to be the de-facto standard.
  However, RIPE data is used as a basis for cymru IP-to-ASN.

  We should document which geolocation solutions exist, how precise they are
  and when to use which.  Sometimes an IP address is in a country A, the RIPE
  entry for it points to an organisation in country B, maxmind will say it is
  country C.

  !!! super idee!!! geolocation history DB "wherewas?"
  organisational country vs. ip address is in use in country X

  --> google or sky hook wireless geolocation? Is it possible to get that data?
  maybe Apple as well?
  XXX insert CCC talk XXX

  traceroute history of ATLAS? did something change?

  Vorschlag: 
   1) TI DB wird aufgeraeumt: 
     a) ASN Felder, netblock, domains are being fetched from DBs (RIPE,  ccTLDs,...)
      make a new column "as-we-think-your-constituency-is"
     b) fix TI DB constituency fields syntax checks!
     c) mail "as-we-think-your-constituency-is" field to CERTs, ask them to verify
     d) share relevant data with RIPE, let them add that to their DB.
	  add field: "this is the relevant CERT" or use the IRT object (link to it
      in stats.ripe.net?)

 
  !! Make document use-cases how CERTs , pentester, ... use contact lookups 
   which data sources exist?

  stat.ripe.net - can we (CERTs) have bulk access??




### active measurements done by the Atlas probes
 - build-in measurements (every probe does it):
   - its own network configuration information
   - current uptime, total uptime and uptime history
   - RTT (round trip time) measurements (on IPv4) to the first and second hops
     (think about the first two lines in your outgoing traceroutes)
   - ping measurements to a number of predetermined destinations
   - traceroute measurements to a number of predetermined destinations
   - DNS queries to root DNS servers (others to come)
   - SSL queries to a number of predetermined destinations

--> all these are very interesting! However, I still need to discover different
use cases for it. For example http://sniffmap.telcomap.org/# would be a nice
use-case for tracerout measurments or SSL queries can give us data such as how
good are the SSL certificates in a country?  BCP38 monitoring (T.Schreck idea)

Interesting: monitoring: are things getting better overall? Or: alarms if
something gets bad

But none of these topics have anything to do with contact lookup.


Concerning BCP38 measurement: there are concerns regarding measuring spoofed
sourc IPs. Question: what happens if this falls into wrong hands?  Do we have a
mechanism to clean this up? Should we measure this in the first place ?


 - user-defined measurement
   - ping/ping6
   - traceroute/traceroute6
   - DNS/DNS6
   - SSLCert/SSLCert6

--> same. 

There might be some measurements (build-in as well as user-defined) that are not public!



{'fraction_non_alphanumeric': 0.07358986454612844, 'fraction_numerical': 0.0068063885706584, 'mean_word_length': 2.8667917448405253, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 69, 'https://': 64, 'lorem ipsum': 0, 'www.': 44, 'xml': 0}, 'pii_count': 16, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 92, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3049033', 'n_tokens_mistral': 20287, 'n_tokens_neox': 18747, 'n_words': 9794}