task_categories:feature-extraction size_categories:10M<n<100M language:en license:apache-2.0 certificates machine identity security region:us
---
license: apache-2.0
task_categories:
- feature-extraction
language:
- en
tags:
- certificates
- machine identity
- security
size_categories:
- 10M<n<100M
pretty_name: Machine Identity Spectra Dataset
configs:
  - config_name: sample_data
    data_files: Data/CertificateFeatures-sample.parquet
---

# Machine Identity Spectra Dataset
<img src="https://huggingface.co/datasets/Venafi/Machine-Identity-Spectra/resolve/main/VExperimentalSpectra.svg" alt="Spectra Dataset" width="250">

## Summary
Venafi is excited to release of the Machine Identity Spectra large dataset. 
This collection of data contains extracted features from 19m+ certificates discovered over HTTPS (port 443) on the 
public internet between July 20 and July 26, 2023.
The features are a combination of X.509 certificate features, RFC5280 compliance checks, 
and other attributes intended to be used for clustering, features analysis, and a base for supervised learning tasks (labels not included).
Some rows may contain nan values as well and as such could require additional pre-processing for certain tasks.

This project is part of Venafi Athena. Venafi is committed to enabling the data science community to increase the adoption of machine learning techniques 
to identify machine identity threats and solutions. 
Phillip Maraveyias at Venafi is the lead researcher for this dataset. 

## Data Structure
The extracted features are contained in the Data folder as certificateFeatures.csv.gz. The unarchived data size is 
approximately 10GB and contains 98 extracted features for approximately 19m certificates. A description of the features
and expected data types is contained in the base folder as features.csv.

The Data folder also contains a 500k row sample of the data in parquet format. This is  displayed in the Data Viewer 
for easy visual inspection of the dataset.

## Clustering and PCA Example

To demonstrate a potential use of the data, clustering and Principal Component Analysis (PCA) were 
conducted on the binary data features in the dataset. 10 clusters were generated and PCA conducted with the top 3 components preserved.

KMeans clustering was performed to generate a total of 10 clusters. In this case we are primarily 
interested in visualizing the data and understanding better how it may be used, so the choice of 10 clusters is mostly
for illustrative purposes.

The top three PCA components accounted for approximately 61%, 10%, and 6% of the total explained variance
(for a total of 77% of the overall data variance). Plots of the first 2 components in 2D space and top 3 components in 
3D space grouped into the 10 clusters are shown below.

### Clusters in 2 Dimensions
![](ClusterAnalysis/clusters2d.png)

### Clusters in 3 Dimensions
![](ClusterAnalysis/clusters3d.png)

## Contact
Please contact athena-community@venafi.com if you have any questions about this dataset.

## References and Acknowledgement
The following papers provided inspiration for this project:
- Li, J.; Zhang, Z.; Guo, C. Machine Learning-Based Malicious X.509 Certificates’ Detection. Appl. Sci. 2021, 11, 2164. https://doi.org/ 10.3390/app11052164
- Liu, J.; Luktarhan, N.; Chang, Y.; Yu, W. Malcertificate: Research and Implementation of a Malicious Certificate Detection Algorithm Based on GCN. Appl. Sci. 2022,12,4440. https://doi.org/ 10.3390/app12094440