TITLE: The MIT Supercloud Dataset 

ABSTRACT: Artificial intelligence (AI) and Machine learning (ML) workloads are an
increasingly larger share of the compute workloads in traditional
High-Performance Computing (HPC) centers and commercial cloud systems. This has
led to changes in deployment approaches of HPC clusters and the commercial
cloud, as well as a new focus on approaches to optimized resource usage,
allocations and deployment of new AI frame- works, and capabilities such as
Jupyter notebooks to enable rapid prototyping and deployment. With these
changes, there is a need to better understand cluster/datacenter operations
with the goal of developing improved scheduling policies, identifying
inefficiencies in resource utilization, energy/power consumption, failure
prediction, and identifying policy violations. In this paper we introduce the
MIT Supercloud Dataset which aims to foster innovative AI/ML approaches to the
analysis of large scale HPC and datacenter/cloud operations. We provide
detailed monitoring logs from the MIT Supercloud system, which include CPU and
GPU usage by jobs, memory usage, file system logs, and physical monitoring
data. This paper discusses the details of the dataset, collection methodology,
data availability, and discusses potential challenge problems being developed
using this data. Datasets and future challenge announcements will be available
via https://dcc.mit.edu.
Artificial intelligence (AI) and Machine learning (ML) workloads are an
increasingly larger share of the compute workloads in traditional
High-Performance Computing (HPC) centers and commercial cloud systems. This has
led to changes in deployment approaches of HPC clusters and the commercial
cloud, as well as a new focus on approaches to optimized resource usage,
allocations and deployment of new AI frame- works, and capabilities such as
Jupyter notebooks to enable rapid prototyping and deployment. With these
changes, there is a need to better understand cluster/datacenter operations
with the goal of developing improved scheduling policies, identifying
inefficiencies in resource utilization, energy/power consumption, failure
prediction, and identifying policy violations. In this paper we introduce the
MIT Supercloud Dataset which aims to foster innovative AI/ML approaches to the
analysis of large scale HPC and datacenter/cloud operations. We provide
detailed monitoring logs from the MIT Supercloud system, which include CPU and
GPU usage by jobs, memory usage, file system logs, and physical monitoring
data. This paper discusses the details of the dataset, collection methodology,
data availability, and discusses potential challenge problems being developed
using this data. Datasets and future challenge announcements will be available
via https://dcc.mit.edu.
The MIT Supercloud Dataset
http://arxiv.org/abs/2108.02037v1