<commit_before>---
title: "Resource Utilization Aware Job Scheduling to Mitigate Performance Variability"
collection: publications
permalink: /publication/2021-05-resource-utilization-aware-job-scheduling
excerpt: "Resource contention on high performance computing (HPC) platforms can lead to significant variation in application performance. Large variations in run-times can lead to less efficient use of system resources as many jobs experience contention on shared resources. It can also lead to users over-estimating their job's expected run-time, which degrades the efficiency of the system scheduler. Mitigating significant performance variation on HPC platforms enables more efficient use of their resources. In this paper, we present a pipeline for collecting and analyzing system and application performance data for jobs submitted over long periods of time. We use a set of machine learning (ML) models trained on this data to classify performance variation using current system counters. Additionally, we present a new resource-aware job scheduling algorithm that utilizes the ML pipeline and current system state to mitigate job variation. We evaluate our pipeline, ML models, and scheduler using various proxy applications and an actual implementation of the scheduler on an Infiniband-based fat-tree cluster."
date: 2022-05-30
venue: "IPDPS '22"
paperurl: 'https://pssg.cs.umd.edu/assets/papers/2022-05-rush-scheduler.pdf'
citation: '<NAME>, <NAME>, <NAME>, <NAME>, <NAME>, “Resource Utilization Aware Job Scheduling to Mitigate Performance Variability”, International Parallel & Distributed Processing Symposium (IPDPS). 2022.'
---

To appear in IPDPS '22.
<commit_msg>fix date<commit_after>---
title: "Resource Utilization Aware Job Scheduling to Mitigate Performance Variability"
collection: publications
permalink: /publication/2022-05-resource-utilization-aware-job-scheduling
excerpt: "Resource contention on high performance computing (HPC) platforms can lead to significant variation in application performance. Large variations in run-times can lead to less efficient use of system resources as many jobs experience contention on shared resources. It can also lead to users over-estimating their job's expected run-time, which degrades the efficiency of the system scheduler. Mitigating significant performance variation on HPC platforms enables more efficient use of their resources. In this paper, we present a pipeline for collecting and analyzing system and application performance data for jobs submitted over long periods of time. We use a set of machine learning (ML) models trained on this data to classify performance variation using current system counters. Additionally, we present a new resource-aware job scheduling algorithm that utilizes the ML pipeline and current system state to mitigate job variation. We evaluate our pipeline, ML models, and scheduler using various proxy applications and an actual implementation of the scheduler on an Infiniband-based fat-tree cluster."
date: 2022-05-30
venue: "IPDPS '22"
paperurl: 'https://pssg.cs.umd.edu/assets/papers/2022-05-rush-scheduler.pdf'
citation: '<NAME>, <NAME>, <NAME>, <NAME>, <NAME>, “Resource Utilization Aware Job Scheduling to Mitigate Performance Variability”, International Parallel & Distributed Processing Symposium (IPDPS). 2022.'
---

To appear in IPDPS '22.

{'fraction_non_alphanumeric': 0.05943536404160475, 'fraction_numerical': 0.01664190193164933, 'mean_word_length': 5.473076923076923, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5044958', 'n_tokens_mistral': 836, 'n_tokens_neox': 763, 'n_words': 423}