<commit_before>---
layout: article
title: "AIOps"
date:
modified:
excerpt: "excerpt"
tags: []
image:
  feature:
  teaser:
  thumb:
ads: false
toc: true
---  

<p>
<a href="http://jorge-cardoso.github.io/research/pdf_img/iforesight3.png">
<img src="http://jorge-cardoso.github.io/research/pdf_img//iforesight3.png" height="25%" width="25%" style="float:left;margin:0 20px 0 0;"/>
</a>
</p>


## Introduction

In planet-scale deployments, the Operation and Maintenance (O&M) of cloud platforms cannot be done any longer 
manually or simply with off-the-shelf solutions. It requires self-developed automated systems, ideally exploiting 
the use of AI to provide tools for autonomous cloud operations. We rely on deep learning, 
distributed traces, and time-series analysis (sequence analysis) to effectively detect and fix anomalous 
cloud infrastructure behaviors during operations to reduce the workload of human operators. 

The iForesight system is being used to evaluate new O&M approaches. iForesight 3.0 is the result of 2 years 
of research with the goal to provide an intelligent new tool aimed at SRE cloud maintenance teams. It enables 
them to quickly detect and predict anomalies thanks to the use of artificial intelligence when cloud services 
are slow or unresponsive. 

## Problem
Existing tools for monitoring IT infrastructures, networks and applications focus on collecting logs, metrics, 
events, and traces from distributed systems mainly for visualization. Nonetheless, the final goal of monitoring 
is to reach a level of technological development where we have tools that conduct root cause analysis with a high 
accuracy and enable to autonomously recover systems. To achieve this goal, we still need to shift from a data 
collection stage to an insight- and action-driven paradigm. One promising path to monitor planet- and large-scale 
platforms is to rely on advanced analytics and explore techniques from statistics, time-series analysis, data mining, 
natural language processing, graph processing, machine learning, and deep learning to extract insights from large 
volumes of monitoring data to support and drive recovery actions. 

## Approach
The mission of the **Intelligent Cloud Operations** SRE team (based in [Munich](https://www.muenchen.de/int/en.html), 
Germany) is to develop new systems and tools to analyze observability data 
from [Huawei Cloud](https://www.huaweicloud.com/en-us/about/about_us.html)
to detect impact to customers, identify the root cause within seconds, and fix the problem using 
the 1/5/10 rule (detection: 1 min, RCA: 5 min, recovery: 10 min).

The following figure from [Gartner](https://www.gartner.com/en) provides a high level architecture of the system 
we are building highliting the main areas of concern: 
+ Real-time streaming and historical data, observations, 
[Big Data](https://en.wikipedia.org/wiki/Big_data), 
Machine Learning (ML), 
[Anomaly Detection](https://en.wikipedia.org/wiki/Anomaly_detection), 
[Root-Cause Analysis](https://en.wikipedia.org/wiki/Root_cause_analysis), 
[Performance Analysis](https://en.wikipedia.org/wiki/Application_performance_management), 
[Predictive Maintenance](https://en.wikipedia.org/wiki/Predictive_maintenance), and 
[Automation](https://en.wikipedia.org/wiki/Robotic_process_automation). 
<p>
<img src="https://blogs.bmc.com/wp-content/uploads/2019/02/AIOpsGraphicGartner2018-768x408.png" height="408" width="768" style="float:center;margin:0 20px 0 0;"/>
</p>

The use of ML for production engineering can support the development of new approaches for: 
1. Monitoring and alerting
2. Anomaly detection and Root Cause Analysis
3. Capacity planing and prediction
4. Canarying validation
5. Service Scaling
6. Operational performance

Our work focuses on points 1) and 2). 

In 2017 we adopted AI in the form of [Data Science](https://en.wikipedia.org/wiki/Data_science) and 
[Machine Learning](https://en.wikipedia.org/wiki/Machine_learning) approaches for anomaly detection, 
root-cause analysis, fault prediction, and automated recovery into our suite. 

These techniques, including **statistical learning**, **time-series analysis**, **deep learning**, **big data**,
**streaming**, and **data visualization**, enabled us to develop new production-ready services for troubleshooting 
Huawei Cloud and detect issues which were previously undetectable.

## AIOPs platform
An AIOps platform architecture consists of functional layers such as:

1. *Big Data processing*. Real-time processing of streaming and historical data.
2. *Data pipeline*. Connected data processing elements ingesting data from multiple sources.
3. *Library of algorithms*: Statistical functions, classical machine learning, and deep learning algorithm.
4. *Automation*. Use runbooks and RPA technology to automate repetitive tasks.
5. *User interface*. Allows IT operations teams t interact with the platform and quickly identify issues and apply corrective actions.

Our work focuses on points 1)-3). 

## Challenges

The challenges of operationalising AI are not limited to the understanding of deep learnign or machine learning algorithms.
Major challenges are related with software engineering, access and processing of large amounts of distributed data, 
model management, updating, deleting and training models on specialized GPUs and hardware, composition of workflows
for orchestrating parallel jobs, and the visual management of models, workflows, and results. 

## Huawei Cloud
Our cloud has planet-scale technical requirements with an 
[microservices](https://en.wikipedia.org/wiki/Microservices) architecture composed of hundreds of services.
They are distributed over thousands of hosts in many geographical regions and operate with an availability 
higher than [five nines](https://en.wikipedia.org/wiki/High_availability). 

Our system monitors Huawei Cloud which is build on top of [OpenStack](https://docs.openstack.org/), an 
opensource cloud operating system. OpenStack controls large pools of compute, storage, and networking
resources throughout tens of datacenters. The base services are predominantly written in Python and Java 
running on Linux. 

Huawei Cloud is one of the largest and fastest growing platforms in the world. 
It has a strong presence throughout the world with over 40 availability zones located across 23 geographical regions,
ranging from Germany, France, South/Central America, Hong Kong and Russia to Thailand and South Africa.

There are three properties that make platforms such as Huawei Cloud far more difficult to monitor and troubleshoot:
1. Amount of data and relationships which O&M teams need to analyze.
2. Due to its distributed nature and complexity, system data has a low signal to noise ratio.
3. Since many different subsystems interact together, semantically reconciliating data is difficult.
<commit_msg>Update iforesight.md<commit_after>---
layout: article
title: "AIOps"
date:
modified:
excerpt: "excerpt"
tags: []
image:
  feature:
  teaser:
  thumb:
ads: false
toc: true
---  

<p>
<a href="http://jorge-cardoso.github.io/research/pdf_img/iforesight3.png">
<img src="http://jorge-cardoso.github.io/research/pdf_img//iforesight3.png" height="25%" width="25%" style="float:left;margin:0 20px 0 0;"/>
</a>
</p>


## Introduction

In planet-scale deployments, the Operation and Maintenance (O&M) of cloud platforms cannot be done any longer 
manually or simply with off-the-shelf solutions. It requires self-developed automated systems, ideally exploiting 
the use of AI to provide tools for autonomous cloud operations. We rely on deep learning, 
distributed traces, and time-series analysis (sequence analysis) to effectively detect and fix anomalous 
cloud infrastructure behaviors during operations to reduce the workload of human operators. 

The iForesight system, labelled under the AIOps software category, is being used to evaluate new O&M approaches. iForesight 3.0 is the result of 3 years of research with the goal to provide an intelligent new tool aimed at SRE cloud maintenance teams. 
It enables them to quickly detect and predict anomalies thanks to the use of artificial intelligence when cloud services 
are slow or unresponsive. 

## Problem
Existing tools for monitoring IT infrastructures, networks and applications focus on collecting logs, metrics, 
events, and traces from distributed systems mainly for visualization. Nonetheless, the final goal of monitoring 
is to reach a level of technological development where we have tools that conduct root cause analysis with a high 
accuracy and enable to autonomously recover systems. To achieve this goal, we still need to shift from a data 
collection stage to an insight- and action-driven paradigm. One promising path to monitor planet- and large-scale 
platforms is to rely on advanced analytics and explore techniques from statistics, time-series analysis, data mining, 
natural language processing, graph processing, machine learning, and deep learning to extract insights from large 
volumes of monitoring data to support and drive recovery actions. 

## Approach
The mission of the **Intelligent Cloud Operations** SRE team (based in [Munich](https://www.muenchen.de/int/en.html), 
Germany) is to develop new AIOps systems and tools to analyze observability data 
from [Huawei Cloud](https://www.huaweicloud.com/en-us/about/about_us.html)
to detect impact to customers, identify the root cause within seconds, and fix the problem using 
the 1/5/10 rule (detection: 1 min, RCA: 5 min, recovery: 10 min).

AIOps (Artificial Intelligence for IT operations) characterizes systems which use complex monitoring platforms, heterogeneous big data, machine learning (ML) and other artificial intelligence technologies to toubleshoot IT problems.
The term can be viewed as the evolution of [IT operations_analytic](https://en.wikipedia.org/wiki/IT_operations_analytics) and
[Software Analytics](http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf).

The following figure from [Gartner](https://www.gartner.com/en) provides a high level architecture of the system 
we are building highliting the main areas of concern: 
+ Real-time streaming and historical data, observations, 
[Big Data](https://en.wikipedia.org/wiki/Big_data), 
Machine Learning (ML), 
[Anomaly Detection](https://en.wikipedia.org/wiki/Anomaly_detection), 
[Root-Cause Analysis](https://en.wikipedia.org/wiki/Root_cause_analysis), 
[Performance Analysis](https://en.wikipedia.org/wiki/Application_performance_management), 
[Predictive Maintenance](https://en.wikipedia.org/wiki/Predictive_maintenance), and 
[Automation](https://en.wikipedia.org/wiki/Robotic_process_automation). 
<p>
<img src="https://blogs.bmc.com/wp-content/uploads/2019/02/AIOpsGraphicGartner2018-768x408.png" height="408" width="768" style="float:center;margin:0 20px 0 0;"/>
</p>

Other areas relevant to AIOps include: 
1. Monitoring and alerting
2. Capacity planing and prediction
3. Canarying validation
4. Service Scaling
5. Operational performance

In 2017 we adopted AI in the form of [Data Science](https://en.wikipedia.org/wiki/Data_science) and 
[Machine Learning](https://en.wikipedia.org/wiki/Machine_learning) approaches for anomaly detection, 
root-cause analysis, fault prediction, and automated recovery into our suite. 

These techniques, including **statistical learning**, **time-series analysis**, **deep learning**, **big data**,
**streaming**, and **data visualization**, enabled us to develop new production-ready services for troubleshooting 
Huawei Cloud and detect issues which were previously undetectable.

## AIOPs platform
An AIOps platform architecture consists of functional layers such as:

1. *Big Data processing*. Real-time processing of streaming and historical data.
2. *Data pipeline*. Connected data processing elements ingesting data from multiple sources.
3. *Library of algorithms*: Statistical functions, classical machine learning, and deep learning algorithm.
4. *Automation*. Use runbooks and RPA technology to automate repetitive tasks.
5. *User interface*. Allows IT operations teams t interact with the platform and quickly identify issues and apply corrective actions.

For 2019-2020, our work focuses on points 1)-3).

## Challenges

The challenges of operationalising AI are not limited to the understanding of deep learnign or machine learning algorithms.
Major challenges are related with software engineering, access and processing of large amounts of distributed data, 
model management, updating, deleting and training models on specialized GPUs and hardware, composition of workflows
for orchestrating parallel jobs, and the visual management of models, workflows, and results. 

## Huawei Cloud
Our cloud has planet-scale technical requirements with an 
[microservices](https://en.wikipedia.org/wiki/Microservices) architecture composed of hundreds of services.
They are distributed over thousands of hosts in many geographical regions and operate with an availability 
higher than [five nines](https://en.wikipedia.org/wiki/High_availability). 

Our system monitors Huawei Cloud which is build on top of [OpenStack](https://docs.openstack.org/), an 
opensource cloud operating system. OpenStack controls large pools of compute, storage, and networking
resources throughout tens of datacenters. The base services are predominantly written in Python and Java 
running on Linux. 

Huawei Cloud is one of the largest and fastest growing platforms in the world. 
It has a strong presence throughout the world with over 40 availability zones located across 23 geographical regions,
ranging from Germany, France, South/Central America, Hong Kong and Russia to Thailand and South Africa.

There are three properties that make platforms such as Huawei Cloud far more difficult to monitor and troubleshoot:
1. Amount of data and relationships which O&M teams need to analyze.
2. Due to its distributed nature and complexity, system data has a low signal to noise ratio.
3. Since many different subsystems interact together, semantically reconciliating data is difficult.

{'fraction_non_alphanumeric': 0.0707550514002127, 'fraction_numerical': 0.011130804679191777, 'mean_word_length': 5.151766245093763, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 31, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2094410', 'n_tokens_mistral': 3734, 'n_tokens_neox': 3467, 'n_words': 1710}