



  

ChubaoFS documentation

 [ChubaoFS](index.html#document-index) 

 stable

 

Getting Started

* [Introduction](index.html#document-overview)

	+ [High Level Architecture](index.html#high-level-architecture)

	+ [Features](index.html#features)

* [Run Cluster by Yum Tools](index.html#document-user-guide/yum)

* [Run Cluster Manually](index.html#document-manual-deploy)

	+ [Building](index.html#building)

	+ [Deployment](index.html#deployment)

	+ [Mount Client](index.html#mount-client)

	+ [Upgrading](index.html#upgrading)

Design Documentation

* [Resource Manager (Master)](index.html#document-design/master)

	+ [Utilization-Based Distribution/Placement](index.html#utilization-based-distribution-placement)

	+ [Replica Placement](index.html#replica-placement)

	+ [Meta Partition Splitting](index.html#meta-partition-splitting)

	+ [Exception Handling](index.html#exception-handling)

* [Metadata Subsystem](index.html#document-design/metanode)

	+ [Internal Structure](index.html#internal-structure)

	+ [Replication](index.html#replication)

	+ [Failure Recovery](index.html#failure-recovery)

* [Data Subsystem](index.html#document-design/datanode)

	+ [Features](index.html#features)

	+ [HTTP APIs](index.html#http-apis)

* [Object Subsystem (ObjectNode)](index.html#document-design/objectnode)

	+ [Structure](index.html#structure)

	+ [Features](index.html#features)

	+ [Semantic Transform](index.html#semantic-transform)

	+ [User](index.html#user)

	+ [Authentication](index.html#authentication)

	+ [Invisible Temporary Data](index.html#invisible-temporary-data)

	+ [Object Mode Conflict (Important)](index.html#object-mode-conflict-important)

	+ [Supported S3 Features](index.html#supported-s3-features)

	+ [Unsupported S3 Features](index.html#unsupported-s3-features)

	+ [Supported APIs](index.html#supported-apis)

	+ [Supported SDKs](index.html#supported-sdks)

* [Client](index.html#document-design/client)

	+ [Client Caching](index.html#client-caching)

	+ [Integration with FUSE](index.html#integration-with-fuse)

User Documentation

* [Resource Manager (Master)](index.html#document-user-guide/master)

	+ [Features](index.html#features)

	+ [Configurations](index.html#configurations)

	+ [Start Service](index.html#start-service)

* [Meta Subsystem](index.html#document-user-guide/metanode)

	+ [Notice](index.html#notice)

* [Data Subsystem](index.html#document-user-guide/datanode)

	+ [How To Start DataNode](index.html#how-to-start-datanode)

	+ [Configurations](index.html#configurations)

	+ [Notice](index.html#notice)

* [Object Subsystem (ObjectNode)](index.html#document-user-guide/objectnode)

	+ [How To start ObjectNode](index.html#how-to-start-objectnode)

	+ [Configurations](index.html#configurations)

	+ [Fetch Authentication Keys](index.html#fetch-authentication-keys)

	+ [Using Object Storage Interface](index.html#using-object-storage-interface)

* [Console](index.html#document-user-guide/console)

	+ [How To Start Console](index.html#how-to-start-console)

	+ [Configurations](index.html#configurations)

	+ [Notice](index.html#notice)

* [Client](index.html#document-user-guide/client)

	+ [Prerequisite](index.html#prerequisite)

	+ [Prepare Config File](index.html#prepare-config-file)

	+ [Mount](index.html#mount)

	+ [Unmount](index.html#unmount)

* [Monitor](index.html#document-user-guide/monitor)

	+ [Metrics](index.html#metrics)

	+ [Grafana DashBoard Config](index.html#grafana-dashboard-config)

* [Tune FUSE Performance](index.html#document-user-guide/fuse)

	+ [Fetch Linux kernel source code](index.html#fetch-linux-kernel-source-code)

	+ [Optimize FUSE linux kernel module](index.html#optimize-fuse-linux-kernel-module)

	+ [Build against current running Linux kernel](index.html#build-against-current-running-linux-kernel)

	+ [Install kernel module](index.html#install-kernel-module)

* [Run Cluster by Yum Tools](index.html#document-user-guide/yum)

* [Run Cluster on Docker](index.html#document-user-guide/docker)

Administration

* [Resource Manager (Master) API](index.html#document-administration)

	+ [Cluster](index.html#document-admin-api/master/cluster)

	+ [Metanode Related](index.html#document-admin-api/master/metanode)

	+ [Datanode Related](index.html#document-admin-api/master/datanode)

	+ [Volume](index.html#document-admin-api/master/volume)

	+ [Meta Partition](index.html#document-admin-api/master/meta-partition)

	+ [Data Partition](index.html#document-admin-api/master/data-partition)

	+ [Master Management](index.html#document-admin-api/master/management)

	+ [User](index.html#document-admin-api/master/user)

* [Meta Node API](index.html#meta-node-api)

	+ [Meta Partition](index.html#document-admin-api/metanode/partition)

	+ [Inode](index.html#document-admin-api/metanode/inode)

	+ [Dentry](index.html#document-admin-api/metanode/dentry)

* [Command Line Interface](index.html#command-line-interface)

	+ [CLI](index.html#document-admin-api/cli/cli)

use case

* [Use Cases](index.html#document-use-case)

	+ [Machine Learning](index.html#machine-learning)

	+ [ElasticSearch](index.html#elasticsearch)

	+ [Nginx Log Storage](index.html#nginx-log-storage)

	+ [Spark](index.html#spark)

	+ [MySQL Database Backup](index.html#mysql-database-backup)

Evaluation

* [Performance](index.html#document-evaluation)

	+ [Environment](index.html#environment)

	+ [Small File Performance and Scalability](index.html#small-file-performance-and-scalability)

	+ [IO Performance and Scalability](index.html#io-performance-and-scalability)

		- [1. Sequential Read](index.html#sequential-read)

		- [2. Sequential Write](index.html#sequential-write)

		- [3. Random Read](index.html#random-read)

		- [4. Random Write](index.html#random-write)

	+ [Metadata Performance and Scalability](index.html#metadata-performance-and-scalability)

* [Integrity](index.html#integrity)

* [Workload](index.html#workload)

* [Scalability](index.html#scalability)

Contributing

* [Contributing to ChubaoFS](index.html#document-contributing)

	+ [Bug Reports](index.html#bug-reports)

	+ [Patch Guidelines](index.html#patch-guidelines)

	+ [Credits](index.html#credits)

FAQ

* [Environment Requirements and Capacity Planning](index.html#document-env)

	+ [Environment Requirements](index.html#environment-requirements)

	+ [Capacity Planning](index.html#capacity-planning)

	+ [Multi-Zone Deploy](index.html#multi-zone-deploy)

* [Q&A](index.html#document-faq)

	+ [Compile](index.html#compile)

	+ [Node & Disk Failure](index.html#node-disk-failure)

	+ [Performance of Data and Metadata](index.html#performance-of-data-and-metadata)

	+ [Capacity Management](index.html#capacity-management)

	+ [Zone](index.html#zone)

	+ [Node status is abnormal](index.html#node-status-is-abnormal)

	+ [Upgrade](index.html#upgrade)

	+ [Update Configuration Online](index.html#update-configuration-online)

	+ [Update Configuration Offline](index.html#update-configuration-offline)

	+ [Handle Logs](index.html#handle-logs)

	+ [Data Loss and Consistence](index.html#data-loss-and-consistence)

	+ [Fuse Client](index.html#fuse-client)

[ChubaoFS](index.html#document-index)

* [Docs](index.html#document-index) »

* ChubaoFS documentation

* [Edit on GitHub](https://github.com/chubaofs/chubaofs/blob/501719b9b4782c078aee421093d29c2f83b07fe8/docs/source/index.rst)

---

Welcome to ChubaoFS(Chubao File System)[¶](#welcome-to-chubaofs-chubao-file-system "Permalink to this headline")

================================================================================================================

Introduction[¶](#introduction "Permalink to this headline")

-----------------------------------------------------------

ChubaoFS(Chubao File System) is a distributed file system that is designed to natively support large scale container platforms.

### High Level Architecture[¶](#high-level-architecture "Permalink to this headline")

[![Architecture](_images/cfs-arch.png)](_images/cfs-arch.png)

ChubaoFS consists of a *metadata subsystem*, a *data subsystem*, and a *resource manager*, and can be accessed by different *clients* (as a set of application processes) hosted on the containers through different file system instances called *volumes*.

The metadata subsystem stores the file metadata, and consists of a set of *meta nodes*. Each meta node consists of a set of *meta partitions*.

The data subsystem stores the file contents, and consists of a set of *data nodes*. Each data node consists of a set of *data partitions*.

The volume is a logical concept in ChubaoFS and consists of one or multiple meta partitions and one or multiple data partitions. Each partition can only be assigned to a single volume.

From a client’s perspective, the volume can be viewed as a file system instance that contains data accessible by the containers.

A volume can be mounted to multiple containers so that files can be shared among different clients simultaneously, and needs to be created at the very beginning before the any file operation.

A ChubaoFS cluster deployed at each data center can have hundreds of thousands of volumes, whose data sizes vary from a few gigabytes to several terabytes.

Generally speaking, the resource manager periodically communicates with the metadata subsystem and data subsystem to manage the meta nodes and data nodes, respectively. Each client periodically communicates with the resource manager to obtain the up-to-date view of the mounted volume. A file operation usually initiates the communications from the client to the corresponding meta node and data node directly, without the involvement of the resource manager. The updated view of the mounted volume, as well as the file metadata are usually cached at the client side to reduce the communication overhead.

### Features[¶](#features "Permalink to this headline")

#### Scalable Meta Management[¶](#scalable-meta-management "Permalink to this headline")

The metadata operations could make up as much as half of typical file system workloads. On our platform, this becomes even more important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system.

The metadata subsystem can be considered as an in-memory

datastore of the file metadata. It can have thousands of meta nodes, each of which can have hundreds of billions of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries.

We employ two b-trees called inodeTree and dentryTree for

fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting.

#### General-Purpose Storage Engine[¶](#general-purpose-storage-engine "Permalink to this headline")

To reduce the storage cost, many applications and services are served from the same shared storage infrastructure (aka “multi-tenancy”). The workloads of different applications and services are mixed together, where the file size can vary from a few kilobytes to hundreds of gigabytes, and the files can be written in a sequential or random fashion. For example, the log files usually need to be written sequentially in the execution order of the code; some data analytics in the machine learning domain are based on the data stored on the underlying file system; and a database engine running on top of the file system can modify the stored data frequently. A dedicated file system needs to be able to serve for all these different workloads with excellent performance.

#### Strong Replication Consistency[¶](#strong-replication-consistency "Permalink to this headline")

E-commence venders who move their line of business applications to the cloud usually prefer strong consistency. For example, an image processing service may not want to provide the customer with an outdated image that does not match the product description. This can be easily achieved if there is only one copy of the file. But to ensure a distributed file system to continue operating properly in the event of machines failures, which can be caused by various reasons such as faulty hard drives, bad motherboards, etc, there are usually multiple replicas of the same file. As a result, in a desired file system, the data read from any of the replicas must be consistent with each other.

#### Relaxed POSIX Semantics and Metadata Atomicity[¶](#relaxed-posix-semantics-and-metadata-atomicity "Permalink to this headline")

In a POSIX-compliant distributed file system, the behavior of serving multiple processes on multiple client nodes should be the same as the behavior of a local file system serving multiple processes on a single node with direct attached storage. ChubaoFS provides POSIX-compliant APIs. However, the POSIX consistency semantics, as well as the atomicity requirement between the inode and dentry of the same file, have been carefully relaxed in order to better align with the needs of applications and to improve the system performance.

Run Cluster by Yum Tools[¶](#run-cluster-by-yum-tools "Permalink to this headline")

-----------------------------------------------------------------------------------

Yum tools to run a ChubaoFS cluster for CentOS 7+ is provided. The list of RPM packages dependencies can be installed with:

```

$ yum install http://storage.jd.com/chubaofsrpm/latest/cfs-install-latest-el7.x86_64.rpm

$ cd /cfs/install

$ tree -L 2

.

├── install_cfs.yml

├── install.sh

├── iplist

├── src

└── template

    ├── client.json.j2

    ├── console.json.j2

    ├── create_vol.sh.j2

    ├── datanode.json.j2

    ├── grafana

    │   ├── grafana.ini

    │   ├── init.sh

    │   └── provisioning

    ├── master.json.j2

    ├── metanode.json.j2

    └── objectnode.json.j2

```

Set parameters of the ChubaoFS cluster in **iplist**.

* **[master]**, **[datanode]** , **[metanode]** , **[console]**, **[monitor]** , **[client]** modules includes member IP addresses of each role.

* **[cfs:vars]** module includes parameters for SSH connection. So make sure the port, username and password for SSH connection is unified before start.

* **#master config** module includes parameters of Master.

Properties[¶](#id1 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| master\_clusterName | string | The cluster identifier | Yes |

| master\_listen | string | Http port which api service listen on | Yes |

| master\_prof | string | golang pprof port | Yes |

| master\_logDir | string | Path for log file storage | Yes |

| master\_logLevel | string | Level operation for logging. Default is *info*. | No |

| master\_retainLogs | string | the number of raft logs will be retain. | Yes |

| master\_walDir | string | Path for raft log file storage. | Yes |

| master\_storeDir | string | Path for RocksDB file storage,path must be exist | Yes |

| master\_exporterPort | int | The prometheus exporter port | No |

| master\_metaNodeReservedMem | string | If the metanode memory is below this value, it will be marked as read-only. Unit: byte. 1073741824 by default. | No |

* **#datanode config** module includes parameters of DataNodes.

Properties[¶](#id2 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| datanode\_listen | string | Port of TCP network to be listen | Yes |

| datanode\_prof | string | Port of HTTP based prof and api service | Yes |

| datanode\_logDir | string | Path for log file storage | Yes |

| datanode\_logLevel | string | Level operation for logging. Default is *info* | No |

| datanode\_raftHeartbeat | string | Port of raft heartbeat TCP network to be listen | Yes |

| datanode\_raftReplica | string | Port of raft replicate TCP network to be listen | Yes |

| datanode\_raftDir | string | Path for raft log file storage | No |

| datanode\_exporterPort | string | Port for monitor system | No |

| datanode\_disks | string slice | 

Format: *PATH:RETAIN*.

PATH: Disk mount point, must exists. RETAIN: Retain space. (Ranges: 20G-50G.)

RETAIN: The minimum free space(Bytes) reserved for the path.

 | Yes |

* **#metanode config** module includes parameters of MetaNodes.

Properties[¶](#id3 "Permalink to this table")

| Key | Type | Description | Mandatory |  |

| --- | --- | --- | --- | --- |

| metanode\_listen | string | Listen and accept port of the server | Yes |  |

| metanode\_prof | string | Pprof port | Yes |  |

| metanode\_logLevel | string | Level operation for logging. Default is *info* | No |  |

| metanode\_metadataDir | string | MetaNode store snapshot directory | Yes |  |

| metanode\_logDir | string | Log directory | Yes |  |

| metanode\_raftDir | string | Raft wal directory | Yes |  |

| metanode\_raftHeartbeatPort | string | Raft heartbeat port | Yes |  |

| metanode\_raftReplicaPort | string | Raft replicate port | Yes |  |

| metanode\_exporterPort | string | Port for monitor system | No |  |

| metanode\_totalMem | string | Max memory metadata used. The value needs to be higher than the value of *metaNodeReservedMem* in the master configuration. Unit: byte | Yes |  |

* **#objectnode config** module includes parameters of ObjectNodes.

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| objectnode\_listen | string | Listen and accept port of the server | Yes |

| objectnode\_domains | string slice | 

Domain of S3-like interface which makes wildcard domain support

Format: `DOMAIN`

 | No |

| objectnode\_logDir | string | Log directory | Yes |

| objectnode\_logLevel | string | 

Level operation for logging.

Default: `error`

 | No |

| objectnode\_exporterPort | string | Port for monitor system | No |

| objectnode\_enableHTTPS | string | Enable HTTPS | Yes |

* **#console config** module includes parameters of Console.

Properties[¶](#id4 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| console\_logDir | string | Path for log file storage | Yes |

| console\_logLevel | string | Level operation for logging. Default is *info* | No |

| console\_listen | string | Port of TCP network to be listen, default is 80 | Yes |

* **#client config** module includes parameters of Client.

Properties[¶](#id5 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| client\_mountPoint | string | Mount point of the client | Yes |

| client\_volName | string | Volume name | No |

| client\_owner | string | Owner id | Yes |

| client\_SizeGB | string | Initial size to create the volume if it not exists | No |

| client\_logDir | string | Path for log file storage | Yes |

| client\_logLevel | string | Level operation for logging. Default is *info* | No |

| client\_exporterPort | string | Port exposed to monitor system | Yes |

| client\_profPort | string | Pprof port | No |

```

[master]

10.196.59.198

10.196.59.199

10.196.59.200

[datanode]

...

[cfs:vars]

ansible\_ssh\_port=22

ansible\_ssh\_user=root

ansible\_ssh\_pass="password"

...

#master config

...

#datanode config

...

datanode\_disks = '"/data0:10737418240","/data1:10737418240"'

...

#metanode config

...

metanode\_totalMem = "28589934592"

...

#objectnode config

...

#console config

...

```

For more configurations, please refer to [Resource Manager (Master)](index.html#document-user-guide/master); [Meta Subsystem](index.html#document-user-guide/metanode); [Data Subsystem](index.html#document-user-guide/datanode); [Client](index.html#document-user-guide/client); [Monitor](index.html#document-user-guide/monitor); [Console](index.html#document-user-guide/console).

Start the resources of ChubaoFS cluster with script **install.sh** . (make sure the Master is started first)

```

$ bash install.sh -h

Usage: install.sh -r | --role [datanode | metanode | master | objectnode | console | monitor | client | all | createvol ]

$ bash install.sh -r master

$ bash install.sh -r metanode

$ bash install.sh -r datanode

$ bash install.sh -r objectnode

$ bash install.sh -r console

$ bash install.sh -r monitor

$ bash install.sh -r client

```

Check mount point at **/cfs/mountpoint** on **client** node defined in **iplist** .

Open <http://consul.prometheus-cfs.local> in browser for monitoring system(the IP of monitoring system is defined in **iplist** ).

Run Cluster Manually[¶](#run-cluster-manually "Permalink to this headline")

---------------------------------------------------------------------------

### Building[¶](#building "Permalink to this headline")

Use following command to build server and client:

```

make build

```

If the build is successful, cfs-server and cfs-client will be found in directory build/bin

### Deployment[¶](#deployment "Permalink to this headline")

#### Start Resource Manager (Master)[¶](#start-resource-manager-master "Permalink to this headline")

```

nohup ./cfs-server -c master.json &

```

Sample *master.json* is shown as follows,

```

{

  "role": "master",

  "ip": "10.196.59.198",

  "listen": "17010",

  "prof":"17020",

  "id":"1",

  "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",

  "retainLogs":"2000",

  "logDir": "/cfs/master/log",

  "logLevel":"info",

  "walDir":"/cfs/master/data/wal",

  "storeDir":"/cfs/master/data/store",

  "consulAddr": "http://consul.prometheus-cfs.local",

  "exporterPort": 9500,

  "clusterName":"chubaofs01",

  "metaNodeReservedMem": "1073741824"

}

```

For detailed explanations of *master.json*, please refer to [Resource Manager (Master)](index.html#document-user-guide/master).

#### Start Metanode[¶](#start-metanode "Permalink to this headline")

```

nohup ./cfs-server -c meta.json &

```

Sample *meta.json is* shown as follows,

```

{

    "role": "metanode",

    "listen": "17210",

    "prof": "17220",

    "logLevel": "info",

    "metadataDir": "/cfs/metanode/data/meta",

    "logDir": "/cfs/metanode/log",

    "raftDir": "/cfs/metanode/data/raft",

    "raftHeartbeatPort": "17230",

    "raftReplicaPort": "17240",

    "totalMem":  "8589934592",

    "consulAddr": "http://consul.prometheus-cfs.local",

    "exporterPort": 9501,

    "masterAddr": [

        "10.196.59.198:17010",

        "10.196.59.199:17010",

        "10.196.59.200:17010"

    ]

}

```

For detailed explanations of *meta.json*, please refer to [Meta Subsystem](index.html#document-user-guide/metanode).

#### Start Datanode[¶](#start-datanode "Permalink to this headline")

1. Prepare data directories

**Recommendation** Using independent disks can reach better performance.

**Disk preparation**

> 

> 1.1 Check available disks

> 

> 

> 

> > 

> > 

> > ```

> > fdisk -l

> > 

> > ```

> > 

> > 

> > 

> 

> 

> 1.2 Build local Linux file system on the selected devices

> 

> 

> 

> > 

> > 

> > ```

> > mkfs.xfs -f /dev/sdx

> > 

> > ```

> > 

> > 

> > 

> 

> 

> 1.3 Make mount point

> 

> 

> 

> > 

> > 

> > ```

> > mkdir /data0

> > 

> > ```

> > 

> > 

> > 

> 

> 

> 1.4 Mount the device on mount point

> 

> 

> 

> > 

> > 

> > ```

> > mount /dev/sdx /data0

> > 

> > ```

> > 

> > 

> > 

> 

> 

>

2. Start datanode

```

nohup ./cfs-server -c datanode.json &

```

Sample *datanode.json* is shown as follows,

```

{

  "role": "datanode",

  "listen": "17310",

  "prof": "17320",

  "logDir": "/cfs/datanode/log",

  "raftDir": "/cfs/datanode/log",

  "logLevel": "info",

  "raftHeartbeat": "17330",

  "raftReplica": "17340",

  "consulAddr": "http://consul.prometheus-cfs.local",

  "exporterPort": 9502,

  "masterAddr": [

     "10.196.59.198:17010",

     "10.196.59.199:17010",

     "10.196.59.200:17010"

  ],

  "disks": [

     "/data0:10737418240",

     "/data1:10737418240"

  ]

}

```

For detailed explanations of *datanode.json*, please refer to [Data Subsystem](index.html#document-user-guide/datanode).

#### Start ObjectNode[¶](#start-objectnode "Permalink to this headline")

```

nohup ./cfs-server -c objectnode.json &

```

Sample *objectnode.json is* shown as follows,

```

{

    "role": "objectnode",

    "domains": [

        "object.cfs.local"

    ],

    "listen": 17410,

    "masterAddr": [

       "10.196.59.198:17010",

       "10.196.59.199:17010",

       "10.196.59.200:17010"

    ],

    "logLevel": "info",

    "logDir": "/cfs/Logs/objectnode"

}

```

For detailed explanations of *objectnode.json*, please refer to [Object Subsystem (ObjectNode)](index.html#document-user-guide/objectnode).

#### Start Console[¶](#start-console "Permalink to this headline")

```

nohup ./cfs-server -c console.json &

```

Sample *console.json is* shown as follows,

```

{

    "role": "console",

    "logDir": "/cfs/log/",

    "logLevel": "debug",

    "listen": "80",

    "masterAddr": [

        "192.168.0.11:17010",

        "192.168.0.12:17010",

        "192.168.0.13:17010"

    ],

    "objectNodeDomain": "object.chubao.io",

    "monitor\_addr": "http://192.168.0.102:9090",

    "dashboard\_addr": "http://192.168.0.103",

    "monitor\_app": "cfs",

    "monitor\_cluster": "cfs"

}

```

For detailed explanations of *console.json*, please refer to [Console](index.html#document-user-guide/console).

#### Create Volume[¶](#create-volume "Permalink to this headline")

By default, there are only a few data partitions allocated upon volume creation, and will be dynamically expanded according to actual usage.

```

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=10000&owner=cfs"

```

For performance evaluation, extra data partitions shall be pre-created according to the amount of data nodes and disks to reach maximum performance.

```

curl -v "http://10.196.59.198:17010/dataPartition/create?name=test&count=120"

```

### Mount Client[¶](#mount-client "Permalink to this headline")

1. Run `modprobe fuse` to insert FUSE kernel module.

2. Run `yum install -y fuse` to install libfuse.

3. Run `cfs-client -c fuse.json` to start a client daemon.

Sample *fuse.json* is shown as follows,

```

{

  "mountPoint": "/cfs/mountpoint",

  "volName": "ltptest",

  "owner": "ltptest",

  "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",

  "logDir": "/cfs/client/log",

  "profPort": "17510",

  "exporterPort": "9504",

  "logLevel": "info"

}

```

For detailed explanations of *fuse.json*, please refer to [Client](index.html#document-user-guide/client).

Note that end user can start more than one client on a single machine, as long as mountpoints are different.

### Upgrading[¶](#upgrading "Permalink to this headline")

1. freeze the cluster

```

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"

```

2. upgrade each module

3. closed freeze flag

```

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=false"

```

Resource Manager (Master)[¶](#resource-manager-master "Permalink to this headline")

-----------------------------------------------------------------------------------

The resource manager manages the file system by processing different types of tasks, such as creating/deleting/updating/loading partitions and keeping track of the resource status (such as the memory/disk utilization). The resource manager is also responsible for creating new volumes and adding new meta/data nodes to the ChubaoFS cluster. It has multiple replicas, among which the consistency is maintained by a consensus algorithm such as Raft, and persisted to a key value store such as RocksDB for backup and recovery.

### Utilization-Based Distribution/Placement[¶](#utilization-based-distribution-placement "Permalink to this headline")

The resource manager is a utilization-based distribution strategy to places the file metadata and contents across different meta and data nodes.

Because each node can have multiple partitions, and the partitions in a volume do not need to reside on the same node, this distribution can be controlled at a finer granularity, resulting a more efficient resource management.

Specifically, the distribution of file metadata and contents works follows:

First, when mounting a volume, the client asks the resource manager for a set of available meta and data partitions. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Later on, when writing a file, the client can only choose the meta and data partitions in a random fashion from the ones allocated by the resource manager.

Second, when the resource manager finds that all the partitions in a volume is about to be full, it automatically adds a set of new partitions to this volume. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Note that, when a partition is full, or a threshold (i.e., the number of files on a meta partition or the number of extents on a data partition) is reached, no new data can be stored on this partition, although it can still be modified or deleted.

### Replica Placement[¶](#replica-placement "Permalink to this headline")

When choosing partitions for the replicas, the resource manager ensures that two replicas of the same partition never reside on the same node.

### Meta Partition Splitting[¶](#meta-partition-splitting "Permalink to this headline")

There is a special requirement when splitting a meta partition.

In particular, if a meta partition is about to reach its upper limit of the number of stored inodes and dentries, a splitting task needs to be performed with the requirement to ensure that the inode ids stored at the newly created partition are unique from the ones stored at the original partition.

To meet this requirement, when splitting a meta partition, the resource manager cuts off the inode range of the meta partition in advance at a upper bound *end*, a value greater than highest inode id used so far (denoted as *maxInodeID*), and sends a split request to the meta node to (1) update the inode range from *1* to *end* for the original meta partition, and (2) create a new meta partition with the inode range from *end + 1* to *infinity* for this volume.

As a result, the inode range for these two meta partitions becomes *[1, end]* and *[end + 1, infinity]*, respectively. If there is another file needs to be created, then its inode id will be chosen as *maxInodeID + 1* in the original meta partition, or *end + 1* in the newly created meta partition.

The *maxInodeID* of each meta partition can be obtained by the periodical communication between the resource manager and the the meta nodes.

### Exception Handling[¶](#exception-handling "Permalink to this headline")

When a request to a meta/data partition times out (e.g., due to network outage), the remaining replicas of this partition are marked as read-only.

When a meta/data partition is no longer available (e.g., due to hardware failures), all the data on this partition will eventually be migrated to a new available partition manually. This unavailability is identified by the multiple failures reported by the node when operating the files.

Metadata Subsystem[¶](#metadata-subsystem "Permalink to this headline")

-----------------------------------------------------------------------

The metadata operations could make up as much as half of typical file system workloads.

this can be important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance/storage bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system.

### Internal Structure[¶](#internal-structure "Permalink to this headline")

The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have a set of meta partitions.

Each meta partition on a meta node stores the file metadata in memory by maintaining a set of *inodes* and a set of *dentries*.

Generally speaking, An inode is an object that represents the underlying file (or directory), and can be identified by an unsigned 64-bit integer called the *inode id*. A dentry is an object that represents the directory hierarchy and can be identified by a string name and the id of the parent inode.

For example, if we have two directories *foo* and *bar*, where *foo* is the parent directory of *bar*, then there are two inodes: one for *foo* called *i1*, and the other for *bar* called *i2*, and one dentry to represent the hierarchy of these two directories where *i2* is the current inode and *i1* is the parent inode.

A meta partition can only store the inodes and dentries of the files from the same volume. We employ two b-trees called *inodeTree* and *dentryTree* for fast lookup of inodes and dentries in the memory. The *inodeTree* is indexed by the inode id, and the *dentryTree* is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as *start* and *end*) stored on a meta partition for splitting (see [Resource Manager (Master)](index.html#document-design/master)).

### Replication[¶](#replication "Permalink to this headline")

The replication during file write is performed in terms of meta partitions.

The replication consistency is ensured by a revision of the Raft consensus protocol called the MultiRaft, which has the advantage of reduced heartbeat network traffic comparing to the original version.

### Failure Recovery[¶](#failure-recovery "Permalink to this headline")

The in-memory meta partitions are persisted to the local disk by snapshots and logs for backup and recovery. Some techniques such as log compaction are used to reduce the log files sizes and shorten the recovery time.

It is worth noting that, a failure that happens during a metadata operation could result an *orphan* inode with which has no dentry to be associated. The memory and disk space occupied by this inode can be hard to free. To minimize the chance of this case to happen, the client always issues a retry after a failure until the request succeeds or the maximum retry limit is reached.

Data Subsystem[¶](#data-subsystem "Permalink to this headline")

---------------------------------------------------------------

The data subsystem is optimized for the storage of large and small files, which can be accessed in a sequential or random fashion.

![Data Subsystem Architecture](_images/data-subsystem.png)

### Features[¶](#features "Permalink to this headline")

* Large File Storage

For large files, the contents are stored as a sequence of one or multiple extents, which can be distributed across different data partitions on different data nodes.

Writing a new file to the extent store always causes the data to be written at the zero-offset of a new extent, which eliminates the need for the offset within the extent. The last extent of a file does not need to fill up its size limit by padding (i.e., the extent does not have holes), and never stores the data from other files.

* Small File Storage

The contents of multiple small files are aggregated and stored in a single extent, and the physical offset of each file content in the extent is recorded in the corresponding meta node. ChubaoFS relies on the punch hole interface, textit{fallocate()}footnote{url{<http://man7.org/linux/man-pages/man2/fallocate.2.html>}}, to textit{asynchronous} free the disk space occupied by the to-be-deleted file. The advantage of this design is to eliminate the need of implementing a garbage collection mechanism and therefore avoid to employ a mapping from logical offset to physical offset in an extent~cite{haystack}. Note that this is different from deleting large files, where the extents of the file can be removed directly from the disk.

* Replication

The replication is performed in terms of partitions during file writes. Depending on the file write pattern, ChubaoFS adopts different replication strategies.

When a file is sequentially written into ChubaoFS, a primary-backup replication protocol is used to ensure the strong consistency with optimized IO throughput.

![_images/workflow-sequential-write.png](_images/workflow-sequential-write.png)

When overwriting an existing file portion during random writes, we employ a MultiRaft-based replication protocol, which is similar to the one used in the metadata subsystem, to ensure the strong consistency.

![_images/workflow-overwriting.png](_images/workflow-overwriting.png)

* Failure Recovery

Because of the existence of two different replication protocols, when a failure on a replica is discovered, we first start the recovery process in the primary-backup-based replication by checking the length of each extent and making all extents aligned. Once this processed is finished, we then start the recovery process in our MultiRaft-based replication.

### HTTP APIs[¶](#http-apis "Permalink to this headline")

| API | Method | Parameters | Description |

| --- | --- | --- | --- |

| /disks | GET | N/A | Get disk list and informations. |

| /partitions | GET | N/A | Get parttion list and infomartions. |

| /partition | GET | partitionId[int] | Get detail of specified partition. |

| /extent | GET | partitionId[int]&extentId[int] | Get extent informations. |

| /stats | GET | N/A | Get status of the datanode. |

Object Subsystem (ObjectNode)[¶](#object-subsystem-objectnode "Permalink to this headline")

-------------------------------------------------------------------------------------------

The object subsystem provides S3-compatible object storage interface.

It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible).

So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.

### Structure[¶](#structure "Permalink to this headline")

> 

> ![_images/cfs-object-subsystem-structure.png](_images/cfs-object-subsystem-structure.png)

> 

The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master).

Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.

ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster

which it belong to directly without any volume-mount operation.

### Features[¶](#features "Permalink to this headline")

* Provides object storage interface compatible with native Amazon S3 SDKs.

* Fusion storage expose two type interface (POSIX and S3-compatible).

* Stateless and high scalability

### Semantic Transform[¶](#semantic-transform "Permalink to this headline")

Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.

| POSIX | Object Storage |

| --- | --- |

| `Cluster` | `Region` |

| `Volume` | `Bucket` |

| `Path` | `Key` |

**Example:**

> 

> 

> > 

> > ![_images/cfs-object-subsystem-semantic.png](_images/cfs-object-subsystem-semantic.png)

> > 

> 

> 

> Put object ‘*example/a/b.txt*’ will be create and write data to file ‘*/a/b.txt*’ in volume ‘*example*’.

> 

> 

> 

### User[¶](#user "Permalink to this headline")

Before using the object storage service, you need to create a user through the Master.

While creating users, *AccessKey* and *SecretKey* will be generated for each user.

ChubaoFS uses the field **Owner** of the volume as the user ID. There are two ways to create users:

* When creating a volume through the API of the Master, if there is no user with the same name as the owner of the volume in the cluster, a user with the user ID of Owner will be automatically created.

* Create a user by calling the user management API of the Master.

### Authentication[¶](#authentication "Permalink to this headline")

The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service.

The authentication consisting of *AccessKey* and *SecretKey* generated by Resource Manager (Master) with user creation, which can be obtained through the Master API.

The *AccessKey* is a 16-character string unique in the entire ChubaoFS cluster.

The user has all access permissions to the volume owned by him. Users can grant other users specified permissions to access volumes under their own names. The permissions are divided into the following three categories:

* Readonly or readwrite permission.

* Permission for a single operation, such as *GetObject*, *PutObject*, etc.

* Custom permission.

When a user uses the object storage service to execute a certain operation, ChubaoFS will identify whether the user has the corresponding permission.

### Invisible Temporary Data[¶](#invisible-temporary-data "Permalink to this headline")

In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary.

The volume operator in ObjectNode puts file data into temporary which only have ‘**inode**’ without ‘**dentry**’ in metadata.

When all the file data stored successfully, the volume operator create or update ‘**dentry**’ in metadata makes it visible to users.

### Object Mode Conflict (Important)[¶](#object-mode-conflict-important "Permalink to this headline")

The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service.

So that the object with key ‘*a/b/c*’ and the object with key ‘*a/b*’ are different object without any conflict.

But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘*b*’ part in key ‘*a/b/c*’ transform to folder ‘*b*’ under the folder ‘*a*’ , and in key ‘*a/b*’ transform to file ‘*b*’ under the folder ‘*a*’.

The object key like this is conflict in ChubaoFS.

When the object being operated has conflicts with the existing object due to the above mode inconsistency, the ObjectNode will return a `409` status code to the client.

### Supported S3 Features[¶](#supported-s3-features "Permalink to this headline")

* File object operations.

* Directory object operations.

* Multipart upload.

* Parallel download for high-level SDK APIs.

* Tagging for bucket and object.

* User-defined metadata for object.

* IP address and network segment black and white list for bucket ACL.

* Signature Algorithm V2 and V4.

* Cross-Origin Resource Sharing (CORS).

### Unsupported S3 Features[¶](#unsupported-s3-features "Permalink to this headline")

* Version

* Restore deleted objects

* Locking objects

* Lifecycle configuration for bucket and object.

* Hosting Websites

* Encryption

* BitTorrent

### Supported APIs[¶](#supported-apis "Permalink to this headline")

| API | Reference |

| --- | --- |

| `AbortMultipartUpload` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html> |

| `CompleteMultipartUpload` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html> |

| `CopyObject` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html> |

| `CreateBucket` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html> |

| `CreateMultipartUpload` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html> |

| `DeleteBucket` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html> |

| `DeleteBucketCors` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketCors.html> |

| `DeleteBucketPolicy` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketPolicy.html> |

| `DeleteBucketTagging` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketTagging.html> |

| `DeleteObject` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html> |

| `DeleteObjects` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html> |

| `DeleteObjectTagging` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html> |

| `GetBucketAcl` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html> |

| `GetBucketCors` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketCors.html> |

| `GetBucketLocation` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html> |

| `GetBucketPolicy` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html> |

| `GetBucketTagging` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketTagging.html> |

| `GetObject` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html> |

| `GetObjectAcl` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html> |

| `GetObjectTagging` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html> |

| `HeadBucket` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html> |

| `HeadObject` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html> |

| `ListBuckets` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html> |

| `ListMultipartUploads` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html> |

| `ListObjects` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html> |

| `ListObjectsV2` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html> |

| `ListParts` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html> |

| `PutBucketAcl` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAcl.html> |

| `PutBucketCors` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html> |

| `PutBucketPolicy` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html> |

| `PutBucketTagging` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketTagging.html> |

| `PutObject` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html> |

| `PutObjectAcl` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html> |

| `PutObjectTagging` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html> |

| `UploadPart` | <https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html> |

### Supported SDKs[¶](#supported-sdks "Permalink to this headline")

Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

| Name | Language | Link |

| --- | --- | --- |

| AWS SDK for Java | `Java` | <https://aws.amazon.com/sdk-for-java/> |

| AWS SDK for JavaScript | `JavaScript` | <https://aws.amazon.com/sdk-for-browser/> |

| AWS SDK for JavaScript in Node.js | `JavaScript` | <https://aws.amazon.com/sdk-for-node-js/> |

| AWS SDK for Go | `Go` | <https://docs.aws.amazon.com/sdk-for-go/> |

| AWS SDK for PHP | `PHP` | <https://aws.amazon.com/sdk-for-php/> |

| AWS SDK for Ruby | `Ruby` | <https://aws.amazon.com/sdk-for-ruby/> |

| AWS SDK for .NET | `.NET` | <https://aws.amazon.com/sdk-for-net/> |

| AWS SDK for C++ | `C++` | <https://aws.amazon.com/sdk-for-cpp/> |

| Boto3 | `Python` | <http://boto.cloudhackers.com> |

Client[¶](#client "Permalink to this headline")

-----------------------------------------------

The client runs on each container executing application code and exposes a file system interface to applications and can access a mounted volume via a user-space file system interface such as FUSE.

### Client Caching[¶](#client-caching "Permalink to this headline")

The client process runs entirely in the user space with its own cache, which has been used in the following cases.

To reduce the communication with the resource manager, the client caches the addresses of the available meta and data partitions assigned to the mounted volume, which can be obtained at the startup, and periodically synchronizes this available partitions with the resource manager.

To reduce the communication with the meta nodes, the client also caches the returned inodes and dentries when creating new files, as well as the data partition id, the extent id and the offset, after the file has been written to the data node successfully. When a file is opened for read/write, the client will force the cache metadata to be synchronous with the meta node.

To reduce the communication with the data nodes, the client caches the most recently identified leader. Our observation is that, when reading a file, the client may not know which data node is the current leader because the leader could change after a failure recovery. As a result, the client may try to send the read request to each replica one by one until a leader is identified. However, since the leader does not change frequently, by caching the last identified leader, the client can have minimized number of retries in most cases.

### Integration with FUSE[¶](#integration-with-fuse "Permalink to this headline")

The ChubaoFS client has been integrated with FUSE to provide a file system interface in the user space. In the past, low performance is considered the main disadvantage of such user-space file systems. But over the years, FUSE has made several improvement on its performance such as multithreading and write-back cache. In the future, we plan to develop our own POSIX-compliant file system interface in the kernel space to completely eliminate the overhead from FUSE.

Currently the write-back cache feature does not work well in ChubaoFS due to the following reason. The default write behavior of FUSE is called directIO, which bypasses the kernel’s page cache. This results in performance problems on writing small files as each write pushes the file data to the user daemon. The solution FUSE implemented was to make the page cache support a write-back policy that aggregates small data first, and then make writes asynchronous. With that change, file data can be pushed to the user daemon in larger blobs at a time. However, in real production, we found that the write-back cache is not very useful, because a write operation usually invoke another process that tries to balance the dirty pages (pages in the main memory that have been modified during writing data to disk are marked as “dirty” and have to be flushed to disk before they can be freed), which incurs extra overhead. This overhead becomes more obvious when small files are continuously written through FUSE.

Resource Manager (Master)[¶](#resource-manager-master "Permalink to this headline")

-----------------------------------------------------------------------------------

The cluster contains dataNodes,metaNodes,vols,dataPartitions and metaPartitions,they are managed by master server. The master server caches the metadata in mem,persist to GoLevelDB,and ensure consistence by raft protocol.

The master server manages dataPartition id to dataNode server mapping,metaPartition id to metaNode server mapping.

At lease 3 master nodes are required in respect to high availability.

### Features[¶](#features "Permalink to this headline")

* Multi-tenant, Resource Isolation

* dataNodes,metaNodes shared,vol owns dataPartition and metaPartition exclusive

* Asynchronous communication with dataNode and metaNode

### Configurations[¶](#configurations "Permalink to this headline")

ChubaoFS use **JSON** as configuration file format.

Properties

 :header: “Key”, “Type”, “Description”, “Mandatory”[¶](#id1 "Permalink to this table")

| role | string | Role of process and must be set to master | Yes |

| ip | string | host ip | Yes |

| listen | string | Http port which api service listen on | Yes |

| prof | string | golang pprof port | Yes |

| id | string | identy different master node | Yes |

| peers | string | the member information of raft group | Yes |

| logDir | string | Path for log file storage | Yes |

| logLevel | string | Level operation for logging. Default is *error*. | No |

| retainLogs | string | the number of raft logs will be retain. | Yes |

| walDir | string | Path for raft log file storage. | Yes |

| storeDir | string | Path for RocksDB file storage,path must be exist | Yes |

| clusterName | string | The cluster identifier | Yes |

| exporterPort | int | The prometheus exporter port | No |

| consulAddr | string | The consul register addr for prometheus exporter | No |

| metaNodeReservedMem | string | If the metanode memory is below this value, it will be marked as read-only. Unit: byte. 1073741824 by default. | No |

| heartbeatPort | string | Raft heartbeat port,5901 by default | No |

| replicaPort | string | Raft replica Port,5902 by default | No |

| nodeSetCap | string | the capacity of node set,18 by default | No |

| missingDataPartitionInterval | string | how much time it has not received the heartbeat of replica,the replica is considered missing ,24 hours by default | No |

| dataPartitionTimeOutSec | string | how much time it has not received the heartbeat of replica, the replica is considered not alive ,10 minutes by default | No |

| numberOfDataPartitionsToLoad | string | the maximum number of partitions to check at a time,40 by default | No |

| secondsToFreeDataPartitionAfterLoad | string | the task that release the memory occupied by loading data partition task can be start, only after secondsToFreeDataPartitionAfterLoad seconds

,300 by default | No |

| tickInterval | string | the interval of timer which check heartbeat and election timeout,500 ms by default | No |

| electionTick | string | how many times the tick timer has reset,the election is timeout,5 by default | No |

**Example:**

```

{

 "role": "master",

 "id":"1",

 "ip": "10.196.59.198",

 "listen": "17010",

 "prof":"17020",

 "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",

 "retainLogs":"20000",

 "logDir": "/cfs/master/log",

 "logLevel":"info",

 "walDir":"/cfs/master/data/wal",

 "storeDir":"/cfs/master/data/store",

 "exporterPort": 9500,

 "consulAddr": "http://consul.prometheus-cfs.local",

 "clusterName":"chubaofs01",

 "metaNodeReservedMem": "1073741824"

}

```

### Start Service[¶](#start-service "Permalink to this headline")

```

nohup ./master -c config.json > nohup.out &

```

Meta Subsystem[¶](#meta-subsystem "Permalink to this headline")

---------------------------------------------------------------

Metanode is the manager of meta partitions and replicated by MultiRaft. Each metanode manages various of partitions. Each partition covers an inode range, and maintains two in-memory btrees: inode btree and dentry btree.

At lease 3 meta nodes are required in respect to high availability.

Properties[¶](#id1 "Permalink to this table")

| Key | Type | Description | Mandatory |  |

| --- | --- | --- | --- | --- |

| role | string | Role of process and must be set to *metanode* | Yes |  |

| listen | string | Listen and accept port of the server | Yes |  |

| prof | string | Pprof port | Yes |  |

| localIP | string | IP of network to be choose | No. If not specified, the ip address used to communicate with the master is used. |  |

| logLevel | string | Level operation for logging. Default is *error* | No |  |

| metadataDir | string | MetaNode store snapshot directory | Yes |  |

| logDir | string | Log directory | Yes |  |

| raftDir | string | Raft wal directory | Yes |  |

| raftHeartbeatPort | string | Raft heartbeat port | Yes |  |

| raftReplicaPort | string | Raft replicate port | Yes |  |

| consulAddr | string | Addresses of monitor system | No |  |

| exporterPort | string | Port for monitor system | No |  |

| masterAddr | string | Addresses of master server | Yes |  |

| zoneName | string | Specified zone. `default` by default. | No |  |

| totalMem | string | Max memory metadata used. The value needs to be higher than the value of *metaNodeReservedMem* in the master configuration. Unit: byte | Yes |  |

| deleteBatchCount | int64 | when deleting inodes, how many are deleted at a time ,500 by default | No |  |

Example:

```

{

     "role": "metanode",

     "listen": "17210",

     "prof": "17220",

     "logLevel": "debug",

     "metadataDir": "/cfs/metanode/data/meta",

     "logDir": "/cfs/metanode/log",

     "raftDir": "/cfs/metanode/data/raft",

     "raftHeartbeatPort": "17230",

     "raftReplicaPort": "17240",

     "consulAddr": "http://consul.prometheus-cfs.local",

     "exporterPort": 9501,

     "totalMem":  "8589934592",

     "masterAddr": [

         "10.196.59.198:17010",

         "10.196.59.199:17010",

         "10.196.59.200:17010"

     ]

 }

```

### Notice[¶](#notice "Permalink to this headline")

> 

> * listen, raftHeartbeatPort, raftReplicaPort can’t be modified after boot startup first time;

> * Above config would be stored under directory raftDir in constcfg file. If need modified forcely，you must delete this file manually;

> * These configuration items associated with master’s metanode infomation . If they have been modified, master would’t be found old metanode;

> 

> 

> 

Data Subsystem[¶](#data-subsystem "Permalink to this headline")

---------------------------------------------------------------

### How To Start DataNode[¶](#how-to-start-datanode "Permalink to this headline")

Start a DataNode process by execute the server binary of ChubaoFS you built with `-c` argument and specify configuration file. At least 4 data nodes are required in respect to high availability.

```

nohup cfs-server -c datanode.json &

```

### Configurations[¶](#configurations "Permalink to this headline")

Properties[¶](#id1 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| role | string | Role of process and must be set to *datanode* | Yes |

| listen | string | Port of TCP network to be listen | Yes |

| localIP | string | IP of network to be choose | No,If not specified, the ip address used to communicate with the master is used. |

| prof | string | Port of HTTP based prof and api service | Yes |

| logDir | string | Path for log file storage | Yes |

| logLevel | string | Level operation for logging. Default is *error* | No |

| raftHeartbeat | string | Port of raft heartbeat TCP network to be listen | Yes |

| raftReplica | string | Port of raft replicate TCP network to be listen | Yes |

| raftDir | string | Path for raft log file storage | No |

| consulAddr | string | Addresses of monitor system | No |

| exporterPort | string | Port for monitor system | No |

| masterAddr | string slice | Addresses of master server | Yes |

| zoneName | string | Specified zone. `default` by default. | No |

| disks | string slice | 

Format: *PATH:RETAIN*.

PATH: Disk mount point. RETAIN: Retain space. (Ranges: 20G-50G.)

 | Yes |

**Example:**

```

{

    "role": "datanode",

    "listen": "17310",

    "prof": "17320",

    "logDir": "/cfs/datanode/log",

    "logLevel": "info",

    "raftHeartbeat": "17330",

    "raftReplica": "17340",

    "raftDir": "/cfs/datanode/log",

    "consulAddr": "http://consul.prometheus-cfs.local",

    "exporterPort": 9502,

    "masterAddr": [

         "10.196.59.198:17010",

         "10.196.59.199:17010",

         "10.196.59.200:17010"

    ],

     "disks": [

        "/data0:10737418240",

        "/data1:10737418240"

    ]

}

```

### Notice[¶](#notice "Permalink to this headline")

> 

> * Since datanode uses **SEEK\_HOLE** and **SEEK\_DATA** operations which is supported by XFS (since Linux 3.5) and ext4 (since Linux 3.8), users should pay attention to the Linux kernel version on which datanodes are deployed.

> * listen, raftHeartbeat, raftReplica can’t be modified after boot startup first time.

> * Above config would be stored under directory raftDir in constcfg file. If need modified forcely, you must delete this file manually.

> * These configuration items associated with master’s datanode infomation. If they have been modified, master would’t be found old datanode.

> 

> 

> 

Object Subsystem (ObjectNode)[¶](#object-subsystem-objectnode "Permalink to this headline")

-------------------------------------------------------------------------------------------

### How To start ObjectNode[¶](#how-to-start-objectnode "Permalink to this headline")

Start a ObjectNode process by execute the server binary of ChubaoFS you built with `-c` argument and specify configuration file.

```

nohup cfs-server -c objectnode.json &

```

*Note: If you do not intend to use the object storage service, you do not need to start the ObjectNode.*

### Configurations[¶](#configurations "Permalink to this headline")

Object Node using JSON format configuration file.

**Properties**

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| role | string | Role of process and must be set to `objectnode` | Yes |

| listen | string | Listen and accept port of the server | Yes |

| domains | string slice | 

Domain of S3-like interface which makes wildcard domain support

Format: `DOMAIN`

 | No |

| logDir | string | Log directory | Yes |

| logLevel | string | 

Level operation for logging.

Default: `error`

 | No |

| masterAddr | string slice | 

Format: `HOST:PORT`.

HOST: Hostname, domain or IP address of master (resource manager).

PORT: port number which listened by this master

 | Yes |

| authNodes | string slice | 

Format: *HOST:PORT*.

HOST: Hostname, domain or IP address of AuthNode.

PORT: port number which listened by this AuthNode

 | Yes |

| exporterPort | string | Port for monitor system | No |

| prof | string | Pprof port | Yes |

**Example:**

```

{

     "role": "objectnode",

     "listen": "17410",

     "domains": [

         "object.cfs.local"

     ],

     "logDir": "/cfs/Logs/objectnode",

     "logLevel": "info",

     "masterAddr": [

         "10.196.59.198:17010",

         "10.196.59.199:17010",

         "10.196.59.200:17010"

     ],

     "exporterPort": 9503,

     "prof": "7013"

}

```

### Fetch Authentication Keys[¶](#fetch-authentication-keys "Permalink to this headline")

First, you need create a user through **User API**, and then get keys information. Refer to [User](index.html#document-admin-api/master/user).

You can also use Command Line Interface (CLI) tool to create and get user’s AccessKey and SecretKey:

```

$ cli user create [USER_ID]

$ cli user info [USER_ID]

```

### Using Object Storage Interface[¶](#using-object-storage-interface "Permalink to this headline")

Object Subsystem (ObjectNode) provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

For detail about list of supported APIs, see **Supported S3-compatible APIs** at [Object Subsystem (ObjectNode)](index.html#document-design/objectnode)

For detail about list of supported SDKs, see **Supported SDKs** at [Object Subsystem (ObjectNode)](index.html#document-design/objectnode)

#### Using S3cmd[¶](#using-s3cmd "Permalink to this headline")

Use s3cmd to access the ObjectNode deployed locally.

**Installation**

Install the `s3cmd` from <https://s3tools.org/s3cmd> .

**Configuration**

Edit s3cmd configuration file `$HOME/.s3cfg`

```

host\_base = 127.0.0.1

host\_bucket = 127.0.0.1

use\_https = False

access\_key = YOUR_ACCESS_KEY

secret\_key = YOUR_SECRET_KEY

```

**Example: making a bucket (volume)**

```

s3cmd mb s3://my_volume

Bucket 's3://my\_volume/' created

```

**Example: uploading an local file to ChubaoFS**

```

s3cmd put data.dat s3://my_volume

upload: 'data.dat' -> 's3://my\_volume/data.dat'

```

**Example: listing buckets (volumes)**

```

s3cmd ls

2020-05-10 15:29 s3://my_volume

```

**Example: listing files stored in a ChubaoFS volume**

```

s3cmd ls s3://my_volume

                         DIR    s3://my_volume/backup/

2020-05-10  15:31   10485760    s3://my_volume/data.dat

2020005-10  15:33         10    s3://my_volume/hello.txt

```

**Example: deleting file stored in a ChubaoFS volume**

```

s3cmd rm s3://my_volume/data.dat

delete: 's3://my\_volume/data.dat'

```

Detail usage for `s3cmd` see <https://s3tools.org/usage> .

#### Using AWS Java SDK[¶](#using-aws-java-sdk "Permalink to this headline")

Use AWS Java SDK to access the ObjectNode deployed locally.

**Install by Maven:**

```

<dependency>

    <groupId>software.amazon.awssdk</groupId>

    <artifactId>s3</artifactId>

    <version>2.10.71</version>

</dependency>

```

**Example: uploading file to ChubaoFS volume (PutObject)**

```

Regions clientRegion = Region.of("\*\*\* Region name \*\*\*"); // Setup region (the cluster name)

String endpoint = "http://127.0.0.1"; // Setup endpoint

String accessKey = "\*\*\* Access Key \*\*\*"; // Setup AccessKey

String secretKey = "\*\*\* Secret Key \*\*\*"; // Setup SecretKey

String bucketName = "\*\*\* Bucket name \*\*\*"; // Setup target bucket (ChubaoFS Volume)

String objectKey = "\*\*\* File object key name \*\*\*"; // Setup object key

[]byte data = "\*\*\* Example File Data as String \*\*".getBytes();

try {

    // Setup credential

    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client

    S3Configuration configuration = S3Configuration.builder()

        .chunkedEncodingEnabled(true)

        .build();

    S3Client client = S3Client.builder()

        .region(region)

        .credentialsProvider(credentialsProvider)

        .endpointOverride(URI.create(endpoint))

        .serviceConfiguration(configuration)

        .build();

    // Upload file

    PutObjectRequest request = PutObjectRequest.builder()

        .bucket(bucketName)

        .key(objectKey)

        .build();

    RequestBody body = RequestBody.fromBytes(data);

    s3Client.putObject(request, body)

} catch (Exception e) {

    e.printStackTrace();

}

```

**Example: downloading file stored in ChubaoFS volume (GetObject)**

```

Regions clientRegion = Region.of("\*\*\* Region name \*\*\*"); // Setup region

String endpoint = "http://127.0.0.1"; // Setup endpoint

String accessKey = "\*\*\* Access Key \*\*\*"; // Setup AccessKey

String secretKey = "\*\*\* Secret Key \*\*\*"; // Setup SecretKey

String bucketName = "\*\*\* Bucket name \*\*\*"; // Setup target bucket (ChubaoFS Volume)

String objectKey = "\*\*\* File object key name \*\*\*"; // Setup object key

try {

    // Setup credential

    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client

    S3Configuration configuration = S3Configuration.builder()

        .chunkedEncodingEnabled(true)

        .build();

    S3Client client = S3Client.builder()

        .region(region)

        .credentialsProvider(credentialsProvider)

        .endpointOverride(URI.create(endpoint))

        .serviceConfiguration(configuration)

        .build();

    // Get file data

    GetObjectRequest request = GetObjectRequest.builder()

        .bucket(bucketName)

        .key(objectKey)

        .build();

    InputStream is = s3Client.getObject(request)

    while (true) {

        if (is.read() == -1) {

            break

        }

    }

} catch (Exception e) {

    e.printStackTrace();

}

```

#### Using AWS Python SDK (Boto3)[¶](#using-aws-python-sdk-boto3 "Permalink to this headline")

Use AWS Python SDK (Boto3) to access the ObjectNode deployed locally.

**Install Boto3 by PIP:**

```

$ pip install boto3

```

**Example: uploading file to ChubaoFS volume (PutObject)**

```

import boto3

endpoint = " \*\* endpoint url \*\* "  # example: http://127.0.0.1

region\_name = " \*\* region name \*\* "

access\_key = " \*\* your access key \*\* "  # your access key

secret\_key = " \*\* your secret key \*\* "  # your secret key

bucket = " \*\* your bucket (volume) \*\* "  # your volume name

key = " \*\* your object key (file path in CFS) \*\* "  # your object name

def put\_file():

    session = boto3.Session(

        aws\_access\_key\_id=access\_key,

        aws\_secret\_access\_key=secret\_key)

    client = session.client(service\_name="s3", region\_name=region\_name, endpoint\_url=endpoint)

    client.put\_object(Bucket=bucket, Key=key, Body=bytes("hello world"))

```

**Example: downloading file stored in ChubaoFS volume (GetObject)**

```

import boto3

endpoint = " \*\* endpoint url \*\* "  # example: http://127.0.0.1

region\_name = " \*\* region name \*\* "

access\_key = " \*\* your access key \*\* "  # your access key

secret\_key = " \*\* your secret key \*\* "  # your secret key

bucket = " \*\* your bucket (volume) \*\* "  # your volume name

key = " \*\* your object key (file path in CFS) \*\* "  # your object name

def get\_file():

    session = boto3.Session(

        aws\_access\_key\_id=access\_key,

        aws\_secret\_access\_key=secret\_key)

    client = session.client(service\_name="s3", region\_name=region\_name, endpoint\_url=endpoint)

    result = client.get\_object(Bucket=bucket, Key=key)

    print(result["Body"].read())

```

Console[¶](#console "Permalink to this headline")

-------------------------------------------------

### How To Start Console[¶](#how-to-start-console "Permalink to this headline")

Start a Console process by execute the server binary of ChubaoFS you built with `-c` argument and specify configuration file.

```

nohup cfs-server -c console.json &

```

### Configurations[¶](#configurations "Permalink to this headline")

Properties[¶](#id1 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| role | string | Role of process and must be set to *console* | Yes |

| logDir | string | Path for log file storage | Yes |

| logLevel | string | Level operation for logging. Default is *error* | No |

| listen | string | Port of TCP network to be listen, default is 80 | Yes |

| masterAddr | string slice | Addresses of master server | Yes |

| objectNodeDomain | string | object domain for sign url for down | Yes |

| monitor\_addr | string | Prometheus the address | Yes |

| dashboard\_addr | string | console menu forward to Grafana | Yes |

| monitor\_app | string | the tag for monitor, it same as master config | Yes |

| monitor\_cluster | string | the tag for monitor, it same as master config | Yes |

**Example:**

```

{

  "role": "console",

  "logDir": "/cfs/log/",

  "logLevel": "debug",

  "listen": "80",

  "masterAddr": [

    "192.168.0.11:17010",

    "192.168.0.12:17010",

    "192.168.0.13:17010"

  ],

  "monitor\_addr": "http://192.168.0.102:9090",

  "dashboard\_addr": "http://grafana.chubao.io",

  "monitor\_app": "cfs",

  "monitor\_cluster": "spark"

}

```

### Notice[¶](#notice "Permalink to this headline")

> 

> * add 127.0.0.1 console.chubao.io grafana.chubao.io in /etc/hosts

> * you can visit it by http://127.0.0.1:80

> * in console default user is root default password is ChubaoFSRoot

> * If you are upgrading your ChubaoFS from v2.0.0 or earlier version, the password may not be compatible, you can use curl -H “Content-Type:application/json” -X POST –data ‘{“id”:”testuser”,”pwd”:”12345”,”type”:2}’ “http://10.196.59.198:17010/user/create” to create new user to use it

> 

> 

> 

Client[¶](#client "Permalink to this headline")

-----------------------------------------------

### Prerequisite[¶](#prerequisite "Permalink to this headline")

Insert FUSE kernel module and install libfuse.

```

modprobe fuse

yum install -y fuse

```

### Prepare Config File[¶](#prepare-config-file "Permalink to this headline")

fuse.json

```

{

  "mountPoint": "/cfs/mountpoint",

  "volName": "ltptest",

  "owner": "ltptest",

  "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",

  "logDir": "/cfs/client/log",

  "logLevel": "info",

  "profPort": "27510"

}

```

Supported Configurations[¶](#id1 "Permalink to this table")

| Name | Type | Description | Mandatory |

| --- | --- | --- | --- |

| mountPoint | string | Mount point | Yes |

| volName | string | Volume name | Yes |

| owner | string | Owner name as authentication | Yes |

| masterAddr | string | Resource manager IP address | Yes |

| logDir | string | Path to store log files | No |

| logLevel | string | Log level：debug, info, warn, error | No |

| profPort | string | Golang pprof port | No |

| exporterPort | string | Performance monitor port | No |

| consulAddr | string | Performance monitor server address | No |

| lookupValid | string | Lookup valid duration in FUSE kernel module, unit: sec | No |

| attrValid | string | Attr valid duration in FUSE kernel module, unit: sec | No |

| icacheTimeout | string | Inode cache valid duration in client | No |

| enSyncWrite | string | Enable DirectIO sync write, i.e. make sure data is fsynced in data node | No |

| autoInvalData | string | Use AutoInvalData FUSE mount option | No |

| rdonly | bool | Mount as read-only file system | No |

| writecache | bool | Leverage the write cache feature of kernel FUSE. Requires the kernel FUSE module to support write cache. | No |

| keepcache | bool | Keep kernel page cache. Requires the writecache option is enabled. | No |

| token | string | Specify the capability of a client instance. | No |

| readRate | int | Read Rate Limit. Unlimited by default. | No |

| writeRate | int | Write Rate Limit. Unlimited by default. | No |

| followerRead | bool | Enable read from follower. False by default. | No |

| accessKey | string | Access key of user who owns the volume. | No |

| secretKey | string | Secret key of user who owns the volume. | No |

| disableDcache | bool | Disable Dentry Cache. False by default. | No |

| subdir | string | Mount sub directory. | No |

| fsyncOnClose | bool | Perform fsync upon file close. True by default. | No |

| maxcpus | int | The maximum number of available CPU cores. Limit the CPU usage of the client process. | No |

| enableXattr | bool | Enable xattr support. False by default. | No |

### Mount[¶](#mount "Permalink to this headline")

Use the example *fuse.json*, and client is mounted on the directory */mnt/fuse*. All operations to */mnt/fuse* would be performed on the backing distributed file system.

```

./cfs-client -c fuse.json

```

### Unmount[¶](#unmount "Permalink to this headline")

It is recommended to use standard Linux `umount` command to terminate the mount.

Monitor[¶](#monitor "Permalink to this headline")

-------------------------------------------------

ChubaoFS use prometheus as metrics collector. It simply config as follow in master, metanode, datanode, client’s config file：

```

{

    "exporterPort": 9505,

    "consulAddr": "http://consul.prometheus-cfs.local"

}

```

* exporterPort：prometheus exporter Port. when set, can export prometheus metrics from URL(<http://$hostip:$exporterPort/metrics>). If not set, prometheus exporter will unavailable；

* consulAddr: consul register address, it can work with prometheus to auto discover deployed ChubaoFS nodes, if not set, consul register will not work.

Using grafana as prometheus metrics web front：

![_images/cfs-grafana-dashboard.png](_images/cfs-grafana-dashboard.png)

Also, user can use prometheus alertmanager to capture and route alerts to the correct receiver. please visit prometheus alertmanger [web-doc](https://prometheus.io/docs/alerting/alertmanager/)

### Metrics[¶](#metrics "Permalink to this headline")

* Cluster

> 

> 

> 	+ The number of nodes: `MasterCount` , `MetaNodeCount` , `DataNodeCount` , `ObjectNodeCount`

> 	+ The number of clients: `ClientCount`

> 	+ The number of volumes: `VolumeCount`

> 	+ The size of nodes: `DataNodeSize` , `MetaNodeSize`

> 	+ The used ratio of nodes: `DataNodeUsedRatio` , `MetaNodeUsedRatio`

> 	+ The number of inactive nodes: `DataNodeInactive` , `MetaNodesInactive`

> 	+ The capacity of volumes: `VolumeTotalSize`

> 	+ The used ratio of volumes: `VolumeUsedRatio`

> 	+ The number of error disks: `DiskError`

>

* Volume

> 

> 

> 	+ The used size of volumes: `VolumeUsedSize`

> 	+ The used ratio of volumes: `VolumeUsedRatio`

> 	+ The capacity change rate of volumes: `VolumeSizeRate`

>

* Master

> 

> 

> 	+ The number of invalid masters: `master\_nodes\_invalid`

> 	+ The number of invalid metanodes: `metanode\_inactive`

> 	+ The number of invalid datanodes:： `datanode\_inactive`

> 	+ The number of invalid clients:： `fuseclient\_inactive`

>

* MetaNode

> 

> 

> 	+ The duration of each operation (Time) and the number of operations per second (Ops) on the metanode, which can be selected from the `MetaNodeOp` drop-down list.

>

* DataNode

> 

> 

> 	+ The duration of each operation (Time) and the number of operations per second (Ops) on the datanode, which can be selected from the `DataNodeOp` drop-down list.

>

* ObjectNode

> 

> 

> 	+ The duration of each operation (Time) and the number of operations per second (Ops) on the objectnode, which can be selected from the `objectNodeOp` drop-down list.

>

* FuseClient

> 

> 

> 	+ The duration of each operation (Time) and the number of operations per second (Ops) on the client, which can be selected from the `fuseOp` drop-down list.

>

*Recommended focus metrics: cluster status, node or disk failure, total size, growth rate, etc.*

### Grafana DashBoard Config[¶](#grafana-dashboard-config "Permalink to this headline")

```

{

  "annotations": {

    "list": [

      {

        "builtIn": 1,

        "datasource": "-- Grafana --",

        "enable": true,

        "hide": true,

        "iconColor": "rgba(0, 211, 255, 1)",

        "name": "Annotations & Alerts",

        "type": "dashboard"

      }

    ]

  },

  "editable": true,

  "gnetId": null,

  "graphTooltip": 0,

  "id": 21,

  "iteration": 1578879879321,

  "links": [],

  "panels": [

    {

      "collapsed": false,

      "gridPos": {

        "h": 1,

        "w": 24,

        "x": 0,

        "y": 0

      },

      "id": 40,

      "panels": [],

      "title": "Cluster",

      "type": "row"

    },

    {

      "cacheTimeout": null,

      "colorBackground": false,

      "colorValue": false,

      "colors": [

        "#299c46",

        "rgba(237, 129, 40, 0.89)",

        "#d44a3a"

      ],

      "datasource": null,

      "description": "master node total count of cluster",

      "format": "none",

      "gauge": {

        "maxValue": 100,

        "minValue": 0,

        "show": false,

        "thresholdLabels": false,

        "thresholdMarkers": true

      },

      "gridPos": {

        "h": 4,

        "w": 4,

        "x": 0,

        "y": 1

      },

      "id": 38,

      "interval": null,

      "links": [],

      "mappingType": 1,

      "mappingTypes": [

        {

          "name": "value to text",

          "value": 1

        },

        {

          "name": "range to text",

          "value": 2

        }

      ],

      "maxDataPoints": 100,

      "nullPointMode": "connected",

      "nullText": null,

      "postfix": "",

      "postfixFontSize": "50%",

      "prefix": "",

      "prefixFontSize": "50%",

      "rangeMaps": [

        {

          "from": "null",

          "text": "N/A",

          "to": "null"

        }

      ],

      "sparkline": {

        "fillColor": "rgba(31, 118, 189, 0.18)",

        "full": false,

        "lineColor": "rgb(31, 120, 193)",

        "show": true

      },

      "tableColumn": "",

      "targets": [

        {

          "expr": "count(go\_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"master\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "refId": "A"

        }

      ],

      "thresholds": "",

      "title": "MasterCount",

      "type": "singlestat",

      "valueFontSize": "80%",

      "valueMaps": [

        {

          "op": "=",

          "text": "N/A",

          "value": "null"

        }

      ],

      "valueName": "current"

    },

    {

      "cacheTimeout": null,

      "colorBackground": false,

      "colorValue": false,

      "colors": [

        "#299c46",

        "rgba(237, 129, 40, 0.89)",

        "#d44a3a"

      ],

      "datasource": null,

      "format": "none",

      "gauge": {

        "maxValue": 100,

        "minValue": 0,

        "show": false,

        "thresholdLabels": false,

        "thresholdMarkers": true

      },

      "gridPos": {

        "h": 4,

        "w": 4,

        "x": 4,

        "y": 1

      },

      "id": 42,

      "interval": null,

      "links": [],

      "mappingType": 1,

      "mappingTypes": [

        {

          "name": "value to text",

          "value": 1

        },

        {

          "name": "range to text",

          "value": 2

        }

      ],

      "maxDataPoints": 100,

      "nullPointMode": "connected",

      "nullText": null,

      "postfix": "",

      "postfixFontSize": "50%",

      "prefix": "",

      "prefixFontSize": "50%",

      "rangeMaps": [

        {

          "from": "null",

          "text": "N/A",

          "to": "null"

        }

      ],

      "sparkline": {

        "fillColor": "rgba(31, 118, 189, 0.18)",

        "full": false,

        "lineColor": "rgb(31, 120, 193)",

        "show": true

      },

      "tableColumn": "",

      "targets": [

        {

          "expr": "count(go\_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"metanode\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "refId": "A"

        }

      ],

      "thresholds": "",

      "title": "MetanodeCount",

      "type": "singlestat",

      "valueFontSize": "80%",

      "valueMaps": [

        {

          "op": "=",

          "text": "N/A",

          "value": "null"

        }

      ],

      "valueName": "current"

    },

    {

      "cacheTimeout": null,

      "colorBackground": false,

      "colorValue": false,

      "colors": [

        "#299c46",

        "rgba(237, 129, 40, 0.89)",

        "#d44a3a"

      ],

      "datasource": "default",

      "format": "none",

      "gauge": {

        "maxValue": 100,

        "minValue": 0,

        "show": false,

        "thresholdLabels": false,

        "thresholdMarkers": true

      },

      "gridPos": {

        "h": 4,

        "w": 4,

        "x": 8,

        "y": 1

      },

      "id": 41,

      "interval": null,

      "links": [],

      "mappingType": 1,

      "mappingTypes": [

        {

          "name": "value to text",

          "value": 1

        },

        {

          "name": "range to text",

          "value": 2

        }

      ],

      "maxDataPoints": 100,

      "nullPointMode": "connected",

      "nullText": null,

      "postfix": "",

      "postfixFontSize": "50%",

      "prefix": "",

      "prefixFontSize": "50%",

      "rangeMaps": [

        {

          "from": "null",

          "text": "N/A",

          "to": "null"

        }

      ],

      "sparkline": {

        "fillColor": "rgba(31, 118, 189, 0.18)",

        "full": false,

        "lineColor": "rgb(31, 120, 193)",

        "show": true

      },

      "tableColumn": "",

      "targets": [

        {

          "expr": "count(go\_info{cluster=\"$cluster\", app=\"$app\", role=\"dataNode\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "refId": "A"

        }

      ],

      "thresholds": "",

      "title": "DatanodeCount",

      "type": "singlestat",

      "valueFontSize": "80%",

      "valueMaps": [

        {

          "op": "=",

          "text": "N/A",

          "value": "null"

        }

      ],

      "valueName": "current"

    },

    {

      "cacheTimeout": null,

      "colorBackground": false,

      "colorValue": false,

      "colors": [

        "#299c46",

        "rgba(237, 129, 40, 0.89)",

        "#d44a3a"

      ],

      "datasource": "default",

      "format": "none",

      "gauge": {

        "maxValue": 100,

        "minValue": 0,

        "show": false,

        "thresholdLabels": false,

        "thresholdMarkers": true

      },

      "gridPos": {

        "h": 4,

        "w": 3,

        "x": 12,

        "y": 1

      },

      "id": 86,

      "interval": null,

      "links": [],

      "mappingType": 1,

      "mappingTypes": [

        {

          "name": "value to text",

          "value": 1

        },

        {

          "name": "range to text",

          "value": 2

        }

      ],

      "maxDataPoints": 100,

      "nullPointMode": "connected",

      "nullText": null,

      "postfix": "",

      "postfixFontSize": "50%",

      "prefix": "",

      "prefixFontSize": "50%",

      "rangeMaps": [

        {

          "from": "null",

          "text": "N/A",

          "to": "null"

        }

      ],

      "sparkline": {

        "fillColor": "rgba(31, 118, 189, 0.18)",

        "full": false,

        "lineColor": "rgb(31, 120, 193)",

        "show": true

      },

      "tableColumn": "",

      "targets": [

        {

          "expr": "count(go\_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"fuseclient\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "refId": "A"

        }

      ],

      "thresholds": "",

      "title": "ClientCount",

      "type": "singlestat",

      "valueFontSize": "80%",

      "valueMaps": [

        {

          "op": "=",

          "text": "N/A",

          "value": "null"

        }

      ],

      "valueName": "current"

    },

    {

      "cacheTimeout": null,

      "colorBackground": false,

      "colorValue": false,

      "colors": [

        "#299c46",

        "rgba(237, 129, 40, 0.89)",

        "#d44a3a"

      ],

      "datasource": "default",

      "description": "Cluster volume total count",

      "format": "none",

      "gauge": {

        "maxValue": 100,

        "minValue": 0,

        "show": false,

        "thresholdLabels": false,

        "thresholdMarkers": true

      },

      "gridPos": {

        "h": 4,

        "w": 3,

        "x": 15,

        "y": 1

      },

      "id": 93,

      "interval": null,

      "links": [],

      "mappingType": 1,

      "mappingTypes": [

        {

          "name": "value to text",

          "value": 1

        },

        {

          "name": "range to text",

          "value": 2

        }

      ],

      "maxDataPoints": 100,

      "nullPointMode": "connected",

      "nullText": null,

      "postfix": "",

      "postfixFontSize": "50%",

      "prefix": "",

      "prefixFontSize": "50%",

      "rangeMaps": [

        {

          "from": "null",

          "text": "N/A",

          "to": "null"

        }

      ],

      "sparkline": {

        "fillColor": "rgba(31, 118, 189, 0.18)",

        "full": false,

        "lineColor": "rgb(31, 120, 193)",

        "show": true

      },

      "tableColumn": "",

      "targets": [

        {

          "expr": "sum([[app]]\_master\_vol\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "refId": "A"

        }

      ],

      "thresholds": "",

      "title": "VolumeCount",

      "type": "singlestat",

      "valueFontSize": "80%",

      "valueMaps": [

        {

          "op": "=",

          "text": "N/A",

          "value": "null"

        }

      ],

      "valueName": "current"

    },

    {

      "cacheTimeout": null,

      "colorBackground": false,

      "colorValue": false,

      "colors": [

        "#299c46",

        "rgba(237, 129, 40, 0.89)",

        "#d44a3a"

      ],

      "datasource": null,

      "decimals": null,

      "format": "dateTimeFromNow",

      "gauge": {

        "maxValue": 100,

        "minValue": 0,

        "show": false,

        "thresholdLabels": false,

        "thresholdMarkers": true

      },

      "gridPos": {

        "h": 4,

        "w": 3,

        "x": 18,

        "y": 1

      },

      "id": 113,

      "interval": null,

      "links": [],

      "mappingType": 1,

      "mappingTypes": [

        {

          "name": "value to text",

          "value": 1

        },

        {

          "name": "range to text",

          "value": 2

        }

      ],

      "maxDataPoints": 100,

      "nullPointMode": "connected",

      "nullText": null,

      "postfix": "",

      "postfixFontSize": "50%",

      "prefix": "",

      "prefixFontSize": "50%",

      "rangeMaps": [

        {

          "from": "null",

          "text": "N/A",

          "to": "null"

        }

      ],

      "sparkline": {

        "fillColor": "rgba(31, 118, 189, 0.18)",

        "full": false,

        "lineColor": "rgb(31, 120, 193)",

        "show": true

      },

      "tableColumn": "",

      "targets": [

        {

          "expr": "process\_start\_time\_seconds{cluster=\"$cluster\", app=\"$app\", instance=~\"$instance\"}\*1000",

          "format": "time\_series",

          "intervalFactor": 1,

          "refId": "A"

        }

      ],

      "thresholds": "",

      "title": "ProcessStartTime",

      "type": "singlestat",

      "valueFontSize": "80%",

      "valueMaps": [

        {

          "op": "=",

          "text": "N/A",

          "value": "null"

        }

      ],

      "valueName": "current"

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 0,

        "y": 5

      },

      "id": 115,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum(go\_info{app=\"$app\", cluster=\"$cluster\"}) by (cluster, app, role)",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "{{cluster}}-{{role}}",

          "refId": "A"

        },

        {

          "expr": "count(go\_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"$role\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "[[role]]-count",

          "refId": "B"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "NodeAlive",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 7,

        "y": 5

      },

      "id": 179,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "sum"

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum(cfs\_master\_disk\_error{app=\"$app\", cluster=\"$cluster\"} > 0)",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "sum",

          "refId": "D"

        },

        {

          "expr": "cfs\_master\_disk\_error{app=\"$app\", cluster=\"$cluster\"} > 0",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "{{addr}}\_{{path}}",

          "refId": "A"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "DiskError",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 14,

        "y": 5

      },

      "id": 180,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "cfs\_master\_metaNodes\_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "{{instance}}",

          "refId": "B"

        },

        {

          "expr": "sum(cfs\_master\_metaNodes\_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "sum",

          "refId": "A"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "MetaNodesInactive",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 0,

        "y": 11

      },

      "id": 181,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "cfs\_master\_dataNodes\_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "{{instance}}",

          "refId": "C"

        },

        {

          "expr": "sum(cfs\_master\_dataNodes\_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "sum",

          "refId": "A"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "DatanodeInactive",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 7,

        "y": 11

      },

      "id": 177,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "/used\_ratio.\*/",

          "yaxis": 2

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "cfs\_master\_dataNodes\_total\_GB{app=\"$app\",cluster=\"$cluster\"}>0",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "TotalGB",

          "refId": "B"

        },

        {

          "expr": "cfs\_master\_dataNodes\_used\_GB{app=\"$app\",cluster=\"$cluster\"}>0",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "UsedGB",

          "refId": "A"

        },

        {

          "expr": "cfs\_master\_dataNodes\_increased\_GB{app=\"$app\",cluster=\"$cluster\"}>0",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "IncreasedGB",

          "refId": "C"

        },

        {

          "expr": "sum(cfs\_master\_dataNodes\_used\_GB{app=\"$app\",cluster=\"$cluster\"}) / sum(cfs\_master\_dataNodes\_total\_GB{app=\"$app\",cluster=\"$cluster\"})",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "UsedRatio",

          "refId": "D"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "DatanodeSize",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "decgbytes",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "percentunit",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 14,

        "y": 11

      },

      "id": 92,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_master\_metaNodes\_total\_GB{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "total\_GB",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_master\_metaNodes\_used\_GB{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "used\_GB",

          "refId": "A"

        },

        {

          "expr": "sum([[app]]\_master\_metaNodes\_increased\_GB{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "increased\_GB",

          "refId": "C"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "MetanodeSize",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "decgbytes",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 0,

        "y": 17

      },

      "id": 175,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "/.\*\_ratio.\*/",

          "yaxis": 2

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "cfs\_master\_vol\_usage\_ratio{app=\"$app\",cluster=\"$cluster\"}",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "{{volName}}\_usage\_ratio",

          "refId": "B"

        },

        {

          "expr": "cfs\_master\_vol\_total\_GB{app=\"$app\",cluster=\"$cluster\"}",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "{{volName}}\_total\_GB",

          "refId": "A"

        },

        {

          "expr": "cfs\_master\_vol\_used\_GB{app=\"$app\",cluster=\"$cluster\"}",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "{{volName}}\_used\_GB",

          "refId": "C"

        },

        {

          "expr": "cfs\_master\_vol\_used\_GB{app=\"$app\",cluster=\"$cluster\"} / cfs\_master\_vol\_total\_GB{app=\"$app\",cluster=\"$cluster\"} ",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "{{volName}}\_used\_ratio",

          "refId": "D"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "VolumeSize",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "decgbytes",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "percentunit",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 7,

        "y": 17

      },

      "id": 90,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_master\_vol\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "vol\_count",

          "refId": "M"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "VolumeCount",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 14,

        "y": 17

      },

      "id": 73,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "/.\*\_QPS/",

          "yaxis": 2

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_metanode\_OpMetaOpen{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Open",

          "refId": "A"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaLookup{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Lookup",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaNodeHeartbeat{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "NodeHeartbeat",

          "refId": "C"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaReadDir{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "ReadDir",

          "refId": "D"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaReleaseOpen{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "ReleaseOpen",

          "refId": "E"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaSetattr{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Setattr",

          "refId": "F"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaTruncate{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Truncate",

          "refId": "G"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaOpen{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Open\_QPS",

          "refId": "H"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaLookup{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Lookup\_QPS",

          "refId": "I"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaReadDir{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "ReadDir\_QPS",

          "refId": "K"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Truncate\_QPS",

          "refId": "L"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaReleaseOpen{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "ReleaseOpen\_QPS",

          "refId": "J"

        },

        {

          "expr": "sum(rate([[app]]cfs\_metanode\_OpMetaSetattr{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Setattr\_QPS",

          "refId": "M"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "MetanodeOperations",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "ns",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "ops",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 0,

        "y": 23

      },

      "id": 71,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "/.\*\_QPS/",

          "yaxis": 2

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_metanode\_OpMetaBatchInodeGet{cluster=~\"$cluster\", instance=\"$instance\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "BatchGet",

          "refId": "A"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaCreateInode{cluster=~\"$cluster\", instance=\"$instance\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Create",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaDeleteInode{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Delete",

          "refId": "C"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaEvictInode{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Evict",

          "refId": "D"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaInodeGet{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "GetInode",

          "refId": "E"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaLinkInode{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "LinkInode",

          "refId": "F"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaCreateInode{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "CreateInode\_QPS",

          "refId": "G"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaDeleteInode{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "DeleteInode\_QPS",

          "refId": "H"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaEvictInode{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "EvictInode\_QPS",

          "refId": "I"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaInodeGet{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "GetInode\_QPS",

          "refId": "J"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaLinkInode{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "LinkInode\_QPS",

          "refId": "K"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpBatchInodeGet{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "BantchInodeGet\_QPS",

          "refId": "L"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaTruncate{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "TruncateInode",

          "refId": "M"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "TruncateInode\_QPS",

          "refId": "N"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaUnlinkInode{cluster=~\"$cluster\", instance=\"$instance\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "UnlinkInode",

          "refId": "O"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaUnlinkInode{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "UnlinkInnode\_QPS",

          "refId": "P"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "MetanodeInodeOperations",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "ns",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "ops",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 7,

        "y": 23

      },

      "id": 45,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "/.\*\_QPS/",

          "yaxis": 2

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_metanode\_OpMetaCreateDentry{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "CreateDentryCreateDentry",

          "refId": "A"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaDeleteDentry{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "DeleteDentry",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpMetaUpdateDentry{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "UpdateDentry",

          "refId": "C"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaCreateDentry{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "CreateDentry\_QPS",

          "refId": "D"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaDeleteDentry{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "DeleteDentry\_QPS",

          "refId": "E"

        },

        {

          "expr": "sum(rate([[app]]\_metanode\_OpMetaUpdateDentry{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "UpdateDentry\_QPS",

          "refId": "F"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "MetanodeDentryOperation",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "ns",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "ops",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 14,

        "y": 23

      },

      "id": 79,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "/.\*\_QPS/",

          "yaxis": 2

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_dataNode\_OpCreateExtent{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "CreateExtent",

          "refId": "A"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpMarkDelete{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "MarkDeleteExtent",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpRandomWrite{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "RandomWrite",

          "refId": "D"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpWrite{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "AppendWrite",

          "refId": "C"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpStreamRead{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Read",

          "refId": "F"

        },

        {

          "expr": "sum(rate([[app]]\_datanode\_OpCreateExtent{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "CreateExtent\_QPS",

          "refId": "E"

        },

        {

          "expr": "sum(rate([[app]]\_dataNode\_OpMarkDelete{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "MarkDeleteExtent\_QPS",

          "refId": "G"

        },

        {

          "expr": "sum(rate([[app]]\_dataNode\_OpRandomWrite{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "RandomWrite\_QPS",

          "refId": "H"

        },

        {

          "expr": "sum(rate([[app]]\_dataNode\_OpWrite{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "AppendWrite\_QPS",

          "refId": "I"

        },

        {

          "expr": "sum(rate([[app]]\_dataNode\_OpStreamRead{cluster=~\"$cluster\"}[1m]))",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Read\_QPS",

          "refId": "J"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "DatanodeExtentOperations",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "ns",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "ops",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 0,

        "y": 29

      },

      "id": 70,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_metanode\_OpCreateMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Create",

          "refId": "A"

        },

        {

          "expr": "sum([[app]]\_metanode\_OpLoadMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "Load",

          "refId": "B"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "MetapartitionOperations",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "ns",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": false,

      "dashLength": 10,

      "dashes": false,

      "datasource": "default",

      "fill": 1,

      "gridPos": {

        "h": 6,

        "w": 7,

        "x": 7,

        "y": 29

      },

      "id": 75,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": true,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "sum([[app]]\_dataNode\_OpLoadDataPartition{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "LoadDataPartition",

          "refId": "G"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpDataNodeHeartbeat{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "DataNodeHeartbeat",

          "refId": "F"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpCreateDataPartition{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "CreateDataPartition",

          "refId": "A"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpAddDataPartitionRaftMember{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "AddDataPartitionRaftMember",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_dataNode\_OpDeleteDataPartition{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "DeleteDataPartition",

          "refId": "C"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "DatapartitionOperations",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "ns",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "collapsed": false,

      "gridPos": {

        "h": 1,

        "w": 24,

        "x": 0,

        "y": 35

      },

      "id": 34,

      "panels": [],

      "title": "Master",

      "type": "row"

    },

    {

      "aliasColors": {},

      "bars": true,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 7,

        "w": 7,

        "x": 0,

        "y": 36

      },

      "id": 162,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": false,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [

        {

          "alias": "master\_count",

          "yaxis": 2

        }

      ],

      "spaceLength": 10,

      "stack": false,

      "steppedLine": false,

      "targets": [

        {

          "expr": "go\_info{cluster=\"$cluster\", role=\"master\"}",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "[[cluster]]\_master\_{{instance}}",

          "refId": "C"

        },

        {

          "expr": "count(go\_info{cluster=~\"$cluster\", role=~\"master\"})",

          "format": "time\_series",

          "hide": false,

          "intervalFactor": 1,

          "legendFormat": "[[cluster]]\_master\_count",

          "refId": "G"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "[[role]]\_nodes",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": true,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 7,

        "w": 7,

        "x": 7,

        "y": 36

      },

      "id": 163,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": false,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": true,

      "steppedLine": false,

      "targets": [

        {

          "expr": "go\_info{cluster=\"$cluster\", role=\"metanode\"}",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "meta\_{{instance}}",

          "refId": "C"

        },

        {

          "expr": "sum([[app]]\_master\_metaNodes\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "metaNode",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_master\_dataNodes\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "datanode",

          "refId": "A"

        },

        {

          "expr": "[[app]]\_master\_metaNodes\_count",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "meta\_count",

          "refId": "D"

        },

        {

          "expr": "[[app]]\_master\_dataNodes\_count",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "datanode\_count",

          "refId": "E"

        },

        {

          "expr": "count(process\_start\_time\_seconds{role=\"fuseclient\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "client\_count",

          "refId": "F"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "[[cluster]]\_metanode\_alive",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": true,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 7,

        "w": 7,

        "x": 0,

        "y": 43

      },

      "id": 164,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": false,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": true,

      "steppedLine": false,

      "targets": [

        {

          "expr": "go\_info{cluster=\"$cluster\", role=\"dataNode\"}",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "{{instance}}",

          "refId": "C"

        },

        {

          "expr": "sum([[app]]\_master\_metaNodes\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "metaNode",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_master\_dataNodes\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "datanode",

          "refId": "A"

        },

        {

          "expr": "[[app]]\_master\_metaNodes\_count",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "meta\_count",

          "refId": "D"

        },

        {

          "expr": "[[app]]\_master\_dataNodes\_count",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "datanode\_count",

          "refId": "E"

        },

        {

          "expr": "count(process\_start\_time\_seconds{role=\"fuseclient\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "client\_count",

          "refId": "F"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "[[cluster]]\_datanode\_alive",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "aliasColors": {},

      "bars": true,

      "dashLength": 10,

      "dashes": false,

      "datasource": null,

      "fill": 1,

      "gridPos": {

        "h": 7,

        "w": 7,

        "x": 7,

        "y": 43

      },

      "id": 165,

      "legend": {

        "avg": false,

        "current": false,

        "max": false,

        "min": false,

        "show": true,

        "total": false,

        "values": false

      },

      "lines": false,

      "linewidth": 1,

      "links": [],

      "nullPointMode": "null",

      "percentage": false,

      "pointradius": 5,

      "points": false,

      "renderer": "flot",

      "seriesOverrides": [],

      "spaceLength": 10,

      "stack": true,

      "steppedLine": false,

      "targets": [

        {

          "expr": "go\_info{cluster=\"$cluster\", role=\"fuseclient\"}",

          "format": "time\_series",

          "intervalFactor": 1,

          "legendFormat": "fuseclient\_{{instance}}",

          "refId": "C"

        },

        {

          "expr": "sum([[app]]\_master\_metaNodes\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "metaNode",

          "refId": "B"

        },

        {

          "expr": "sum([[app]]\_master\_dataNodes\_count{cluster=~\"$cluster\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "datanode",

          "refId": "A"

        },

        {

          "expr": "[[app]]\_master\_metaNodes\_count",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "meta\_count",

          "refId": "D"

        },

        {

          "expr": "[[app]]\_master\_dataNodes\_count",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "datanode\_count",

          "refId": "E"

        },

        {

          "expr": "count(process\_start\_time\_seconds{role=\"fuseclient\"})",

          "format": "time\_series",

          "hide": true,

          "intervalFactor": 1,

          "legendFormat": "client\_count",

          "refId": "F"

        }

      ],

      "thresholds": [],

      "timeFrom": null,

      "timeShift": null,

      "title": "[[cluster]]\_fuseclient\_alive",

      "tooltip": {

        "shared": true,

        "sort": 0,

        "value\_type": "individual"

      },

      "type": "graph",

      "xaxis": {

        "buckets": null,

        "mode": "time",

        "name": null,

        "show": true,

        "values": []

      },

      "yaxes": [

        {

          "format": "locale",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        },

        {

          "format": "short",

          "label": null,

          "logBase": 1,

          "max": null,

          "min": null,

          "show": true

        }

      ],

      "yaxis": {

        "align": false,

        "alignLevel": null

      }

    },

    {

      "collapsed": true,

      "gridPos": {

        "h": 1,

        "w": 24,

        "x": 0,

        "y": 50

      },

      "id": 36,

      "panels": [

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 3

          },

          "id": 117,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateInode\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_metanode\_OpMetaCreateInode\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateInode\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "InodeCreate",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 3

          },

          "id": 44,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateInode",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteInode\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaEvictInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaInodeGet\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLinkInode\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "InodeBatchGet",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 3

          },

          "id": 119,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateInode",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteInode\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaEvictInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaInodeGet\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLinkInode\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "rate([[app]]\_metanode\_OpMetaLinkInode\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLinkInode\_ops\_{{instance}}",

              "refId": "G"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "InodeLink",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 11

          },

          "id": 120,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateInode",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteInode\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaEvictInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaInodeGet\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLinkInode\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "InodeEvict",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "description": "",

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 11

          },

          "id": 121,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpCreateMetaPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpLoadMetaPartition\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "LoadMetaPartition",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 11

          },

          "id": 58,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpCreateMetaPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpLoadMetaPartition\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "CreateMetaPartition",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 19

          },

          "id": 88,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaExtentsList\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaExtentsList",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 19

          },

          "id": 50,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaExtentsAdd\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaExtentsList\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaExtentsAdd",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 19

          },

          "id": 126,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaNodeHeartbeat\_{{instance}}",

              "refId": "C"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaNodeHeartbeat",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 27

          },

          "id": 72,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_{{instance}}",

              "refId": "C"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaCreateDentry",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 27

          },

          "id": 125,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaOpen\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLookup\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaNodeHeartbeat\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReadDir\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReleaseOpen\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaSetattr\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaTruncate\_{{instance}}",

              "refId": "G"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaLookup",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 27

          },

          "id": 122,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_{{instance}}",

              "refId": "C"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaDeleteDentry",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 35

          },

          "id": 124,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReadDir\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "rate([[app]]\_metanode\_OpMetaReadDir\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReadDir\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaReadDir",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 35

          },

          "id": 129,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaOpen\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLookup\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaNodeHeartbeat\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReadDir\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReleaseOpen\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaSetattr\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaTruncate\_{{instance}}",

              "refId": "G"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaReleaseOpen",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 35

          },

          "id": 116,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateInode",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteInode\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaEvictInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaInodeGet\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLinkInode\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaDeleteInode",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 43

          },

          "id": 127,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaOpen\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLookup\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaNodeHeartbeat\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReadDir\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReleaseOpen\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaSetattr\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaTruncate\_{{instance}}",

              "refId": "G"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaTruncate",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 43

          },

          "id": 128,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaOpen\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLookup\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaNodeHeartbeat\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReadDir\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReleaseOpen\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaSetattr\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaTruncate\_{{instance}}",

              "refId": "G"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaSetattr",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 43

          },

          "id": 123,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_{{instance}}",

              "refId": "C"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaUpdateDentry",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 51

          },

          "id": 46,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_metanode\_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaOpen\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaLookup\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaNodeHeartbeat\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReadDir\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaReleaseOpen\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaSetattr\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_metanode\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaTruncate\_{{instance}}",

              "refId": "G"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaOpen",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        }

      ],

      "title": "Metanode",

      "type": "row"

    },

    {

      "collapsed": true,

      "gridPos": {

        "h": 1,

        "w": 24,

        "x": 0,

        "y": 51

      },

      "id": 27,

      "panels": [

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "description": "",

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 52

          },

          "id": 132,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_ops\_{{instance}}",

              "refId": "Q"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "DatanodeHeartbeat",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 52

          },

          "id": 133,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpCreateDataPartition\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DecommissionDataPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetPartitionSize\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_dataNode\_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "[[app]]\_dataNode\_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MarkDelete\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "[[app]]\_dataNode\_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "NotifyReplicasToRepair\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_dataNode\_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpRandomWrite\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_dataNode\_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_dataNode\_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_{{instance}}",

              "refId": "M"

            },

            {

              "expr": "[[app]]\_dataNode\_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Write\_{{instance}}",

              "refId": "N"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "CreateDataPartition",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 52

          },

          "id": 137,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpLoadDataPartition\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "LoadDataPartition",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 60

          },

          "id": 138,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateDataPartition\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DecommissionDataPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetPartitionSize\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_dataNode\_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "[[app]]\_dataNode\_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MarkDelete\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "[[app]]\_dataNode\_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "NotifyReplicasToRepair\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_dataNode\_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpRandomWrite\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_dataNode\_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_dataNode\_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_{{instance}}",

              "refId": "M"

            },

            {

              "expr": "[[app]]\_dataNode\_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Write\_{{instance}}",

              "refId": "N"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "NotifyReplicasToRepair",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 60

          },

          "id": 134,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpGetAllWatermarks\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "GetAllWatermarks",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 60

          },

          "id": 135,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpGetAppliedId\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_ops\_{{instance}}",

              "refId": "Q"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "GetAppliedId",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 68

          },

          "id": 142,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_{{instance}}",

              "refId": "M"

            },

            {

              "expr": "rate([[app]]dataNode\_OpStreamRead\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "StreamRead",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 68

          },

          "id": 143,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "Write\_{{instance}}",

              "refId": "N"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "Write",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 68

          },

          "id": 139,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateDataPartition\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DecommissionDataPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetPartitionSize\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_dataNode\_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "[[app]]\_dataNode\_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MarkDelete\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "[[app]]\_dataNode\_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "NotifyReplicasToRepair\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_dataNode\_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpRandomWrite\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_dataNode\_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_dataNode\_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_{{instance}}",

              "refId": "M"

            },

            {

              "expr": "[[app]]\_dataNode\_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Write\_{{instance}}",

              "refId": "N"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MarkDelete",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 76

          },

          "id": 144,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "CreateExtent",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 76

          },

          "id": 145,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpExtentRepairRead\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "ExtentRepairdRead",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 76

          },

          "id": 74,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_ops\_{{instance}}",

              "refId": "Q"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "BroadcastMinAppliedID",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 84

          },

          "id": 141,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpReadTinyDelete\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "ReadTinyDelete",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 84

          },

          "id": 136,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateDataPartition\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DecommissionDataPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "GetPartitionSize\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_dataNode\_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "[[app]]\_dataNode\_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MarkDelete\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "[[app]]\_dataNode\_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "NotifyReplicasToRepair\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_dataNode\_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpRandomWrite\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_dataNode\_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_dataNode\_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_{{instance}}",

              "refId": "M"

            },

            {

              "expr": "[[app]]\_dataNode\_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Write\_{{instance}}",

              "refId": "N"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "GetPartitionSize",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 16,

            "y": 84

          },

          "id": 146,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpTinyExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "TinyExtentRepairdRead\_{{instance}}",

              "refId": "P"

            },

            {

              "expr": "rate([[app]]\_dataNode\_OpTinyExtentRepairRead\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "TinyExtentRepairdRead\_ops\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "TinyExtentRepairdRead",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 0,

            "y": 92

          },

          "id": 131,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateDataPartition\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "DecommissionDataPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetPartitionSize\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_dataNode\_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "[[app]]\_dataNode\_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MarkDelete\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "[[app]]\_dataNode\_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "NotifyReplicasToRepair\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_dataNode\_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpRandomWrite\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_dataNode\_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_dataNode\_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_{{instance}}",

              "refId": "M"

            },

            {

              "expr": "[[app]]\_dataNode\_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Write\_{{instance}}",

              "refId": "N"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "DecommissionDataPartition",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 8,

            "w": 8,

            "x": 8,

            "y": 92

          },

          "id": 140,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_dataNode\_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpBroadcastMinAppliedID\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateDataPartition\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DataNodeHeartbeat\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_dataNode\_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DecommissionDataPartition\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAllWatermarks\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetAppliedId\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_dataNode\_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetPartitionSize\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_dataNode\_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "LoadDataPartition\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "[[app]]\_dataNode\_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MarkDelete\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "[[app]]\_dataNode\_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "NotifyReplicasToRepair\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_dataNode\_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpRandomWrite\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_dataNode\_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ReadTinyDelete\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_dataNode\_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamRead\_{{instance}}",

              "refId": "M"

            },

            {

              "expr": "[[app]]\_dataNode\_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Write\_{{instance}}",

              "refId": "N"

            },

            {

              "expr": "[[app]]\_dataNode\_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateExtent\_{{instance}}",

              "refId": "O"

            },

            {

              "expr": "[[app]]\_dataNode\_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "ExtentRepairdRead\_{{instance}}",

              "refId": "P"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "RandomWrite",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        }

      ],

      "title": "Datanode",

      "type": "row"

    },

    {

      "collapsed": true,

      "gridPos": {

        "h": 1,

        "w": 24,

        "x": 0,

        "y": 52

      },

      "id": 66,

      "panels": [

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 5

          },

          "id": 64,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaReadDir\_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_ops\_{{instance}}",

              "refId": "E"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaReadDir",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 8,

            "y": 5

          },

          "id": 157,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaBatchInodeGet\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_count\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_count\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_count\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_count\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_count\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_count\_{{instance}}",

              "refId": "L"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaCreateInode",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 5

          },

          "id": 156,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaBatchInodeGet\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_count\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_count\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaDeleteInode\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_count\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_count\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_count\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_count\_{{instance}}",

              "refId": "L"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "DeleteInode",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 12

          },

          "id": 160,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateDentry\_count{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_count\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaDeleteDentry\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_count\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaUpdateDentry\_count{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_count\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaDeleteDentry",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 8,

            "y": 12

          },

          "id": 68,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaCreateDentry\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_count\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteDentry\_count{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_count\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaUpdateDentry\_count{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_count\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaCreateDentry",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 12

          },

          "id": 102,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "rate([[app]]\_fuseclient\_StreamWrite\_count{app=\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamWrite\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_StreamRead\_count{app=\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "StreamRead",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_StreamSyncWrite\_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "SyncWrite",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_StreamSyncOverwrite\_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "SyncOverwrite",

              "refId": "D"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_StreamOverwrite\_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Overwrite",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaExtentsList\_count{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaExtentsList\_count\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaExtentsList{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaExtentsList\_{{instance}}",

              "refId": "G"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaExtentsList",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 19

          },

          "id": 69,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaExtentsAdd\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaExtentsAdd\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "OpMetaExtentsAdd\_count\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaExtentsList{instance=~\"$instance\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "GetExtents",

              "refId": "I"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaExtentsAdd",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 8,

            "y": 19

          },

          "id": 148,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaLookup{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaOpen\_count{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_ops\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_ops\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_ops\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaLookup\_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_ops\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_ops\_{{instance}}",

              "refId": "H"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaLookup",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 19

          },

          "id": 67,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaBatchInodeGet\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_count\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_count\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_count\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_count\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_count\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_count\_{{instance}}",

              "refId": "L"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaBatchInodeGet",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": "default",

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 26

          },

          "id": 152,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_filecreate{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "filecreate\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filecreate\_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "filecreate\_count\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_filesync{app=~\"$app\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filesync\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filesync\_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filesync\_count\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_filewrite{app=~\"$app\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filewrite\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filewrite\_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filewrite\_count\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "FileCreate",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 8,

            "y": 26

          },

          "id": 158,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaInodeGet\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_count\_{{instance}}",

              "refId": "K"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "InodeGet",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 26

          },

          "id": 155,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_{{instance}}",

              "refId": "J"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaBatchInodeGet\_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaBatchInodeGet\_count\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "CreateInode\_count\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "DeleteInode\_count\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaEvictInode\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "EvictInode\_count\_{{instance}}",

              "refId": "I"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "InodeGet\_count\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_count\_{{instance}}",

              "refId": "L"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "EvictInode",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": "default",

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 33

          },

          "id": 153,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_filecreate{app=~\"$app\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filecreate\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filecreate\_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filecreate\_count\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_filesync{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "filesync\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filesync\_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "filesync\_count\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_filewrite{app=~\"$app\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filewrite\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filewrite\_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filewrite\_count\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "FileSync",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": "default",

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 8,

            "y": 33

          },

          "id": 100,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_fileread{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "fileread\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_fileread\_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "fileread\_count\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "FileRead",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": "default",

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 33

          },

          "id": 154,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_filecreate{app=~\"$app\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filecreate\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filecreate\_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filecreate\_count\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_filesync{app=~\"$app\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filesync\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filesync\_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "filesync\_count\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_filewrite{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "filewrite\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_filewrite\_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "filewrite\_count\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "FileWrite",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 40

          },

          "id": 147,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaOpen\_count{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_ops\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_ops\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_ops\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_ops\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate\_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_ops\_{{instance}}",

              "refId": "H"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaTruncate",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 8,

            "y": 40

          },

          "id": 149,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaOpen\_count{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_ops\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_ops\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaSetattr\_cout{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_ops\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaLookup\_count{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_ops\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate\_count{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_ops\_{{instance}}",

              "refId": "H"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaSetattr",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 40

          },

          "id": 150,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_ops\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaOpen{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_{{instance}}",

              "refId": "K"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_{{instance}}",

              "refId": "L"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaOpen\_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "MetaOpen\_ops\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaReaddir\_ops\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaSetattr\_ops\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaLookup\_ops\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "MetaTruncate\_ops\_{{instance}}",

              "refId": "H"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaOpen",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 47

          },

          "id": 106,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "FlushInRead\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_StreamPrepareReadRequests{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "PrepareReadRequest\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "FlushInRead",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 8,

            "y": 47

          },

          "id": 161,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaCreateDentry\_count{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaCreateDentry\_count\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaDeleteDentry\_count{app=~\"$app\", cluster=\"$cluster\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "OpMetaDeleteDentry\_count\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_OpMetaUpdateDentry\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "OpMetaUpdateDentry\_count\_{{instance}}",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "MetaUpdateDentry",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 47

          },

          "id": 98,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_StreamWrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "Write",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_StreamRead{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "Read",

              "refId": "C"

            },

            {

              "expr": "[[app]]\_fuseclient\_StreamOverwrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "Overwrite",

              "refId": "D"

            },

            {

              "expr": "[[app]]\_fuseclient\_StreamCreateExtent{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "CreateExtent",

              "refId": "B"

            },

            {

              "expr": "[[app]]\_fuseclient\_StreamSyncWrite{instance=~\"$instance\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "SyncWrite",

              "refId": "E"

            },

            {

              "expr": "[[app]]\_fuseclient\_StreamSyncOverwrite{instance=~\"$instance\",cluster=\"$cluster\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "SyncOverwrite",

              "refId": "F"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "StreamSDKLatency",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 0,

            "y": 54

          },

          "id": 159,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "/.\*\_count\_.\*/",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "rate([[app]]\_fuseclient\_OpMetaTruncate\_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "Truncate\_count\_{{instance}}",

              "refId": "L"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "Truncate",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "ops",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 7,

            "w": 8,

            "x": 16,

            "y": 54

          },

          "id": 151,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "[[app]]\_fuseclient\_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "FlushInRead\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "[[app]]\_fuseclient\_PrepareReadRequest{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "PrepareReadRequest\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "PrepareReadRequest",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "ns",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        }

      ],

      "title": "FuseClient",

      "type": "row"

    },

    {

      "collapsed": true,

      "gridPos": {

        "h": 1,

        "w": 24,

        "x": 0,

        "y": 53

      },

      "id": 60,

      "panels": [

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 0,

            "y": 2

          },

          "id": 61,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "go\_goroutines\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "go\_threads\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "go\_goroutines",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 7,

            "y": 2

          },

          "id": 169,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "go\_goroutines\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "go\_threads\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "go\_threads",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 14,

            "y": 2

          },

          "id": 63,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "gc\_rate",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_gc\_duration\_seconds{instance=~\"$instance\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "seconds\_{{quantile}}",

              "refId": "A"

            },

            {

              "expr": "rate(go\_gc\_duration\_seconds\_count{instance=~\"$instance\"}[1m])",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "gc\_rate",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "gc\_duration",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "s",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 0,

            "y": 8

          },

          "id": 109,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "gc\_rate",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "process\_resident\_memory\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "resident\_memory\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "process\_virtual\_memory\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "virtual\_memory\_bytes\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "process\_virtual\_memory\_max\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "virtual\_memory\_max\_bytes\_{{instance}}",

              "refId": "C"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "process\_resident\_memory",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 7,

            "y": 8

          },

          "id": 166,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "gc\_rate",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "process\_resident\_memory\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "resident\_memory\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "process\_virtual\_memory\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "virtual\_memory\_bytes\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "process\_virtual\_memory\_max\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "virtual\_memory\_max\_bytes\_{{instance}}",

              "refId": "C"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "process\_virtual\_memory",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 14,

            "y": 8

          },

          "id": 167,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "gc\_rate",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "process\_resident\_memory\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "resident\_memory\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "process\_virtual\_memory\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "virtual\_memory\_bytes\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "process\_virtual\_memory\_max\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "virtual\_memory\_max\_bytes\_{{instance}}",

              "refId": "C"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "process\_virtual\_memory\_max",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": "default",

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 0,

            "y": 14

          },

          "id": 110,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "gc\_rate",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "process\_open\_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "open\_fds\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "process\_max\_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "max\_fds\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "open\_fds",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": "default",

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 7,

            "y": 14

          },

          "id": 168,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "gc\_rate",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "process\_open\_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "open\_fds\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "process\_max\_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "max\_fds\_{{instance}}",

              "refId": "B"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "max\_fds",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 14,

            "y": 14

          },

          "id": 111,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [

            {

              "alias": "gc\_rate",

              "yaxis": 2

            }

          ],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "rate(process\_cpu\_seconds\_total{cluster=\"$cluster\", role=\"$role\"}[1m])",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "cpu\_seconds\_{{instance}}",

              "refId": "A"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "process\_cpu",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "s",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "locale",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 0,

            "y": 20

          },

          "id": 62,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_memstats\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_memstats\_alloc\_bytes\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_total\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "go\_memstats\_heap\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_alloc\_bytes\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "go\_memstats\_heap\_inuse\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_inuse\_bytes\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "go\_memstats\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "sys\_bytes\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "go\_memstats\_mallocs\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mallocs\_total\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "go\_memstats\_frees\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "frees\_total\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "go\_memstats\_other\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "other\_sys\_bytes\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "go\_memstats\_mcache\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mcache\_sys\_bytes\_{{instance}}",

              "refId": "I"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "alloc\_bytes",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 7,

            "y": 20

          },

          "id": 172,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_memstats\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_memstats\_alloc\_bytes\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_total\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "go\_memstats\_heap\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "heap\_alloc\_bytes\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "go\_memstats\_heap\_inuse\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_inuse\_bytes\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "go\_memstats\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "sys\_bytes\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "go\_memstats\_mallocs\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mallocs\_total\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "go\_memstats\_frees\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "frees\_total\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "go\_memstats\_other\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "other\_sys\_bytes\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "go\_memstats\_mcache\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mcache\_sys\_bytes\_{{instance}}",

              "refId": "I"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "heap\_alloc",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 14,

            "y": 20

          },

          "id": 171,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_memstats\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_memstats\_alloc\_bytes\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_total\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "go\_memstats\_heap\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_alloc\_bytes\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "go\_memstats\_heap\_inuse\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "heap\_inuse\_bytes\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "go\_memstats\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "sys\_bytes\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "go\_memstats\_mallocs\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mallocs\_total\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "go\_memstats\_frees\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "frees\_total\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "go\_memstats\_other\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "other\_sys\_bytes\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "go\_memstats\_mcache\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mcache\_sys\_bytes\_{{instance}}",

              "refId": "I"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "heap\_inuse",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 0,

            "y": 26

          },

          "id": 170,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_memstats\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_memstats\_alloc\_bytes\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_total\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "go\_memstats\_heap\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_alloc\_bytes\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "go\_memstats\_heap\_inuse\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_inuse\_bytes\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "go\_memstats\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "sys\_bytes\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "go\_memstats\_mallocs\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mallocs\_total\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "go\_memstats\_frees\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "frees\_total\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "go\_memstats\_other\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "other\_sys\_bytes\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "go\_memstats\_mcache\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mcache\_sys\_bytes\_{{instance}}",

              "refId": "I"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "sys\_bytes",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 7,

            "y": 26

          },

          "id": 173,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_memstats\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_memstats\_alloc\_bytes\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_total\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "go\_memstats\_heap\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_alloc\_bytes\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "go\_memstats\_heap\_inuse\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_inuse\_bytes\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "go\_memstats\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "sys\_bytes\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "go\_memstats\_mallocs\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mallocs\_total\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "go\_memstats\_frees\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "frees\_total\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "go\_memstats\_other\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "other\_sys\_bytes\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "go\_memstats\_mcache\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mcache\_sys\_bytes\_{{instance}}",

              "refId": "I"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "other\_sys",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        },

        {

          "aliasColors": {},

          "bars": false,

          "dashLength": 10,

          "dashes": false,

          "datasource": null,

          "fill": 1,

          "gridPos": {

            "h": 6,

            "w": 7,

            "x": 14,

            "y": 26

          },

          "id": 174,

          "legend": {

            "avg": false,

            "current": false,

            "max": false,

            "min": false,

            "show": true,

            "total": false,

            "values": false

          },

          "lines": true,

          "linewidth": 1,

          "links": [],

          "nullPointMode": "null",

          "percentage": false,

          "pointradius": 5,

          "points": false,

          "renderer": "flot",

          "seriesOverrides": [],

          "spaceLength": 10,

          "stack": false,

          "steppedLine": false,

          "targets": [

            {

              "expr": "go\_memstats\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_{{instance}}",

              "refId": "A"

            },

            {

              "expr": "go\_memstats\_alloc\_bytes\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "alloc\_bytes\_total\_{{instance}}",

              "refId": "B"

            },

            {

              "expr": "go\_memstats\_heap\_alloc\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_alloc\_bytes\_{{instance}}",

              "refId": "C"

            },

            {

              "expr": "go\_memstats\_heap\_inuse\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "heap\_inuse\_bytes\_{{instance}}",

              "refId": "D"

            },

            {

              "expr": "go\_memstats\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "sys\_bytes\_{{instance}}",

              "refId": "E"

            },

            {

              "expr": "go\_memstats\_mallocs\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "mallocs\_total\_{{instance}}",

              "refId": "F"

            },

            {

              "expr": "go\_memstats\_frees\_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "frees\_total\_{{instance}}",

              "refId": "G"

            },

            {

              "expr": "go\_memstats\_other\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": true,

              "intervalFactor": 1,

              "legendFormat": "other\_sys\_bytes\_{{instance}}",

              "refId": "H"

            },

            {

              "expr": "go\_memstats\_mcache\_sys\_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",

              "format": "time\_series",

              "hide": false,

              "intervalFactor": 1,

              "legendFormat": "mcache\_sys\_bytes\_{{instance}}",

              "refId": "I"

            }

          ],

          "thresholds": [],

          "timeFrom": null,

          "timeShift": null,

          "title": "mcache\_sys",

          "tooltip": {

            "shared": true,

            "sort": 0,

            "value\_type": "individual"

          },

          "type": "graph",

          "xaxis": {

            "buckets": null,

            "mode": "time",

            "name": null,

            "show": true,

            "values": []

          },

          "yaxes": [

            {

              "format": "decbytes",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            },

            {

              "format": "short",

              "label": null,

              "logBase": 1,

              "max": null,

              "min": null,

              "show": true

            }

          ],

          "yaxis": {

            "align": false,

            "alignLevel": null

          }

        }

      ],

      "title": "GoRuntime",

      "type": "row"

    }

  ],

  "refresh": false,

  "schemaVersion": 16,

  "style": "dark",

  "tags": [],

  "templating": {

    "list": [

      {

        "allValue": null,

        "current": {

          "selected": true,

          "text": "cfs",

          "value": "cfs"

        },

        "hide": 2,

        "includeAll": false,

        "label": "App",

        "multi": false,

        "name": "app",

        "options": [

          {

            "selected": true,

            "text": "cfs",

            "value": "cfs"

          }

        ],

        "query": "cfs",

        "type": "custom"

      },

      {

        "allValue": null,

        "current": {

          "tags": [],

          "text": "spark",

          "value": "spark"

        },

        "datasource": "default",

        "hide": 0,

        "includeAll": false,

        "label": "Cluster",

        "multi": false,

        "name": "cluster",

        "options": [],

        "query": "label\_values(go\_info{app=~\"$app\"}, cluster)",

        "refresh": 1,

        "regex": "",

        "sort": 0,

        "tagValuesQuery": "",

        "tags": [],

        "tagsQuery": "",

        "type": "query",

        "useTags": false

      },

      {

        "allValue": null,

        "current": {

          "text": "fuseclient",

          "value": "fuseclient"

        },

        "datasource": "default",

        "hide": 0,

        "includeAll": false,

        "label": "Role",

        "multi": false,

        "name": "role",

        "options": [],

        "query": "label\_values(go\_info{app=~\"$app\", cluster=~\"$cluster\"}, role)",

        "refresh": 1,

        "regex": "",

        "sort": 0,

        "tagValuesQuery": "",

        "tags": [],

        "tagsQuery": "",

        "type": "query",

        "useTags": false

      },

      {

        "allValue": null,

        "current": {

          "text": "10.170.6.247:9500",

          "value": "10.170.6.247:9500"

        },

        "datasource": "default",

        "hide": 0,

        "includeAll": false,

        "label": "Instance",

        "multi": false,

        "name": "instance",

        "options": [],

        "query": "label\_values(go\_info{app=~\"$app\", role=~\"$role\", cluster=~\"$cluster\"}, instance)",

        "refresh": 1,

        "regex": "",

        "sort": 0,

        "tagValuesQuery": "",

        "tags": [],

        "tagsQuery": "",

        "type": "query",

        "useTags": false

      },

      {

        "allValue": null,

        "current": {

          "text": "10.170.6.247",

          "value": "10.170.6.247"

        },

        "datasource": "default",

        "hide": 2,

        "includeAll": false,

        "label": "Host",

        "multi": false,

        "name": "hostip",

        "options": [],

        "query": "label\_values(go\_info{instance=~\"$instance\", cluster=~\"$cluster\"}, instance)",

        "refresh": 1,

        "regex": "/([^:]+):.\*/",

        "sort": 0,

        "tagValuesQuery": "",

        "tags": [],

        "tagsQuery": "",

        "type": "query",

        "useTags": false

      },

      {

        "allValue": null,

        "current": {

          "tags": [],

          "text": "clickhourse",

          "value": "clickhourse"

        },

        "datasource": "default",

        "hide": 0,

        "includeAll": false,

        "label": "Vol",

        "multi": false,

        "name": "vol",

        "options": [],

        "query": "label\_values(cfs\_master\_vol\_total\_GB{app=\"$app\",cluster=\"$cluster\"}, volName)",

        "refresh": 1,

        "regex": "",

        "sort": 0,

        "tagValuesQuery": "",

        "tags": [],

        "tagsQuery": "",

        "type": "query",

        "useTags": false

      }

    ]

  },

  "time": {

    "from": "now-1h",

    "to": "now"

  },

  "timepicker": {

    "refresh\_intervals": [

      "5s",

      "10s",

      "30s",

      "1m",

      "5m",

      "15m",

      "30m",

      "1h",

      "2h",

      "1d"

    ],

    "time\_options": [

      "5m",

      "15m",

      "1h",

      "6h",

      "12h",

      "24h",

      "2d",

      "7d",

      "30d"

    ]

  },

  "timezone": "",

  "title": "chubaofs",

  "uid": "J8XJyOmZk",

  "version": 484

}

```

Tune FUSE Performance[¶](#tune-fuse-performance "Permalink to this headline")

-----------------------------------------------------------------------------

### Fetch Linux kernel source code[¶](#fetch-linux-kernel-source-code "Permalink to this headline")

Download the corresponding src rpm, and use the following commands to install source code.

```

rpm -i kernel-3.10.0-327.28.3.el7.src.rpm 2>&1 | grep -v exist

cd ~/rpmbuild/SPECS

rpmbuild -bp --target=$(uname -m) kernel.spec

```

The source code will be installed in `~/rpmbuild/BUILD/`

### Optimize FUSE linux kernel module[¶](#optimize-fuse-linux-kernel-module "Permalink to this headline")

In order to achieve maximum throughput performance, several FUSE kernel parameters have to be modified, such as `FUSE\_MAX\_PAGES\_PER\_REQ` and `FUSE\_DEFAULT\_MAX\_BACKGROUND`.

Update source code according to the following lines.

```

/\* fs/fuse/fuse\_i.h \*/

#define FUSE\_MAX\_PAGES\_PER\_REQ 256

/\* fs/fuse/inode.c \*/

#define FUSE\_DEFAULT\_MAX\_BACKGROUND 32

```

### Build against current running Linux kernel[¶](#build-against-current-running-linux-kernel "Permalink to this headline")

```

yum install kernel-devel-3.10.0-327.28.3.el7.x86_64

cd ~/rpmbuild/BUILD/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/fuse

make -C /lib/modules/`uname -r`/build M=$PWD

```

### Install kernel module[¶](#install-kernel-module "Permalink to this headline")

```

cp fuse.ko /lib/modules/`uname -r`/kernel/fs/fuse

rmmod fuse

depmod -a

modprobe fuse

```

Run Cluster on Docker[¶](#run-cluster-on-docker "Permalink to this headline")

-----------------------------------------------------------------------------

Under the docker directory, a helper tool called run\_docker.sh is provided to run ChubaoFS with docker-compose.

To start a minimal ChubaoFS cluster from scratch, note that **/data/disk** is arbitrary, and make sure there are at least 30G available space.

```

$ docker/run_docker.sh -r -d /data/disk

```

If client starts successfully, use mount command in client docker shell to check mount status:

```

$ mount | grep chubaofs

```

Open <http://127.0.0.1:3000> in browser, login with admin/123456 to view grafana monitor metrics.

Or run server and client seperately by following commands:

```

$ docker/run_docker.sh -b

$ docker/run_docker.sh -s -d /data/disk

$ docker/run_docker.sh -c

$ docker/run_docker.sh -m

```

For more usage:

```

$ docker/run_docker.sh -h

```

Prometheus and Grafana confg can be found in docker/monitor directory.

Resource Manager (Master) API[¶](#resource-manager-master-api "Permalink to this headline")

-------------------------------------------------------------------------------------------

### Cluster[¶](#cluster "Permalink to this headline")

#### Overview[¶](#overview "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/admin/getCluster" | python -m json.tool

```

Display the base information of the cluster, such as the detail of metaNode, dataNode, vol and so on.

response

```

{

    "Name": "test",

    "LeaderAddr": "10.196.59.198:17010",

    "DisableAutoAlloc": false,

    "Applied": 225,

    "MaxDataPartitionID": 100,

    "MaxMetaNodeID": 3,

    "MaxMetaPartitionID": 1,

    "DataNodeStatInfo": {},

    "MetaNodeStatInfo": {},

    "VolStatInfo": {},

    "BadPartitionIDs": {},

    "BadMetaPartitionIDs": {},

    "MetaNodes": {},

    "DataNodes": {}

}

```

#### Freeze[¶](#freeze "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"

```

If cluster is freezed, the vol never allocates dataPartitions automatically.

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| enable | bool | if enable is true, the cluster is freezed |

#### Statistics[¶](#statistics "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/cluster/stat"

```

Show cluster space information by zone.

response

```

{

    "DataNodeStatInfo": {

        "TotalGB": 1,

        "UsedGB": 0,

        "IncreasedGB": -2,

        "UsedRatio": "0.0"

    },

    "MetaNodeStatInfo": {

        "TotalGB": 1,

        "UsedGB": 0,

        "IncreasedGB": -8,

        "UsedRatio": "0.0"

    },

    "ZoneStatInfo": {

        "zone1": {

            "DataNodeStat": {

                "TotalGB": 1,

                "UsedGB": 0,

                "AvailGB": 0,

                "UsedRatio": 0,

                "TotalNodes": 0,

                "WritableNodes": 0

            },

            "MetaNodeStat": {

                "TotalGB": 1,

                "UsedGB": 0,

                "AvailGB": 0,

                "UsedRatio": 0,

                "TotalNodes": 0,

                "WritableNodes": 0

            }

        }

    }

}

```

#### Topology[¶](#topology "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/topo/get"

```

Show cluster topology information by zone.

response

```

[

    {

        "Name": "zone1",

        "Status": "available",

        "NodeSet": {

            "700": {

                "DataNodeLen": 0,

                "MetaNodeLen": 0,

                "MetaNodes": [],

                "DataNodes": []

            }

        }

    },

    {

        "Name": "zone2",

        "Status": "available",

        "NodeSet": {

            "800": {

                "DataNodeLen": 0,

                "MetaNodeLen": 0,

                "MetaNodes": [],

                "DataNodes": []

            }

        }

    }

]

```

#### Update Zone[¶](#update-zone "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/zone/update?name=zone1&enable=false"

```

Set the status of the zone to available or unavailable.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | zone name |

| enable | bool | if enable is true, the cluster is available |

#### Get Zone[¶](#get-zone "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/zone/list"

```

Get name and status of all zones.

response

```

[

    {

        "Name": "zone1",

        "Status": "available",

        "NodeSet": {}

    },

    {

        "Name": "zone2",

        "Status": "available",

        "NodeSet": {}

    }

]

```

#### Get Node Info[¶](#get-node-info "Permalink to this headline")

```

curl -v "http://192.168.0.11:17010/admin/getNodeInfo"

```

Get node info of cluster.

response

```

{

    "code": 0,

    "msg": "success",

    "data": {

        "batchCount": 0,

        "deleteWorkerSleepMs": 0,

        "markDeleteRate": 0

    }

}

```

#### Set Node Info[¶](#set-node-info "Permalink to this headline")

```

curl -v "http://192.168.0.11:17010/admin/setNodeInfo?batchCount=100&markDeleteRate=100&deleteWorkerSleepMs=1000"

```

Set node info of cluster.

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| batchCount | uint64 | metanode delete batch count |

| deleteWorkerSleepMs | uint64 | metanode delete worker sleep time with millisecond. if 0 for no sleep |

| markDeleteRate | uint64 | datanode batch markdelete limit rate. if 0 for no infinity limit |

### Metanode Related[¶](#metanode-related "Permalink to this headline")

#### GET[¶](#get "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/metaNode/get?addr=10.196.59.202:17210"  | python -m json.tool

```

Show the base information of the metaNode, such as addr, total memory, used memory and so on.

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| addr | string | the addr which communicate with master |

response

```

{

    "ID": 3,

    "Addr": "10.196.59.202:17210",

    "IsActive": true,

    "Zone": "zone1",

    "MaxMemAvailWeight": 66556215048,

    "TotalWeight": 67132641280,

    "UsedWeight": 576426232,

    "Ratio": 0.008586377967698518,

    "SelectCount": 0,

    "Carry": 0.6645600532184904,

    "Threshold": 0.75,

    "ReportTime": "2018-12-05T17:26:28.29309577+08:00",

    "MetaPartitionCount": 1,

    "NodeSetID": 2,

    "PersistenceMetaPartitions": {}

}

```

#### Decommission[¶](#decommission "Permalink to this headline")

```

curl -v "http://127.0.0.1/metaNode/decommission?addr=127.0.0.1:9021"

```

Remove the metaNode from cluster, meta partitions which locate the metaNode will be migrate other available metaNode asynchronous.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| addr | string | the addr which communicate with master |

#### Threshold[¶](#threshold "Permalink to this headline")

```

curl -v "http://127.0.0.1/threshold/set?threshold=0.75"

```

If the used memory percent arrives the threshold, the status of the meta partitions which locate the metaNode will be read only.

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| threshold | float64 | the max percent of memory which metaNode can use |

### Datanode Related[¶](#datanode-related "Permalink to this headline")

#### GET[¶](#get "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/dataNode/get?addr=10.196.59.201:17310"  | python -m json.tool

```

Show the base information of the dataNode, such as addr, disk total size, disk used size and so on.

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| addr | string | the addr which communicate with master |

response

```

{

    "TotalWeight": 39666212700160,

    "UsedWeight": 2438143586304,

    "AvailableSpace": 37228069113856,

    "ID": 2,

    "Zone": "zone1",

    "Addr": "10.196.59.201:17310",

    "ReportTime": "2018-12-06T10:56:38.881784447+08:00",

    "IsActive": true

    "UsageRatio": 0.06146650815226848,

    "SelectTimes": 5,

    "Carry": 1.0655859145960367,

    "DataPartitionReports": {},

    "DataPartitionCount": 21,

    "NodeSetID": 3,

    "PersistenceDataPartitions": {},

    "BadDisks": {}

}

```

#### Decommission[¶](#decommission "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/dataNode/decommission?addr=10.196.59.201:17310"

```

Remove the dataNode from cluster, data partitions which locate the dataNode will be migrate other available dataNode asynchronous.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| addr | string | the addr which communicate with master |

### Volume[¶](#volume "Permalink to this headline")

#### Create[¶](#create "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=100&owner=cfs&mpCount=3"

```

Allocate a set of data partition and a meta partition to the user.

Default create 10 data partition and 3 meta partition when create volume.

ChubaoFS uses the **Owner** parameter as the user ID. When creating a volume, if there is no user named the owner of the volume, a user with the user ID same as **Owner** will be automatically created; if a user named Owner already exists in the cluster, the volume will be owned by the user. For details, please see:: doc: /admin-api/master/user

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description | Mandatory | Default |

| --- | --- | --- | --- | --- |

| name | string | volume name | Yes | None |

| capacity | int | the quota of vol, unit is GB | Yes | None |

| owner | string | the owner of vol, and user ID of a user | Yes | None |

| mpCount | int | the amount of initial meta partitions | No | 3 |

| enableToken | bool | whether to enable the token mechanism to control client permissions | No | false |

| size | int | the size of data partitions, unit is GB | No | 120 |

| followerRead | bool | enable read from follower | No | false |

| crossZone | bool | cross zone or not. If it is true, parameter *zoneName* must be empty | No | false |

| zoneName | string | specified zone | No | default (if *crossZone* is false) |

#### Delete[¶](#delete "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/vol/delete?name=test&authKey=md5(owner)"

```

Mark the vol status to MarkDelete first, then delete data partition and meta partition asynchronous, finally delete meta data from persist store.

While deleting the volume, the policy information related to the volume will be deleted from all user information.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | volume name |

| authKey | string | calculates the 32-bit MD5 value of the owner field as authentication information |

#### Get[¶](#get "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/client/vol?name=test&authKey=md5(owner)" | python -m json.tool

```

Show the base information of the vol, such as name, the detail of data partitions and meta partitions and so on.

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | volume name |

| authKey | string | calculates the 32-bit MD5 value of the owner field as authentication information |

response

```

{

    "Name": "test",

    "Owner": "user",

    "Status": "0",

    "FollowerRead": "true",

    "MetaPartitions": {},

    "DataPartitions": {},

    "DataPartitions": {},

    "CreateTime": 0

}

```

#### Stat[¶](#stat "Permalink to this headline")

```

curl -v http://10.196.59.198:17010/client/volStat?name=test

```

Show the status information of volume.

Parameters[¶](#id4 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | volume name |

response

```

{

    "Name": "test",

    "TotalSize": 322122547200000000,

    "UsedSize": 155515112832780000,

    "UsedRatio": "0.48",

    "EnableToken": false

}

```

#### Update[¶](#update "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/vol/update?name=test&capacity=100&authKey=md5(owner)"

```

Increase the quota of volume, or adjust other parameters.

Parameters[¶](#id5 "Permalink to this table")

| Parameter | Type | Description | Mandatory |

| --- | --- | --- | --- |

| name | string | volume name | Yes |

| authKey | string | calculates the 32-bit MD5 value of the owner field as authentication information | Yes |

| capacity | int | the quota of vol, unit is GB | Yes |

| zoneName | string | update zone name | Yes |

| enableToken | bool | whether to enable the token mechanism to control client permissions. `False` by default. | No |

| followerRead | bool | enable read from follower | No |

#### List[¶](#list "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/vol/list?keywords=test"

```

List all volumes information, and can be filtered by keywords.

Parameters[¶](#id6 "Permalink to this table")

| Parameter | Type | Description | Mandatory |

| --- | --- | --- | --- |

| keywords | string | get volumes information which contains this keyword | No |

response

```

[

   {

       "Name": "test1",

       "Owner": "cfs",

       "CreateTime": 0,

       "Status": 0,

       "TotalSize": 155515112832780000,

       "UsedSize": 155515112832780000

   },

   {

       "Name": "test2",

       "Owner": "cfs",

       "CreateTime": 0,

       "Status": 0,

       "TotalSize": 155515112832780000,

       "UsedSize": 155515112832780000

   }

]

```

#### Add Token[¶](#add-token "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/token/add?name=test&tokenType=1&authKey=md5(owner)"

```

Add the token that controls read and write permissions.

Parameters[¶](#id7 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | the name of vol |

| tokenType | int | 1 is readonly token, 2 is readWrite token |

| authKey | string | calculates the 32-bit MD5 value of the owner field as authentication information |

#### Update Token[¶](#update-token "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/token/update?name=test&token=xx&tokenType=1&authKey=md5(owner)"

```

Update token type.

Parameters[¶](#id8 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | the name of vol |

| token | string | the token value |

| tokenType | int | 1 is readonly token, 2 is readWrite token |

| authKey | string | calculates the 32-bit MD5 value of the owner field as authentication information |

#### Delete Token[¶](#delete-token "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/token/delete?name=test&token=xx&authKey=md5(owner)"

```

Delete specified token.

Parameters[¶](#id9 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | the name of vol |

| token | string | the token value |

| authKey | string | calculates the 32-bit MD5 value of the owner field as authentication information |

#### Get Token[¶](#get-token "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/token/get?name=test&token=xx"

```

Show token information.

Parameters[¶](#id10 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | the name of vol |

| token | string | the token value |

response

```

{

    "TokenType":2,

    "Value":"siBtuF9hbnNqXzJfMTU48si3nzU4MzE1Njk5MDM1NQ==",

    "VolName":"test"

}

```

### Meta Partition[¶](#meta-partition "Permalink to this headline")

#### Create[¶](#create "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/metaPartition/create?name=test&start=10000"

```

Split meta partition manually. If max meta partition of the vol which range is `[0,end)` and `end` larger than `start` parameter, old meta partition range will be `[0,start]`, new meta partition will be `[start+1,end)`.

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| name | string | the name of vol |

| start | uint64 | the start value of meta partition which will be create |

#### Get[¶](#get "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/metaPartition/get?id=1" | python -m json.tool

```

Show base information of meta partition, such as id, start, end and so on.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| id | uint64 | the id of meta partition |

response

```

{

    "PartitionID": 1,

    "Start": 0,

    "End": 9223372036854776000,

    "MaxNodeID": 1,

    "VolName": "test",

    "Replicas": {},

    "ReplicaNum": 3,

    "Status": 2,

    "IsRecover": true,

    "Hosts": {},

    "Peers": {},

    "Zones": {},

    "MissNodes": {},

    "LoadResponse": {}

}

```

#### Decommission[¶](#decommission "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/metaPartition/decommission?id=13&addr=10.196.59.202:17210"

```

Remove the replica of meta partition, and create new replica asynchronous.

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| id | uint64 | the id of meta partition |

| addr | string | the addr of replica which will be decommission |

#### Load[¶](#load "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/metaPartition/load?id=1"

```

Send load task to the metaNode which meta partition locate on, then check the crc of each replica in the meta partition.

Parameters[¶](#id4 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| id | uint64 | the id of data partition |

### Data Partition[¶](#data-partition "Permalink to this headline")

#### Create[¶](#create "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/dataPartition/create?count=400&name=test"

```

Create a set of data partition.

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| count | int | the num of dataPartitions will be create |

| name | string | the name of vol |

#### Get[¶](#get "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/dataPartition/get?id=100"  | python -m json.tool

```

Get information of the specified data partition.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| id | uint64 | the id of data partition |

response

```

{

    "PartitionID": 100,

    "LastLoadedTime": 1544082851,

    "ReplicaNum": 3,

    "Status": 2,

    "Replicas": {},

    "Hosts": {},

    "Peers": {},

    "Zones": {},

    "MissingNodes": {},

    "VolName": "test",

    "VolID": 2,

    "FileInCoreMap": {},

    "FilesWithMissingReplica": {}

}

```

#### Decommission[¶](#decommission "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/dataPartition/decommission?id=13&addr=10.196.59.201:17310"

```

Remove the replica of data partition, and create new replica asynchronous.

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| id | uint64 | the id of data partition |

| addr | string | the addr of replica which will be decommission |

#### Load[¶](#load "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/dataPartition/load?id=1"

```

Send load task to the dataNode which data partition locate on, then check the crc of each file in the data partition asynchronous.

Parameters[¶](#id4 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| id | uint64 | the id of data partition |

#### Offline Disk[¶](#offline-disk "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/disk/decommission?addr=10.196.59.201:17310&disk=/cfs1"

```

Synchronously offline all the data partitions on the disk, and create a new replica for each data partition in the cluster.

Parameters[¶](#id5 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| addr | string | replica address |

| disk | string | disk path |

### Master Management[¶](#master-management "Permalink to this headline")

#### Add[¶](#add "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/raftNode/add?addr=10.196.59.197:17010&id=3"

```

Add a new master node to master raft group.

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| addr | string | the addr of master server, format is ip:port |

| id | uint64 | the node id of master server |

#### Remove[¶](#remove "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/raftNode/remove?addr=10.196.59.197:17010&id=3"

```

Remove the master node from master raft group.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| addr | string | the addr of master server, format is ip:port |

| id | uint64 | the node id of master server |

### User[¶](#user "Permalink to this headline")

#### Create[¶](#create "Permalink to this headline")

```

curl -H "Content-Type:application/json" -X POST --data '{"id":"testuser","pwd":"12345","type":3}' "http://10.196.59.198:17010/user/create"

```

Create a user in the cluster to access object storage service. When the cluster starts, the `root` user is automatically created (the value of `type` is `0x1`).

ChubaoFS treats every `Owner` of volume as a `user`. For example, if the value of **Owner** is `testuser` when creating a volume, the volume is owned by user `testuser`.

If there is no user ID with the same value as the **Owner** when creating the volume, the user named the value of **Owner** will be automatically created when creating the volume.

body key[¶](#id1 "Permalink to this table")

| Key | Type | Description | Range | Mandatory | Default |

| --- | --- | --- | --- | --- | --- |

| id | string | user ID | Consists of letters, numbers and underscores, no more than 20 characters | Yes | None |

| pwd | string | user’s password | Unlimited | No | `ChubaoFSUser` |

| ak | string | Access Key | Consists of 16-bits letters and numbers | No | Random value |

| sk | string | Secret Key | Consists of 32-bits letters and numbers | No | Random value |

| type | int | user type | 2: [admin] / 3: [normal user] | Yes | None |

#### Delete[¶](#delete "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/user/delete?user=testuser"

```

Delete the specified user in the cluster.

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| user | string | user ID |

#### Get[¶](#get "Permalink to this headline")

Show basic user information, including user ID, Access Key, Secret Key, list of owned volumes, list of permissions granted by other users, user type, creation time.

The field `policy` shows the volumes which the user has permission, of which `own\_vols` indicates that volumes owned by the user, and `authorized\_vols` indicates the volume authorized by other users to the user with restrictions.

There are two ways to obtain:

##### Query by User ID[¶](#query-by-user-id "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/user/info?user=testuser" | python -m json.tool

```

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| user | string | user ID |

##### Query by Access Key[¶](#query-by-access-key "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/user/akInfo?ak=0123456789123456" | python -m json.tool

```

Parameters[¶](#id4 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| ak | string | Access Key |

response

```

{

     "user\_id": "testuser",

     "access\_key": "0123456789123456",

     "secret\_key": "ZVY5RHlrnOrCjImW9S3MajtYZyxSegcf",

     "policy": {

         "own\_vols": ["vol1"],

         "authorized\_vols": {

             "ltptest": [

                 "perm:builtin:ReadOnly",

                 "perm:custom:PutObjectAction"

             ]

         }

     },

     "user\_type": 3,

     "create\_time": "2020-05-11 09:25:04"

}

```

#### List Users[¶](#list-users "Permalink to this headline")

```

curl -v "http://10.196.59.198:17010/user/list?keywords=test" | python -m json.tool

```

Query information about all users in a cluster whose user ID contains the keyword.

Parameters[¶](#id5 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| keywords | string | check user ID contains this or not |

#### Update[¶](#update "Permalink to this headline")

```

curl -H "Content-Type:application/json" -X POST --data '{"user\_id":"testuser","access\_key":"KzuIVYCFqvu0b3Rd","secret\_key":"iaawlCchJeeuGSnmFW72J2oDqLlSqvA5","type":3}' "http://10.196.59.198:17010/user/update"

```

Update the specified user’s information, including access key, secret key and user type.

body key[¶](#id6 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| user\_id | string | user ID value after updating | Yes |

| access\_key | string | Access Key value after updating | No |

| secret\_key | string | Secret Key value after updating | No |

| type | int | user type value after updating | No |

#### Update Permission[¶](#update-permission "Permalink to this headline")

```

curl -H "Content-Type:application/json" -X POST --data '{"user\_id":"testuser","volume":"vol","policy":["perm:builtin:ReadOnly","perm:custom:PutObjectAction"]}' "http://10.196.59.198:17010/user/updatePolicy"

```

Update the specified user’s permission to a volume. There are three types of values for `policy`:

* Grant read-only or read-write permission, the value is `perm:builtin:ReadOnly` or `perm:builtin:Writable`.

* Grant a permission of the specified action, the format is `action:oss:XXX`, take *GetObject* action as an example, the value of policy is `action:oss:GetObject`.

* Grant a custom permission, the format is `perm:custom:XXX`, where *XXX* is customized by the user.

After the permissions are specified, the user can only access the volume within the specified permissions when using the object storage. If the user already has permissions for this volume, this operation will overwrite the original permissions.

body key[¶](#id7 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| user\_id | string | user ID to be set | Yes |

| volume | string | volume name to be set | Yes |

| policy | string slice | policy to be set | Yes |

#### Remove Permission[¶](#remove-permission "Permalink to this headline")

```

curl -H "Content-Type:application/json" -X POST --data '{"user\_id":"testuser","volume":"vol"}' "http://10.196.59.198:17010/user/removePolicy"

```

Remove all permissions of a specified user for a volume.

body key[¶](#id8 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| user\_id | string | user ID to be deleted | Yes |

| volume | string | volume name to be deleted | Yes |

#### Transfer Volume[¶](#transfer-volume "Permalink to this headline")

```

curl -H "Content-Type:application/json" -X POST --data '{"volume":"vol","user\_src":"user1","user\_dst":"user2","force":"true"}' "http://10.196.59.198:17010/user/transferVol"

```

Transfer the ownership of the specified volume. This operation removes the specified volume from the `owner\_vols` of source user name and adds it to the `owner\_vols` of target user name; At the same time, the value of the field `Owner` in the volume structure will also be updated to the target user ID.

body key[¶](#id9 "Permalink to this table")

| Key | Type | Description | Mandatory |

| --- | --- | --- | --- |

| volume | string | Volume name to be transfered | Yes |

| user\_src | string | Original owner of the volume, and must be the same as the `Owner` of the volume | Yes |

| user\_dst | string | Target user ID after transferring | Yes |

| force | bool | Force to transfer the volume. If the value is set to true, even if the value of `user\_src` is different from the value of the owner of the volume, the volume will also be transferred to the target user | No |

Meta Node API[¶](#meta-node-api "Permalink to this headline")

-------------------------------------------------------------

### Meta Partition[¶](#meta-partition "Permalink to this headline")

#### Get Partitions[¶](#get-partitions "Permalink to this headline")

```

curl -v  http://10.196.59.202:17210/getPartitions

```

Get all meta-partition base information of the metanode.

#### Get Partition by ID[¶](#get-partition-by-id "Permalink to this headline")

```

curl -v http://10.196.59.202:17210/getPartitionById?pid=100

```

Get the specified partition information, this result contains: leader address, raft group peer and cursor.

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| pid | integer | meta-partition id |

### Inode[¶](#inode "Permalink to this headline")

#### Get Inode[¶](#get-inode "Permalink to this headline")

```

curl -v http://10.196.59.202:17210/getInode?pid=100&ino=1024

```

Get inode information

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| pid | integer | meta-partition id |

| ino | integer | inode id |

#### Get Extents by Inode[¶](#get-extents-by-inode "Permalink to this headline")

```

curl -v http://10.196.59.202:17210/getExtentsByInode?pid=100&ino=1024

```

Get inode all extents information

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| pid | integer | meta-partition id |

| ino | integer | inode id |

#### Get All Inodes[¶](#get-all-inodes "Permalink to this headline")

```

curl -v http://10.196.59.202:17210/getAllInodes?pid=100

```

Get all inodes of the specified partition

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| pid | integer | meta-partition id |

### Dentry[¶](#dentry "Permalink to this headline")

#### Get Dentry[¶](#get-dentry "Permalink to this headline")

```

curl -v 'http://10.196.59.202:17210/getDentry?pid=100&name=""&parentIno=1024'

```

Get dentry information

Parameters[¶](#id1 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| pid | integer | meta partition id |

| name | string | file or directory name |

| parentIno | integer | file or directory parent directory inode |

#### Get Directory[¶](#get-directory "Permalink to this headline")

```

curl -v "http://10.196.59.202:17210/getDirectory?pid=100&parentIno=1024"

```

Get all files of the parent inode is 1024

Parameters[¶](#id2 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| pid | integer | partition id |

| ino | integer | inode id |

#### Get All Dentry[¶](#get-all-dentry "Permalink to this headline")

```

curl -v "http://10.196.59.202:17210/getAllDentry?pid=100"

```

Parameters[¶](#id3 "Permalink to this table")

| Parameter | Type | Description |

| --- | --- | --- |

| pid | integer | partition id |

Command Line Interface[¶](#command-line-interface "Permalink to this headline")

-------------------------------------------------------------------------------

### CLI[¶](#cli "Permalink to this headline")

Using the Command Line Interface (CLI) can manage the cluster conveniently. With this tool, you can view the status of the cluster and each node, and manage each node, volumes, or users.

As the CLI continues to improve, it will eventually achieve 100% coverage of the APIs of each node in the cluster.

#### Compile and Configure[¶](#compile-and-configure "Permalink to this headline")

After downloading the ChubaoFS source code, execute the command `go build` in the directory `chubaofs/cli` to generate `cli`.

At the same time, a configuration file named `.cfs-cli.json` will be generated in the directory `root`, and the master address can be changed to the current cluster master address. You can also get or set the master address by executing the command `./cli config info` or `./cli config set`.

#### Bug Shooting[¶](#bug-shooting "Permalink to this headline")

The logs of `cfs-cli` tool are in the directory `/tmp/cfs/cli`, which offer detail running information for bug shooting.

#### Usage[¶](#usage "Permalink to this headline")

In the directory `chubaofs/cli`, execute the command `./cli --help` or `./cli -h` to get the CLI help document.

CLI is mainly divided into seven types of management commands.

Commands List[¶](#id1 "Permalink to this table")

| Command | description |

| --- | --- |

| cli cluster | Manage cluster components |

| cli metanode | Manage meta nodes |

| cli datanode | Manage data nodes |

| cli datapartition | Manage data partitions |

| cli metapartition | Manage meta partitions |

| cli config | Manage configuration for cli tool |

| cli completion | Generating bash completions |

| cli volume, vol | Manage cluster volumes |

| cli user | Manage cluster users |

| cli compatibility | Compatibility test |

##### Cluster Management[¶](#cluster-management "Permalink to this headline")

```

./cli cluster info          #Show cluster summary information

```

```

./cli cluster stat          #Show cluster status information

```

```

./cli cluster freeze [true/false]        #Turn on or turn off the automatic allocation of the data partitions.

```

```

./cli cluster threshold [float]     #Set the threshold of memory on each meta node.

```

##### MetaNode Management[¶](#metanode-management "Permalink to this headline")

```

./cli metanode list          #List information of meta nodes

```

```

./cli metanode info [Address]     #Show detail information of a meta node

```

```

./cli metanode decommission [Address] #Decommission partitions in a meta node to other nodes

```

##### DataNode Management[¶](#datanode-management "Permalink to this headline")

```

./cli datanode list          #List information of data nodes

```

```

./cli datanode info [Address]         #Show detail information of a data node

```

```

./cli datanode decommission [Address]   #Decommission partitions in a data node to other nodes

```

##### DataPartition Management[¶](#datapartition-management "Permalink to this headline")

```

./cli datapartition info [Partition ID]        #Display detail information of a data partition

```

```

./cli datapartition decommission [Address] [Partition ID]   #Decommission a replication of the data partition to a new address

```

```

./cli datapartition add-replica [Address] [Partition ID]    #Add a replication of the data partition on a new address

```

```

./cli datapartition del-replica [Address] [Partition ID]    #Delete a replication of the data partition from a fixed address

```

```

./cli datapartition check    #Diagnose partitions, display the partitions those are corrupt or lack of replicas

```

##### MetaPartition Management[¶](#metapartition-management "Permalink to this headline")

```

./cli metapartition info [Partition ID]        #Display detail information of a meta partition

```

```

./cli metapartition decommission [Address] [Partition ID]   #Decommission a replication of the meta partition to a new address

```

```

./cli metapartition add-replica [Address] [Partition ID]    #Add a replication of the meta partition on a new address

```

```

./cli metapartition del-replica [Address] [Partition ID]    #Delete a replication of the meta partition from a fixed address

```

```

./cli metapartition check    #Diagnose partitions, display the partitions those are corrupt or lack of replicas

```

##### Config Management[¶](#config-management "Permalink to this headline")

```

./cli config info     #Show configurations of cli

```

```

./cli config set [flags]    #Set configurations of cli

Flags:

    --addr      string      #Specify master address [{HOST}:{PORT}]

    --timeout   uint16      #Specify timeout for requests [Unit: s] (default 60)

```

##### Completion Management[¶](#completion-management "Permalink to this headline")

```

./cli completion      #Generate bash completions

```

##### Volume Management[¶](#volume-management "Permalink to this headline")

```

./cli volume create [VOLUME NAME] [USER ID] [flags]     #Create a new volume

Flags:

    --capacity uint                                     #Specify volume capacity [Unit: GB] (default 10)

    --dp-size  uint                                     #Specify size of data partition size [Unit: GB] (default 120)

    --follower-read                                     #Enable read form replica follower (default true)

    --mp-count int                                      #Specify init meta partition count (default 3)

    -y, --yes                                           #Answer yes for all questions

```

```

./cli volume delete [VOLUME NAME] [flags]               #Delete a volume from cluster

Flags:

    -y, --yes                                           #Answer yes for all questions

```

```

./cli volume info [VOLUME NAME] [flags]                 #Show volume information

Flags:

    -d, --data-partition                                #Display data partition detail information

    -m, --meta-partition                                #Display meta partition detail information

```

```

./cli volume add-dp [VOLUME] [NUMBER]                   #Create and add more data partition to a volume

```

```

./cli volume list                                       #List cluster volumes

```

```

./cli volume transfer [VOLUME NAME] [USER ID] [flags]   #Transfer volume to another user. (Change owner of volume)

Flags：

    -f, --force                                         #Force transfer without current owner check

    -y, --yes                                           #Answer yes for all questions

```

##### User Management[¶](#user-management "Permalink to this headline")

```

./cli user create [USER ID] [flags]         #Create a new user

Flags：

    --access-key string                     #Specify user access key for object storage interface authentication

    --secret-key string                     #Specify user secret key for object storage interface authentication

    --password string                       #Specify user password

    --user-type string                      #Specify user type [normal | admin] (default "normal")

    -y, --yes                               #Answer yes for all questions

```

```

./cli user delete [USER ID] [flags]         #Delete specified user

Flags：

    -y, --yes                               #Answer yes for all questions

```

```

./cli user info [USER ID]                   #Show detail information about specified user

```

```

./cli user list                             #List cluster users

```

```

./cli user perm [USER ID] [VOLUME] [PERM]   #Setup volume permission for a user

                                            #The value of [PERM] is READONLY, RO, READWRITE, RW or NONE

```

```

./cli user update [USER ID] [flags]         #Update information about specified user

Flags：

    --access-key string                     #Update user access key

    --secret-key string                     #Update user secret key

    --user-type string                      #Update user type [normal | admin]

    -y, --yes                               #Answer yes for all questions

```

##### Compatibility Test[¶](#compatibility-test "Permalink to this headline")

```

./cli cptest meta [Snapshot Path] [Host] [Partition ID]         #Metadata compatibility test

Parameters：

        [Snapshot Path] string                     #The path which snapshot file located

        [Host] string                              #The metanode host which generated the snapshot file

        [Partition ID] string                      #The meta partition ID which to be compared

```

Example:

> 

> 1. Use the old version to prepare metadata, stop writing metadata,after waiting for the latest snapshot to be generated(about 5 minutes), copy the snapshot file to the local machine

> 2. Execute the metadata comparison command on local machine

> 

> 

> 

> ```

> [Verify result]

> All dentry are consistent

> All inodes are consistent

> All meta has checked

> 

> ```

> 

> 

> 

Use Cases[¶](#use-cases "Permalink to this headline")

-----------------------------------------------------

ChubaoFS is a distributed file system that is compatible with most POSIX file system semantics. When ChubaoFS is mounted, it can be as simple as using a local file system. Basically, it can be used in any case where a file system is needed, replacing the local file system, and realizing infinitely expandable storage without physical boundaries. It has been applied in various scenarios, and the following are some of the extracted scenes.

### Machine Learning[¶](#machine-learning "Permalink to this headline")

Disadvantages of using a local disk to store training data sets:

* The local disk space is small, and there are multiple models. The training data set of each model reaches the TB level. If you use a local disk to store the training data set, you need to reduce the size of the training data set.

* Training data sets need to be updated frequently, requiring more disk space.

* Risk of loss of training data set if machine crashes.

The advantages of using ChubaoFS to store training data sets:

* Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.

* Multiple replicas of data to ensure high data reliability without worrying about losing data.

* Compatible with POSIX file system interface, no changes required by the application.

### ElasticSearch[¶](#elasticsearch "Permalink to this headline")

Using local disks to store data often encounters the following problems:

* Disk usage is uneven and disk IO cannot be fully utilized.

* Local disk space is limited.

The advantages of using ChubaoFS as a backend storage:

* Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.

* Disk IO usage is uniform and disk IO is fully utilized.

* Ensure data is highly reliable without worrying about losing data.

### Nginx Log Storage[¶](#nginx-log-storage "Permalink to this headline")

Do you often worry about running out of local disk space? With ChubaoFS, you can store datas in a distributed file system without worrying about running out of disk space.

If you use a local disk to store logs, you may often worry about the following issues:

* Docker local disk space is small.

* If the docker container crashes, the log is lost and unrecoverable.

* The mixed deployment of physical machine and docker machine is difficult to manage and has high operation and maintenance cost.

The advantages of using ChubaoFS to store nginx logs are:

* The disk space is unlimited and easy to expand. The disk capacity is automatically expanded according to the percentage of disk usage. The storage system can be expanded on demand, which greatly saves storage costs.

* Ensure data is highly reliable and do not worry about losing data.

* Multiple replicas to solve the problem of unable to write to the log caused by disk error and datanode crashes.

* Compatible with the POSIX file system interface, no changes required by the application.

* The operation and maintenance of ChubaoFS are simple, and one person can easily manage the cluster of tens of thousands of machines.

### Spark[¶](#spark "Permalink to this headline")

In the big data set scenario, are you worried about the amount of data stored in Spark intermediate calculation results in order to carefully calculate each task? You can store the shuffle results to ChubaoFS, and no longer worry about the disk has no free space which causes the task to fail. This enables the separation of storage and computation.

The pain points of using local disk to store shuffle intermediate results:

* Insufficient disk space.

* Too many temporary directory files to create new files.

The advantages of using ChubaoFS:

* Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.

* Meta node manages file metadata, which can be expanded horizontally and the number of files is unlimited.

### MySQL Database Backup[¶](#mysql-database-backup "Permalink to this headline")

Disadvantages of using OSS(Object Storage Service) to back up MySQL database:

* Need to use the OSS SDK or RESTful API to develop backup programs which increases the difficulty of operation and maintenance.

* If backup file fails, troubleshooting is more difficult.

* After backing up files to OSS, it is not convenient to check whether the files are successfully uploaded.

* Backup files are processed by multiple layers of services, which affects performance.

Advantages of using ChubaoFS to back up MySQL databases:

* Easy to use, compatible with POSIX file interface, and can be used as a local file system.

* Complete and detailed operation logs are stored in the local file system, making it easy to troubleshoot problems.

* Simply execute the ls command to verify that the file was successfully uploaded.

* Supports PageCache and WriteCache, and file read and write performance is significantly improved compared to OSS.

Performance[¶](#performance "Permalink to this headline")

---------------------------------------------------------

### Environment[¶](#environment "Permalink to this headline")

**Cluster Information**

| Instance | Nodes | CPU | Memory | Storage | Network | Description |

| Master | 3 | 32 | 32 GB | 120 GB SSD | 10 Gb/s |  |

| MetaNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |

| DataNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |

**Volume Setup**

| Parameter | Default | Recommend | Description |

| FollowerRead | True | True |  |

| Capacity | 10 GB | 300 000 000 GB |  |

| Data Replica Number | 3 | 3 |  |

| Meta Replica Number | 3 | 3 |  |

| Data Partition Size | 120 GB | 120 GB | Logical upper limit with no pre-occupied space. |

| Data Partition Count | 10 | 1500 |  |

| Meta Partition Count | 3 | 10 |  |

| Cross Zone | False | False |  |

Set volume parameters by following:

```

$ cfs-cli volume create test-vol {owner} --capacity=300000000 --mp-count=10

Create a new volume:

  Name                : test-vol

  Owner               : ltptest

  Dara partition size : 120 GB

  Meta partition count: 10

  Capacity            : 300000000 GB

  Replicas            : 3

  Allow follower read : Enabled

Confirm (yes/no)[yes]: yes

Create volume success.

$ cfs-cli volume add-dp test-vol 1490

```

**client configuration**

| Parameter | Default | Recommend |

| rate limit | -1 | -1 |

```

#get current iops, default:-1(no limits on iops)：

$ http://[ClientIP]:[ProfPort]/rate/get

#set iops

$ http://[ClientIP]:[ProfPort]/rate/set?write=800&read=800

```

### Small File Performance and Scalability[¶](#small-file-performance-and-scalability "Permalink to this headline")

Small file operation performance and scalability benchmark test by [mdtest](https://github.com/LLNL/mdtest).

**Setup**

```

#!/bin/bash

set -e

TARGET\_PATH="/mnt/test/mdtest" # mount point of ChubaoFS volume

for FILE_SIZE in 1024 2048 4096 8192 16384 32768 65536 131072 # file size

do

mpirun --allow-run-as-root -np 512 --hostfile hfile64 mdtest -n 1000 -w $i -e $FILE\_SIZE -y -u -i 3 -N 1 -F -R -d $TARGET\_PATH;

done

```

**Benchmark**

[![Small File Benchmark](_images/cfs-small-file-benchmark.png)](_images/cfs-small-file-benchmark.png)

| File Size (KB) | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |

| Creation (TPS) | 70383 | 70383 | 73738 | 74617 | 69479 | 67435 | 47540 | 27147 |

| Read (TPS) | 108600 | 118193 | 118346 | 122975 | 116374 | 110795 | 90462 | 62082 |

| Removal (TPS) | 87648 | 84651 | 83532 | 79279 | 85498 | 86523 | 80946 | 84441 |

| Stat (TPS) | 231961 | 263270 | 264207 | 252309 | 240244 | 244906 | 273576 | 242930 |

### IO Performance and Scalability[¶](#io-performance-and-scalability "Permalink to this headline")

IO Performance and benchmark scalability test by [fio](https://github.com/axboe/fio).

*Note: Multiple clients mount the same volume. And the process refers to the fio process.*

#### 1. Sequential Read[¶](#sequential-read "Permalink to this headline")

**Setup**

```

#!/bin/bash

fio -directory={} \

    -ioengine=psync \

    -rw=read \  # sequential read

    -bs=128k \  # block size

    -direct=1 \ # enable direct IO

    -group_reporting=1 \

    -fallocate=none \

    -time_based=1 \

    -runtime=120 \

    -name=test_file_c{} \

    -numjobs={} \

    -nrfiles=1 \

    -size=10G

```

**Bandwidth(MB/s)**

[![Sequential Read Bandwidth (MB/s)](_images/cfs-fio-sequential-read-bandwidth.png)](_images/cfs-fio-sequential-read-bandwidth.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 148.000 | 626.000 | 1129.000 | 1130.000 |

| 2 Clients | 284.000 | 1241.000 | 2258.000 | 2260.000 |

| 4 Clients | 619.000 | 2640.000 | 4517.000 | 4515.000 |

| 8 Clients | 1193.000 | 4994.000 | 9006.000 | 9034.000 |

**IOPS**

[![Sequential Read IOPS](_images/cfs-fio-sequential-read-iops.png)](_images/cfs-fio-sequential-read-iops.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 1180.000 | 5007.000 | 9031.000 | 9040.000 |

| 2 Clients | 2275.000 | 9924.000 | 18062.000 | 18081.000 |

| 4 Clients | 4954.000 | 21117.000 | 36129.000 | 36112.000 |

| 8 Clients | 9531.000 | 39954.000 | 72048.000 | 72264.000 |

**Latency(Microsecond)**

[![Sequential Read Latency (Microsecond)](_images/cfs-fio-sequential-read-latency.png)](_images/cfs-fio-sequential-read-latency.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 842.200 | 794.340 | 1767.310 | 7074.550 |

| 2 Clients | 874.255 | 801.690 | 1767.370 | 7071.715 |

| 4 Clients | 812.363 | 760.702 | 1767.710 | 7077.065 |

| 8 Clients | 837.707 | 799.851 | 1772.620 | 7076.967 |

#### 2. Sequential Write[¶](#sequential-write "Permalink to this headline")

**Setup**

```

#!/bin/bash

fio -directory={} \

    -ioengine=psync \

    -rw=write \ # sequential write

    -bs=128k \  # block size

    -direct=1 \ # enable direct IO

    -group_reporting=1 \

    -fallocate=none \

    -name=test_file_c{} \

    -numjobs={} \

    -nrfiles=1 \

    -size=10G

```

**Bandwidth(MB/s)**

[![Sequential Write Bandwidth (MB/s)](_images/cfs-fio-sequential-write-bandwidth.png)](_images/cfs-fio-sequential-write-bandwidth.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 52.200 | 226.000 | 956.000 | 1126.000 |

| 2 Clients | 104.500 | 473.000 | 1763.000 | 2252.000 |

| 4 Clients | 225.300 | 1015.000 | 2652.000 | 3472.000 |

| 8 Clients | 480.600 | 1753.000 | 3235.000 | 3608.000 |

**IOPS**

[![Sequential Write IOPS](_images/cfs-fio-sequential-write-iops.png)](_images/cfs-fio-sequential-write-iops.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 417 | 1805 | 7651 | 9004 |

| 2 Clients | 835 | 3779 | 14103 | 18014 |

| 4 Clients | 1801 | 8127 | 21216 | 27777 |

| 8 Clients | 3841 | 14016 | 25890 | 28860 |

**Latency(Microsecond)**

[![Sequential Write Latency (Microsecond)](_images/cfs-fio-sequential-write-latency.png)](_images/cfs-fio-sequential-write-latency.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 2385.400 | 2190.210 | 2052.360 | 7081.320 |

| 2 Clients | 2383.610 | 2081.850 | 2233.790 | 7079.450 |

| 4 Clients | 2216.305 | 1947.688 | 2946.017 | 8842.903 |

| 8 Clients | 2073.921 | 2256.120 | 4787.496 | 17002.425 |

#### 3. Random Read[¶](#random-read "Permalink to this headline")

**Setup**

```

#!/bin/bash

fio -directory={} \

    -ioengine=psync \

    -rw=randread \ # random read

    -bs=4k \       # block size

    -direct=1 \    # enable direct IO

    -group_reporting=1 \

    -fallocate=none \

    -time_based=1 \

    -runtime=120 \

    -name=test_file_c{} \

    -numjobs={} \

    -nrfiles=1 \

    -size=10G

```

**Bandwidth(MB/s)**

[![Random Read Bandwidth (MB/s)](_images/cfs-fio-random-read-bandwidth.png)](_images/cfs-fio-random-read-bandwidth.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 6.412 | 39.100 | 216.000 | 534.000 |

| 2 Clients | 14.525 | 88.100 | 409.000 | 1002.000 |

| 4 Clients | 33.242 | 200.200 | 705.000 | 1693.000 |

| 8 Clients | 59.480 | 328.300 | 940.000 | 2369.000 |

**IOPS**

[![Random Read IOPS](_images/cfs-fio-random-read-iops.png)](_images/cfs-fio-random-read-iops.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 1641 | 10240 | 56524.800 | 140288 |

| 2 Clients | 3718 | 23142.4 | 107212.8 | 263168 |

| 4 Clients | 8508 | 52428.8 | 184627.2 | 443392 |

| 8 Clients | 15222 | 85072.8 | 246681.6 | 621056 |

**Latency(Microsecond)**

[![Random Read Latency (Microsecond)](_images/cfs-fio-random-read-latency.png)](_images/cfs-fio-random-read-latency.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 603.580 | 395.420 | 287.510 | 466.320 |

| 2 Clients | 532.840 | 351.815 | 303.460 | 497.100 |

| 4 Clients | 469.025 | 317.140 | 355.105 | 588.847 |

| 8 Clients | 524.709 | 382.862 | 530.811 | 841.985 |

#### 4. Random Write[¶](#random-write "Permalink to this headline")

**Setup**

```

#!/bin/bash

fio -directory={} \

    -ioengine=psync \

    -rw=randwrite \ # random write

    -bs=4k \        # block size

    -direct=1 \     # enable direct IO

    -group_reporting=1 \

    -fallocate=none \

    -time_based=1 \

    -runtime=120 \

    -name=test_file_c{} \

    -numjobs={} \

    -nrfiles=1 \

    -size=10G

```

**Bandwidth(MB/s)**

[![Random Write Bandwidth (MB/s)](_images/cfs-fio-random-write-bandwidth.png)](_images/cfs-fio-random-write-bandwidth.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 3.620 | 17.500 | 118.000 | 318.000 |

| 2 Clients | 7.540 | 44.800 | 230.000 | 476.000 |

| 4 Clients | 16.245 | 107.700 | 397.900 | 636.000 |

| 8 Clients | 39.274 | 208.100 | 487.100 | 787.100 |

**IOPS**

[![Random Write IOPS](_images/cfs-fio-random-write-iops.png)](_images/cfs-fio-random-write-iops.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 926.000 | 4476.000 | 31027.200 | 83251.200 |

| 2 Clients | 1929.000 | 11473.000 | 60313.600 | 124620.800 |

| 4 Clients | 4156.000 | 27800.000 | 104243.200 | 167014.400 |

| 8 Clients | 10050.000 | 53250.000 | 127692.800 | 206745.600 |

**Latency**

[![Random Write Latency](_images/cfs-fio-random-write-latency.png)](_images/cfs-fio-random-write-latency.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 1073.150 | 887.570 | 523.820 | 784.030 |

| 2 Clients | 1030.010 | 691.530 | 539.525 | 1042.685 |

| 4 Clients | 955.972 | 575.183 | 618.445 | 1552.205 |

| 8 Clients | 789.883 | 598.393 | 1016.185 | 2506.424 |

### Metadata Performance and Scalability[¶](#metadata-performance-and-scalability "Permalink to this headline")

Metadata performance and scalability benchmark test by [mdtest](https://github.com/LLNL/mdtest).

**Setup**

```

#!/bin/bash

TEST\_PATH=/mnt/cfs/mdtest # mount point of ChubaoFS volume

for CLIENTS in 1 2 4 8 # number of clients

do

mpirun --allow-run-as-root -np $CLIENTS --hostfile hfile01 mdtest -n 5000 -u -z 2 -i 3 -d $TEST\_PATH;

done

```

**Dir Creation**

[![Dir Creation](_images/cfs-mdtest-dir-creation.png)](_images/cfs-mdtest-dir-creation.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 448.618 | 2421.001 | 14597.97 | 43055.15 |

| 2 Clients | 956.947 | 5917.576 | 28930.431 | 72388.765 |

| 4 Clients | 2027.02 | 13213.403 | 54449.056 | 104771.356 |

| 8 Clients | 4643.755 | 27416.904 | 89641.301 | 119542.62 |

**Dir Removal**

[![Dir Removal](_images/cfs-mdtest-dir-removal.png)](_images/cfs-mdtest-dir-removal.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 399.779 | 2118.005 | 12351.635 | 34903.672 |

| 2 Clients | 833.353 | 5176.812 | 24471.674 | 50242.973 |

| 4 Clients | 1853.617 | 11462.927 | 46413.313 | 91128.059 |

| 8 Clients | 4441.435 | 24133.617 | 74401.336 | 115013.557 |

**Dir Stat**

[![Dir Stat](_images/cfs-mdtest-dir-stat.png)](_images/cfs-mdtest-dir-stat.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 283232.761 | 1215309.524 | 4231088.104 | 12579177.02 |

| 2 Clients | 572834.143 | 2169669.058 | 8362749.217 | 18120970.71 |

| 4 Clients | 1263474.549 | 3333746.786 | 10160929.29 | 31874265.88 |

| 8 Clients | 2258670.069 | 8715752.83 | 22524794.98 | 77533648.04 |

**File Creation**

[![File Creation](_images/cfs-mdtest-file-creation.png)](_images/cfs-mdtest-file-creation.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 448.888 | 2400.803 | 13638.072 | 27785.947 |

| 2 Clients | 925.68 | 5664.166 | 25889.163 | 50434.484 |

| 4 Clients | 2001.137 | 12986.968 | 50330.952 | 91387.825 |

| 8 Clients | 4479.831 | 25933.437 | 86667.966 | 112746.199 |

**File Removal**

[![File Removal](_images/cfs-mdtest-file-removal.png)](_images/cfs-mdtest-file-removal.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 605.143 | 3678.138 | 18631.342 | 47035.912 |

| 2 Clients | 1301.151 | 8365.667 | 34005.25 | 64860.041 |

| 4 Clients | 3032.683 | 14017.426 | 50938.926 | 80692.761 |

| 8 Clients | 7170.386 | 32056.959 | 68761.908 | 88357.563 |

**Tree Creation**

[![Tree Creation](_images/cfs-mdtest-tree-creation.png)](_images/cfs-mdtest-tree-creation.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 305.778 | 229.562 | 86.299 | 23.917 |

| 2 Clients | 161.31 | 211.119 | 76.301 | 24.439 |

| 4 Clients | 260.562 | 223.153 | 81.209 | 23.867 |

| 8 Clients | 350.038 | 220.744 | 81.621 | 17.144 |

**Tree Removal**

[![Tree Removal](_images/cfs-mdtest-tree-removal.png)](_images/cfs-mdtest-tree-removal.png)

|  | 1 Process | 4 Processes | 16 Processes | 64 Processes |

| 1 Client | 137.462 | 70.881 | 31.235 | 7.057 |

| 2 Clients | 217.026 | 113.36 | 23.971 | 7.128 |

| 4 Clients | 231.622 | 113.539 | 30.626 | 7.299 |

| 8 Clients | 185.156 | 83.923 | 20.607 | 5.515 |

Integrity[¶](#integrity "Permalink to this headline")

-----------------------------------------------------

* Linux Test Project / fs

Workload[¶](#workload "Permalink to this headline")

---------------------------------------------------

* Database backup

* Java application logs

* Code git repo

* Database systems

MyRocks,

MySQL Innodb,

HBase,

Scalability[¶](#scalability "Permalink to this headline")

---------------------------------------------------------

* Volume Scalability: tens to millions of cfs volumes

* Metadata Scalability: a big volume with billions of files/directories

Contributing to ChubaoFS[¶](#contributing-to-chubaofs "Permalink to this headline")

-----------------------------------------------------------------------------------

### Bug Reports[¶](#bug-reports "Permalink to this headline")

Please make sure the bug is not already reported by [searching the repository](https://github.com/chubaofs/chubaofs/search?q=&type=Issues&utf8=%E2%9C%93) with reasonable keywords. Then, [open an issue](https://github.com/chubaofs/chubaofs/issues) with steps to reproduce.

### Patch Guidelines[¶](#patch-guidelines "Permalink to this headline")

In order for the patch to be accepted with higher possibility, there are a few things you might want to pay attention to:

* [Filesystem stress tests](https://github.com/linux-test-project/ltp/blob/master/runtest/fs) is required before opening a pull request by

```

runltp -f fs -d [MOUNTPOINT]

```

* A good commit message describing the bug fix or the new feature is preferred.

* [DCO](https://github.com/apps/dco) is required, so please add Signed-off-by to the commit.

### Credits[¶](#credits "Permalink to this headline")

Sections of this documents have been borrowed from [CoreDNS](https://github.com/coredns/coredns/blob/master/CONTRIBUTING.md) and [Fluentd](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md)

Environment Requirements and Capacity Planning[¶](#environment-requirements-and-capacity-planning "Permalink to this headline")

-------------------------------------------------------------------------------------------------------------------------------

### Environment Requirements[¶](#environment-requirements "Permalink to this headline")

The following table lists the system and hardware requirements of the performance test environment and production environment. You can also refer to the capacity planning chapter to accurately customize the deployment plan based on your cluster’s actual capacity planning.

Note that since the DataNode used some features of linux kernal, so that the kernel version of servers which used for deploy DataNode must be later than 3.10.

In order to speed up read and write of meta data, the meta data is stored in memory, while the DataNode mainly occupies disk resources. To maximize the use of node resources, you can mix-deploy DataNode and MetaNode on the same node.

| Role | Spec | Test | Product |

| Master | CPU | >=4C | >=8C |

|  | Memory | >=4G | >=16G |

|  | Kernel | >=3.10 | >=3.10 |

|  | Nodes | 3 | 3 |

| DataNode | CPU | >=4C | >=4C |

|  | Memory | >=4G | >=8G |

|  | Kernel | >=3.10 | >=3.10 |

|  | Disk Capacity | >=1TB | >=2TB |

|  | Disk Type | sata | ssd | sata | ssd |

|  | File System | xfs | etx4 | xfs | etx4 |

|  | Nodes | >=3 | 100~1000 |

| MetaNode | CPU | >=4C | >=8C |

|  | Memory | >=8G | >=16G |

|  | Kernel | >=3.10 | >=3.10 |

|  | Nodes | >=4 | 100~1000 |

| Client | CPU | >=2C | >=2C |

|  | Memory | >=4G | >=1G |

|  | Kernel | >=3.10 | >=3.10 |

### Capacity Planning[¶](#capacity-planning "Permalink to this headline")

First of all, you have to assess the highest expected number of files and storage capacity of the cluster in the future.

Secondly, you need to know the machine resources you currently have, and the total memory, CPU cores, and disks on each machine.

If you have been clear about those statistics, you can use the empirical reference values ​​given in the second section to see which scale your current environment belongs to, what file size it can carry,or you need to prepare for the current file experience requirements How many resources to prevent frequent expansion of machine resources.

| Total File Count | Total File Size | Total memory | Total Disk Space |

| --- | --- | --- | --- |

| 1,000,000,000 | 10PB | 2048 GB | 10PB |

The higher the proportion of large files, the greater the MetaNode pressure.

Of course, if you feel that the current resources are adequately used, you don’t need to meet the capacity growth requirements all at once. Then you can pay attention to the capacity warning information of MetaNode/DataNode in time. When the memory or disk is about to run out, dynamically increase MetaNode/DataNode to adjust the capacity. In other words, if you find that the disk space is not enough, you can increase the disk or increase DataNode. If you find that all MetaNode memory is too full, you can increase MetaNode to relieve memory pressure.

### Multi-Zone Deploy[¶](#multi-zone-deploy "Permalink to this headline")

If you want the cluster to support fault tolerance in the computer room, you can deploy a ChubaoFS cluster across computer rooms. At the same time, it should be noted that since the communication delay between computer rooms is higher than that of a single computer room, if the requirements for high availability are greater than low latency, you can choose a cross-computer room deployment solution. If you have higher performance requirements, it is recommended to deploy clusters in a single computer room.

Configuration scheme: Modify the zoneName parameter in the DataNode/MetaNode configuration file, specify the name of the computer room where you are, and then start the DataNode/MetaNode process, the computer room will be stored and recorded by the Master along with the registration of DataNode/MetaNode.

Create a single zone volume:

```

$ cfs-cli volume create {name} --zone-name={zone}

```

In order to prevent volume initialization failure in a single computer room, please ensure that the DataNode of a single computer room is not less than 3 and MetaNode is not less than 4.

Create a cross-zone volume:

```

$ cfs-cli volume create {name} --cross-zone=true

```

Q&A[¶](#q-a "Permalink to this headline")

-----------------------------------------

* If you are new to ChubaoFS and want to start quickly, please refer to [Run Cluster on Docker](index.html#document-user-guide/docker)

* If you are interested in ChubaoFS and want to conduct a performance test before applying it in the product environment, please refer to [Performance](index.html#document-evaluation)

* If you have completed the assessment of ChubaoFS and want to put it into product environment, and want to learn how to carry out capacity planning and environment preparing, please refer to [Environment Requirements and Capacity Planning](index.html#document-env)

* If you want to know about best practices of ChubaoFS, please refer to [Use Cases](index.html#document-use-case)

* If you encounter some problems in the product environment, the following content may help to solve your problems.

For the convenience of description, we define the following keyword abbreviations

| Full Name | Abbreviation |

| --- | --- |

| Data Partition | dp |

| Meta Partition | mp |

| Data Partition Replica | dpr |

| Meta Partition Replica | mpr |

| NodeSet | ns |

| DataNode | dn |

| MetaNode | mn |

### Compile[¶](#compile "Permalink to this headline")

1. Compile ChubaoFS on one machine, but it cannot be started when deployed to other machines.

First please make sure to use the `PORTABLE=1 make static\_lib` command to compile RocksDB, then use the `ldd` command to check whether the dependent libraries are installed on the machine. After installing the missing libraries, execute the `ldconfig` command.

2. A problem similar to undefined reference to ‘ZSTD\_versionNumber’.

There are two solutions.

* It can be compiled by adding the specified library to CGO\_LDFLAGS.

For example: `CGO\_LDFLAGS="-L/usr/local/lib -lrocksdb -lzstd"`. This requires that the `zstd` library is also installed on other deployment machines.

* Remove the script that automatically detects whether the ‘zstd’ library is installed.

Example of file location: rockdb-5.9.2/build\_tools/build\_detect\_platform

The deleted content is as follows.

```

# Test whether zstd library is installed

    $CXX $CFLAGS $COMMON\_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF

 #include <zstd.h>

 int main() {}

EOF

    if [ "$?" = 0 ]; then

        COMMON\_FLAGS="$COMMON\_FLAGS -DZSTD"

        PLATFORM\_LDFLAGS="$PLATFORM\_LDFLAGS -lzstd"

        JAVA\_LDFLAGS="$JAVA\_LDFLAGS -lzstd"

    fi

```

### Node & Disk Failure[¶](#node-disk-failure "Permalink to this headline")

If a node or disk fails, you can take the failed node or disk offline by decommission command.

1. Decommission Datanode/Metanode

> 

> 

> ```

> $ cfs-cli metanode/datanode decommission 192.168.0.21:17210

> 

> ```

> 

> 

> 

> 	* At what time to decommission mn/dn？

> If a node fails and cannot reboot, it has to be removed from the cluster and the partitions on this machine is automatically migrated to other healthy nodes.

> 

> 

> 

> 	* The decommission caused an acute increasing on disk io and network io.

> The decommission will trigger automatic partition migration witch consumes a lot of network resources. Therefore, try to execute the decommission during off peak hours and avoid taking multiple nodes decommission at the same time.

> 

> 

> 

> 	* What is the sign of decommission completion？

> 	

> 	

> 	

> 	> 

> 	> 

> 	> ```

> 	> $ cfs-cli datapartition check

> 	> 

> 	> ```

> 	> 

> 	> 

> 	> If bad partition ids is empty, the decommission is done.

> 	> 

> 	> 

> 	> 

> 	> 	+ Common error1: There is no mn available, all mn memory or disks are full at this time, new mn needs to be added to the cluster

> 	> 	+ Common error2: The port number is wrong, the identifier of each mn should be a combination of ip+port , and port is the listen port in the mn configuration file, which cannot be wrong

> 	>

>

2. Decommission Disk

If a disk fails but the node is healthy, you can partially decommission the disk. Similarly, as with dn/mn decommission, please try to avoid multiple decommission operations during off peak hours.

```

$ cfs-cli disk decommission {disk} 192.168.0.11:17310

```

If correct, all the dp on the disk is migrated to other disks or nodes.

The common error is similar to decommission dn.

3. Decommission Data Partition/Meta Partition

> 

> 

> ```

> $ cfs-cli datapartition decommission 192.168.0.11:17310 {Partition ID}

> 

> ```

> 

> 

>

If correct, the dp is migrated to other nodes.

> 

> * At what time to decommission a partition？

> 

> 

> 

> > 

> > 

> > 	+ There is too many partitions on one node, decommission some partitions to reduce the pressure on this node.

> > 	+ Take a small step instead of dn/mn decommission to prevent overloading of the cluster.

> >

> * The common error is similar to decommission dn/mn.

> 

> 

> 

4. If the disk is full, will it explode?

It is recommended set reserved space in the dn startup json file, it is behind the disk path parameter, disk: “{disk path}:{reserved space}”. If the remaining space is less than reserved space, the dn turns readonly.

### Performance of Data and Metadata[¶](#performance-of-data-and-metadata "Permalink to this headline")

1. How does ChubaoFS compare with its alternatives ?

* Ceph

Ceph (<https://ceph.com/>) is a widely-used free-software storage platform. It can be configured for object storage, block storage, as well as file system. But many nice features provided by

Ceph (e.g., various storage backends) also make it very complicated and difficult to learn and deploy. In addition, in certain scenarios such as metadata operations and small file operations, its performance in a multi-client environment can be hard to optimize.

* GFS and HDFS

GFS and its open source implementation HDFS (<https://github.com/apache/hadoop>) are designed for storing large files with sequential access.

Both of them adopt the master-slave architecture, where the single master stores all the file metadata. Unlike GFS and HDFS, ChubaoFS employs a separate metadata subsystem to provide a scalable solution for the metadata storage so that the resource manager has less chance to become the bottleneck.

* Hadoop Ozone

Hadoop Ozone is a scalable distributed object storage system designed for Hadoop. It was originally proposed to solve the problem of HDFS namenode expansion. It reconstructs the namenode metadata management part of hdfs and reuses the datanode of hdfs.

ChubaoFS has many of the same design concepts like ozone such as: supporting for volume isolation, compatible with both raft/master-slave synchronous replication mechanisms, implenting for s3-compatible interfaces. In addition, ChubaoFS’s POSIX fuse-client interface supports random file reading and writing, and optimizes reading and writing of small files.

* Haystack

Haystack from Facebook takes after log-structured filesystems to serve long tail of requests seen by sharing photos in a large social network. The key insight is to avoid disk operations when accessing metadata.

ChubaoFS adopts similar ideas by putting the file metadata into the main memory.

However, different from Haystack, the actually physical offsets instead of logical indices of the file contents are stored in the memory,

and deleting a file is achieved by the punch hole interface provided by the underlying file system instead of relying on the garbage collector to perform merging and compacting regularly for more efficient disk utilization. In addition, Haystack does not guarantee the strong consistency among the replicas when deleting the files, and it needs to perform merging and compacting regularly for more efficient disk utilization, which could be a performance killer.

ChubaoFS takes a different design principle to separate the storage of file metadata and contents. In this way, we can have more flexible and cost-effective deployments of meta and data nodes.

* Public Cloud

Windows Azure Storage (<https://azure.microsoft.com/en-us/>) is a cloud storage system that provides strong consistency and multi-tenancy to the clients.

Different from ChubaoFS, it builds an extra partition layer to handle random writes before streaming data into the lower level.

AWS EFS (<https://aws.amazon.com/efs/>) is a cloud storage service that provides scalable and elastic file storage.

Depending on the use cases, there could be a considerable amount of cost associated with using these cloud storage services.

* Others

GlusterFS (<https://www.gluster.org/>) is a scalable distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. MooseFS (<https://moosefs.com/>) is a fault- tolerant, highly available, POSIX-compliant, and scalable distributed file system. However, similar to HDFS, it employs a single master to manage the file metadata.

2. If the scale of metadata is huge, how to improve cluster performance?

The metadata of ChubaoFS is stored in the memory. Expanding memory of the mn or expanding the amount of mn horizontally will significantly improve the metadata performance and support a large number of small files.

3. If a dn/mn is added to the cluster, will it be automatically rebalanced, for example the dp/mp on the old node are migrated to the new node?

No. Considering that the rebalance may cause overloading and risks data loss, it will not automatically rebalance. If you want the new node to carry more dp/mp to disperse the pressure of the old node, you can create new dp for this volume and the new dp may locate on new nodes, or you can decommission the dp on the old node.

4. There are scenes of batch delete files witch cause cluster overloads

You can set and view the background file deletion rate by following command, the default value is 0, which means unlimited. It is recommended to set markdeleterate=1000, and then dynamically adjust it according to the cpu status of the nodes in the cluster.

```

$ cfs-cli cluster info

$ cfs-cli cluster delelerate -h

 Set delete parameters

 Usage:

   cfs-cli cluster delelerate [flags]

 Flags:

       --auto-repair-rate string         DataNode auto repair rate

       --delete-batch-count string       MetaNode delete batch count

       --delete-worker-sleep-ms string   MetaNode delete worker sleep time with millisecond. if 0 for no sleep

   -h, --help                            help for delelerate

       --mark-delete-rate string         DataNode batch mark delete limit rate. if 0 for no infinity limit

```

### Capacity Management[¶](#capacity-management "Permalink to this headline")

1. What if the capacity of Volume is used-out?

```

$ cfs-cli volume expand {volume name} {capacity / GB}

```

2. How to optimize the read/write performance from the Volume side？

The more dp that can be read and written, the better read and write performance of the Volume. ChubaoFS adopts a dynamic space allocation mechanism. After creating a Volume, it will pre-allocate a certain data partition dp for the Volume. When the number of dp that can be read and written is less than 10, the dp number will be automatically expanded by a step of 10. If you want to manually increase the number of readable and writable dp, you can use the following command:

```

$ cfs-cli volume create-dp {volume name} {number}

```

The default size of a dp is 120GB. Please add dp based on the actual cost to avoid overdrawing all dp.

3. How to reclaim the excess space of Volume

```

$ cfs-cli volume shrink {volume name} {capacity / GB}

```

If the set value is less than %120 of the used amount, the operation fails.

5. What if the cluster space is not enough?

Prepare the new dn and mn, start it by json configuration file with the current master hosts, and the dn/mn will be automatically added to the cluster.

### Zone[¶](#zone "Permalink to this headline")

Setting the zone can prevent the failure of a single zone witch causes the entire cluster unavailable. If parameter zoneName is set correctly, the node will automatically join the zone(cell).

1. See zones list

```

$ cfs-cli zone list

```

2. What if accidentally set the volume zone by mistake?

```

$ cfs-cli volume update {volume name}  --zone-name={zone name}

```

2. What happens if MetaNde and DataNode do not set zone?

Most of the parameters in the cluster have default values, and the default zone name is default. Please note that there must be enough dn and mn in a zone at the same time, otherwise, creating a volume in the zone will either fail to initialize data partitions or initialize meta partitions.

3. The meaning of NodeSet？

Each zone will have several NodeSets, and each NodeSet can carry 18 nodes by default. Because ChubaoFS implements multi-raft, each node starts a raft server process, and each raft server manages m raft instances on the node. If the other replication group members of these raft members are distributed on N nodes, then the raft heartbeat will be transmitted between N nodes. As the cluster scale expands, N will become larger. Through the NodeSet restriction, the heartbeat is relatively independent within the NodeSet, which avoids the heartbeat storm of the cluster dimension. We use the multi raft and nodeset mechanisms together to avoid the problem of the raft heartbeat storm.

![nodeset](_images/nodeset.png)

4. How to distribute dp/mp of a volume in NodeSet?

dp/mp is evenly distributed in NodeSet. On dp/mp creating, it will locate on the NodeSet behind the NodeSet of last dp/mp overall.

5. How to plan the amount of NodeSet？

For 3-replicas dp/mp, a dp/mp will select the ns only when there are at least 3 available nodes in ns. count (ns) >= 18 \* n + 3

### Node status is abnormal[¶](#node-status-is-abnormal "Permalink to this headline")

View node status through cli tool

```

$ cfs-cli datanode list

[Data nodes]

ID        ADDRESS               WRITABLE    STATUS

7         192.168.0.31:17310    No          Inactive

8         192.168.0.32:17310    No          Inactive

9         192.168.0.33:17310    Yes         Active

10        192.168.0.35:17310    Yes         Active

11        192.168.0.34:17310    Yes         Active

$ cfs-cli metanode list

[Meta nodes]

ID        ADDRESS               WRITABLE    STATUS

2        192.168.0.21:17210    No          Inactive

3        192.168.0.22:17210    No          Inactive

4        192.168.0.23:17210    Yes         Active

5        192.168.0.25:17210    Yes         Active

6        192.168.0.24:17210    Yes         Active

```

1. Reasons for Datanode WRITABLE=No

> 

> * The node is waiting for the decommission to complete

> * Node disk is out of use

> * The node has just started and is restoring data locally

> 

> 

> 

2. Reasons for Metanode WRITABLE=No

> 

> * The node is waiting for the decommission to complete

> * The node memory has reached totalMemory

> * The node has just started and is restoring data locally

> 

> 

> 

3. If one of the three masters in the cluster is broken, can the remaining two restarts provide services normally?

Yes. Since the Master uses the RAFT, it can provide services normally when the number of remaining nodes exceeds 50% of the total number of nodes.

4. Reasons for STATUS=Inactive

> 

> * The network connection between the node and the master is interrupted. You need to check the network status and restore the network connection

> * The node process hangs, you need to check whether the server process of the node is abnormally terminated, at this time restart the process to recover

> 

> 

> 

### Upgrade[¶](#upgrade "Permalink to this headline")

1. Steps

> 

> 1. Download and unzip the latest binary file compression package from ChubaoFS official website <https://github.com/chubaofs/chubaofs/releases>

> 2. Freeze the cluster

> 

> 

> 

> ```

> $ cfs-cli cluster freeze true

> 

> ```

> 

> 

> 3. Confirm the startup configuration file, do not change important information such as the data directory and port in the configuration file

> 4. Stop the old server process

> 5. Start the new server process

> 6. Check that the node status recovered healthy after the upgrade, IsActive: Active

> 

> 

> 

> ```

> $ cfs-cli datanode info 192.168.0.33:17310

> [Data node info]

>  ID                  : 9

>  Address             : 192.168.0.33:17310

>  Carry               : 0.06612836801123345

>  Used ratio          : 0.0034684352702178426

>  Used                : 96 GB

>  Available           : 27 TB

>  Total               : 27 TB

>  Zone                : default

>  IsActive            : Active

>  Report time         : 2020-07-27 10:23:20

>  Partition count     : 16

>  Bad disks           : []

>  Persist partitions  : [2 3 5 7 8 10 11 12 13 14 15 16 17 18 19 20]

> 

> ```

> 

> 

> 7. Upgrade the next node (in order to reduce the impact on the client, especially in a relatively large user volume, you need to upgrade the MetaNode nodes one by one), the upgrade sequence is shown bellow

> 

> 

> ![upgrade](_images/upgrade-en.png)

> 

2. After upgrading a master, it is found that the monitoring system is not displayed in time?

Check whether the configuration of this master node is correct, especially the id; check the master error log to find out whether there is a large number of no leader errors, and query the keyword leaderChange in the master warn log to check the reason for the leader change, and then check the raft warn log for further analysis.

3. Can I modify the port number of the configuration during the upgrade?

No, ip+port constitutes the unique identifier of mn and dn instances, and will be treated as a new node after modification.

### Update Configuration Online[¶](#update-configuration-online "Permalink to this headline")

1. Update mn threshold

```

$ cfs-cli cluster set threshold { value }

```

2. Update cluster configuration

3. Update Volume configuration

```

$ cfs-cli volume set -h

Set configuration of the volume

Usage:

  cfs-cli volume set [VOLUME NAME] [flags]

Flags:

      --authenticate string    Enable authenticate

      --capacity uint          Specify volume capacity [Unit: GB]

      --enable-token string    ReadOnly/ReadWrite token validation for fuse client

      --follower-read string   Enable read form replica follower

  -h, --help                   help for set

      --replicas int           Specify volume replicas number

  -y, --yes                    Answer yes for all questions

      --zonename string        Specify volume zone name

```

4. Update log level

An interface to update the log level of master, MetaNode, and DataNode online is provided:

```

$ http://127.0.0.1:{profPort}/loglevel/set?level={log-level}

```

Supported log-level: debug,info,warn,error,critical,read,write,fatal

### Update Configuration Offline[¶](#update-configuration-offline "Permalink to this headline")

1. Update master IP

After the ip address of the three-node master is replaced, all mn, dn and other applications that reference the master ip address need to be restarted after modifying the configuration.

2. Update DataNode/MetaNode port number

It is not recommended to modify the dn/mn port. Because dn/mn is registered through ip:port in the master. If the port is modified, the master will consider it as a brand new node, and the old node is in the state of Inactive.

3. Update MetaNode totalMemory

totalMemory refers to the total memory size of MetaNode. When the memory usage of MetaNode is higher than this value, MetaNode becomes read-only. Usually this value is smaller than the node memory. If MetaNode and DataNode are deployed in a mixed deployment, extra memory needs to be reserved for the DataNode.

4. Update DataNode reservedSpace

In the dn startup configuration json file, the number in the second half of the disk parameter is the value of Reserved Space, unit(byte).

```

{ ...

"disks": [

   "/cfs/disk:10737418240"

  ],

  ...

  }

```

5. For more configurations, see [Resource Manager (Master)](index.html#document-user-guide/master) [Data Subsystem](index.html#document-user-guide/datanode) [Meta Subsystem](index.html#document-user-guide/metanode)[Client](index.html#document-user-guide/client).

### Handle Logs[¶](#handle-logs "Permalink to this headline")

1. What to do if tens of GB of logs are generated every day, which takes up too much disk space?

In a production environment, you can set the log level to warn or error, which will significantly reduce the amount of logs.

```

$ http://127.0.0.1:{profPort}/loglevel/set?level={log-level}

```

Supported log-level: debug,info,warn,error,critical,read,write,fatal

2. Datanode warn log

> 

> 

> ```

> checkFileCrcTaskErr clusterID[xxx] partitionID:xxx File:xxx badCrc On xxx:

> 

> ```

> 

> 

> Analysis: The Master schedules the dn to check the crc data every few hours. This error indicates that the crc check failed and the file data is wrong. At this time, it is necessary to further analyze the error according to the partitionID and File in the warn message and with the assistance of the info log.

> 

> 

> 

3. Datanode error log

4. Master error log

> 

> 

> ```

> clusterID[xxx] addr[xxx]_op[xx] has no response util time out

> 

> ```

> 

> 

> Analysis:The response timed out when the Master sends the [Op] command to mn or dn, check the network between Master and mn/dn; check whether the dn/mn service process is alive.

> 

> 

> 

5. Master warn log

6. Metanode error log

> 

> 

> ```

> Error metaPartition(xx) changeLeader to (xx):

> 

> ```

> 

> 

> Analysis:Leader change, a normal action.

> 

> 

> 

> ```

> inode count is not equal, vol[xxx], mpID[xx]

> 

> ```

> 

> 

> Analysis:The number of inode is inconsistent. Because as long as two of the three copies are successful, the write is successful, so there will be inconsistencies in the three copies. Check the log for the specific reason.

> 

> 

> 

7. Metanode warn log

8. Client warn log

> 

> 

> ```

> operation.go:189: dcreate: packet(ReqID(151)Op(OpMetaCreateDentry)PartitionID(0)ResultCode(ExistErr)) mp(PartitionID(1) Start(0) End(16777216) Members([192.168.0.23:17210 192.168.0.24:17210 192.168.0.21:17210]) LeaderAddr(192.168.0.23:17210) Status(2)) req({ltptest 1 1 16777218 test.log 420}) result(ExistErr)

> 

> ```

> 

> 

> Analysis: ExistErr indicates that the file name already exists during the rename operation. It is an upper-level business operation problem. Maintainers can ignore this.

> 

> 

> 

> ```

> extent_handler.go:498: allocateExtent: failed to create extent, eh(ExtentHandler{ID(xxx)Inode(xxx)FileOffset(xxxx)StoreMode(1)}) err(createExtent: ResultCode NOK, packet(ReqID(xxxxx)Op(OpCreateExtent)Inode(0)FileOffset(0)Size(86)PartitionID(xxxxx)ExtentID(xx)ExtentOffset(0)CRC(0)ResultCode(IntraGroupNetErr)) datapartionHosts(1.1.0.0:17310) ResultCode(IntraGroupNetErr))

> 

> ```

> 

> 

> Analysis: The client sends a request to create an extent to an mp and fails, it will soon try to request another mp.

> 

> 

> 

9. Client error log

> 

> 

> ```

> appendExtentKey: packet(%v) mp(%v) req(%v) result(NotExistErr)

> 

> ```

> 

> 

> Analysis: This error indicates that the file was deleted before written, which is an upper-level business operation problem. Maintainers can ignore this.

> 

> 

> 

> ```

> conn.go:103:sendToMetaPartition: retry failed req(ReqID(xxxx)Op(OpMetaInodeGet)PartitionID(0)ResultCode(Unknown ResultCode(0)))mp(PartitionID(xxxx) Start(xxx) End(xxx) Members([xxx xxxx xxxx]) LeaderAddr(xxxx) Status(2)) mc(partitionID(xxxx) addr(xxx)) err([conn.go 129] Failed to read from conn, req(ReqID(xxxx)Op(OpMetaInodeGet)PartitionID(0)ResultCode(Unknown ResultCode(0))) :: read tcp 10.196.0.10:42852->11.196.1.11:9021: i/o timeout) resp(<nil>)

> 

> ```

> 

> 

> Analysis 1:The network connection between the client and the MetaNode is abnormal. According to the error message “10.196.0.10:42852->11.196.1.11:9021”, check whether the network between the two addresses is normal

> 

> 

> Analysis 2:Check if the MetaNode process hangs on “11.196.1.11:9021”

> 

> 

> 

10. Raft warn log

11. Raft error log

> 

> 

> ```

> raft.go:446: [ERROR] raft partitionID[1105] replicaID[6] not active peer["nodeID":"6","peerID":"0","priority":"0","type":"PeerNormal"]

> 

> ```

> 

> 

> Analysis: This is caused by excessive network pressure and increased delay. After the raft election interval is exceeded, the raft replication group loses the leader. After the network is restored, re-elect the leader, the error will disappear by itself.

> 

> 

> 

### Data Loss and Consistence[¶](#data-loss-and-consistence "Permalink to this headline")

1. All data of a single dn/mn is lost

This situation can be equivalent to a dn/mn failure. You can log off the node through decommission, and then restart the node to re-register the node to the Master, and the Master will treat it as a new member.

2. Accidentally deleted file data in a dp directory in dn

dn has the function of automatically repairing data. If the data has not been repaired for a long time, you can manually restart the current dn process, which will trigger the data repair process.

### Fuse Client[¶](#fuse-client "Permalink to this headline")

1. Memory and performance optimization issues

> 

> * The Fuse client occupies too much memory, which has a large impact on other services

> 

> 

> 

> > 

> > Offline: Set the readRate and writeRate parameters in the configuration file and restart the client.

> > 

> > 

> > Online: <http:/>/{clientIP}:{profPort} /rate/set?write=800&read=800

> > 

> > 

> >

> * For more, see(<https://chubaofs.readthedocs.io/zh_CN/latest/user-guide/fuse.html>)

> 

> 

> 

2. Mount issues

> 

> * Does it support subdirectory mounting?

> 

> 

> Yes. Set subdir in the configuration file

> 

> 

> * What are the reasons for the mount failure

> 

> 

> 

> > 

> > 

> > 	+ If you see the following output,

> > 

> > ```

> > $ ... err(readFromProcess: sub-process: fusermount: exec: "fusermount": executable file not found in $PATH)

> > 

> > ```

> > 

> > 

> > Check if fuse is installed, if not, install it

> > 

> > 

> > 

> > ```

> > $ rpm –qa|grep fuse

> >   yum install fuse

> > 

> > ```

> > 

> > 

> > 

> > 	+ Check if the mount directory exists

> > 	+ Check whether the mount point directory is empty

> > 	+ Check whether the mount point has been umount

> > 	+ Check whether the mount point status is normal. If the following message appears on the mount point mnt, you need to umount first, and then start the client

> > 

> > ```

> > $ ls -lih

> > ls: cannot access 'mnt': Transport endpoint is not connected

> > total 0

> > 6443448706 drwxr-xr-x 2 root root 73 Jul 29 06:19 bin

> >  811671493 drwxr-xr-x 2 root root 43 Jul 29 06:19 conf

> > 6444590114 drwxr-xr-x 3 root root 28 Jul 29 06:20 log

> >          ? d????????? ? ?    ?     ?            ? mnt

> >  540443904 drwxr-xr-x 2 root root 45 Jul 29 06:19 script

> > 

> > ```

> > 

> > 

> > 

> > 	+ Check whether the configuration file is correct, master address, volume name and other information

> > 	+ If none of the above problems exist, locate the error through the client error log to see if it is the mount failure caused by the MetaNode or master service

> >

> 

> 

> 

3. IO issues

> 

> * IOPS is too high and the client’s memory usage exceeds 3GB or even higher. Is there any way to limit IOPS?

> 

> 

> Limit the frequency of client response to io requests by modifying the client rate limit.

> 

> 

> 

> ```

> #see current iops:

> $ http://[ClientIP]:[profPort]/rate/get

> #set iops, default:-1(no limits)

> $ http://[ClientIP]:[profPort]/rate/set?write=800&read=800

> 

> ```

> 

> 

> * io delay too high for ls or other operations

> 

> 

> 

> > 

> > 

> > 	+ Because the client reads and writes files through the http protocol, please check whether the network status is healthy

> > 	+ Check whether there is an overloaded mn, whether the mn process is hanging, you can restart mn, or expand a new mn to the cluster and take the mp on the overloaded mn decommission to relieve the pressure of mn

> >

> 

> 

> 

4. Does ChubaoFS provide strong consistence guarantees?

No.ChubaoFS has relaxed POSIX consistency semantics, i.e., instead of providing strong consistency guarantees, it only ensures sequential consistency for file/directory operations, and does not have any leasing mechanism to prevent multiple clients writing to the same file/directory. It depends on the upperlevel application to maintain a more restrict consistency level if necessary.

5. Is it feasible to kill the client to directly stop the client service

No. It is recommended to umount first. After umount, the client process will automatically stop.

