<commit_before>---
layout: post
title: 嵌套虚拟化使用
category: VMM
tags: VMM
description: 嵌套虚拟化使用
---
#  嵌套虚拟化使用

创建需要多台物理机创建openstack集群，但是又没有多余的物理，只能再一台主机中创建多个虚拟机搭建。搭建好的openstack集群又需要创建虚拟机，因此牵扯到嵌套虚拟化相关的东西（说白了就是虚拟机中创建虚拟机）。

### 实验环境

宿主机系统（host）：CentOS Linux release 7.4.1708 (Core)

libvirt：3.2.0

### 检查host是否支持KVM

```sh
lscpu |grep vmx
```

如果有，则说明此服务器支持虚拟化（有些服务器没有没有开启此特性，需要手动在BIOS中开启VT-X选项）。

如果使用嵌套虚拟化，要保证每一级的宿主机都有此特性，才能够创建虚拟机。

### 检查liunx内核版本是否支持

liunx内核版本3.X以上版本支持此特性

```
uname -a
3.10.0-693.21.1.el7.x86_64
```

### 物理服务器上开启nested支持

开启虚拟化嵌套其实就打开kvm_intel模块的参数，默认是没有开启。

查看当前系统是否支持nested

```
systool -m kvm_intel -v  | grep -i nested
nested              = "N"
或者
cat /sys/module/kvm_intel/parameters/nested
N
```

N表示没有开启，Y表示已经开启。

更改内核kvm_intel启动参数：

- 修改启动选项，重启生效

  ```
  vim /boot/grub2/grub.cfg
  ```

  在内核启动时添加 kvm-intel.nested=1

- 重新加载kvm_intel，立即不用重启，重启失效。

  ```
  lsmod|grep kvm_intel
  kvm_intel             170200  0 
  rmmod -r kvm_intel
  modprobe kvm_intel nested=1
  ```


### 创建支持嵌套虚拟化的虚拟机

#### 虚拟机xml中cpu模式的设定

- host-passthrough

  ```
  host-passthrough With this mode, the CPU visible to the guest virtual machine is exactly the same as the host physical machine CPU including elements that cause errors within libvirt. The obvious the downside of this mode is that the guest virtual machine environment cannot be reproduced on different hardware and therefore this mode is recommended with great caution. Neither model nor feature elements are allowed in this mode.
  ```

- host-model（默认模式）

  ```
  this is essentially a shortcut to copying host physical machine CPU definition from the capabilities XML into the domain XML. As the CPU definition is copied just before starting a domain, the same XML can be used on different host physical machines while still providing the best guest virtual machine CPU each host physical machine supports. Neither the match attribute nor any feature elements can be used in this mode. For more information see libvirt domain XML CPU models
  ```

- custom

  ```
  describes how the CPU is presented to the guest virtual machine. This is the default setting when the mode attribute is not specified. This mode makes it so that a persistent guest virtual machine will see the same hardware no matter what host physical machine the guest virtual machine is booted on.
  ```

三种mode的性能排序是：host-passthrough > host-model > custom

三种mode的热迁移通用性是： custom > host-model > host-passthrough

#### 模式选择

为了虚拟机中的cpu带有vmx功能，以及执行virsh capabilities时获取的能力与宿主机一致，在创建虚拟机时选择模式为host-passthrough。

```
<domain type='kvm'>
    <memory unit='GiB'>8</memory>
    <cpu mode='host-passthrough' />
    <vcpu>8</vcpu>
    ...
</domain>
```

- 默认模式下的vm：

  ```
  [root@localhost ~]# lscpu 
  Architecture:          x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
  CPU(s):                8
  On-line CPU(s) list:   0-7
  Thread(s) per core:    1
  Core(s) per socket:    1
  Socket(s):             8
  NUMA node(s):          1
  Vendor ID:             GenuineIntel
  CPU family:            6
  Model:                 13
  Model name:            QEMU Virtual CPU version 2.5+
  Stepping:              3
  CPU MHz:               2194.916
  BogoMIPS:              4389.83
  Hypervisor vendor:     KVM
  Virtualization type:   full
  L1d cache:             32K
  L1i cache:             32K
  L2 cache:              4096K
  L3 cache:              16384K
  NUMA node0 CPU(s):     0-7
  Flags:                 fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology pni cx16 x2apic hypervisor lahf_lm
  ```

  并没有带有vmx特性，不支持创建虚拟机，在加载kvm-intel也会报错。

  ```
  modprobe kvm_intel nested=1
  modprobe: ERROR: could not insert 'kvm_intel': Operation not supported
  ```

- host-passthrough

  ```
  [root@localhost ~]# lscpu |grep vmx
  Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
  [root@localhost ~]# lsmod |grep kvm<commit_msg>modify<commit_after>---
layout: post
title: 嵌套虚拟化使用
category: VMM
tags: VMM
description: 嵌套虚拟化使用
---
#  嵌套虚拟化使用

创建需要多台物理机创建openstack集群，但是又没有多余的物理，只能再一台主机中创建多个虚拟机搭建。搭建好的openstack集群又需要创建虚拟机，因此牵扯到嵌套虚拟化相关的东西（说白了就是虚拟机中创建虚拟机）。

### 实验环境

宿主机系统（host）：CentOS Linux release 7.4.1708 (Core)

libvirt：3.2.0

### 检查host是否支持KVM

```sh
lscpu |grep vmx
```

如果有，则说明此服务器支持虚拟化（有些服务器没有没有开启此特性，需要手动在BIOS中开启VT-X选项）。

如果使用嵌套虚拟化，要保证每一级的宿主机都有此特性，才能够创建虚拟机。

### 检查liunx内核版本是否支持

liunx内核版本3.X以上版本支持此特性

```
uname -a
3.10.0-693.21.1.el7.x86_64
```

### 物理服务器上开启nested支持

开启虚拟化嵌套其实就打开kvm_intel模块的参数，默认是没有开启。

查看当前系统是否支持nested

```
systool -m kvm_intel -v  | grep -i nested
nested              = "N"
或者
cat /sys/module/kvm_intel/parameters/nested
N
```

N表示没有开启，Y表示已经开启。

更改内核kvm_intel启动参数：

- 修改启动选项，重启生效

  ```
  vim /boot/grub2/grub.cfg
  ```

  在内核启动时添加 kvm-intel.nested=1

- 重新加载kvm_intel，立即不用重启，重启失效。

  ```
  lsmod|grep kvm_intel
  kvm_intel             170200  0 
  rmmod -r kvm_intel
  modprobe kvm_intel nested=1
  ```


### 创建支持嵌套虚拟化的虚拟机

#### 虚拟机xml中cpu模式的设定

- host-passthrough

  ```
  host-passthrough With this mode, the CPU visible to the guest virtual machine is 
  exactly the same as the host physical machine CPU including elements that cause 
  errors within libvirt. The obvious the downside of this mode is that the guest virtual 
  machine environment cannot be reproduced on different hardware and therefore this mode
  is recommended with great caution. Neither model nor feature elements are allowed in
  this mode.
  ```

- host-model（默认模式）

  ```
  this is essentially a shortcut to copying host physical machine CPU definition from 
  the capabilities XML into the domain XML. As the CPU definition is copied just before
  starting a domain, the same XML can be used on different host physical machines while
  still providing the best guest virtual machine CPU each host physical machine 
  supports. Neither the match attribute nor any feature elements can be used in this
  mode. For more information see libvirt domain XML CPU models
  ```

- custom

  ```
  describes how the CPU is presented to the guest virtual machine. This is the default 
  setting when the mode attribute is not specified. This mode makes it so that a 
  persistent guest virtual machine will see the same hardware no matter what host 
  physical machine the guest virtual machine is booted on.
  ```

三种mode的性能排序是：host-passthrough > host-model > custom

三种mode的热迁移通用性是： custom > host-model > host-passthrough

#### 模式选择

为了虚拟机中的cpu带有vmx功能，以及执行virsh capabilities时获取的能力与宿主机一致，在创建虚拟机时选择模式为host-passthrough。

```
<domain type='kvm'>
    <memory unit='GiB'>8</memory>
    <cpu mode='host-passthrough' />
    <vcpu>8</vcpu>
    ...
</domain>
```

- 默认模式下的vm

  ```
  [root@localhost ~]# lscpu 
  Architecture:          x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
  CPU(s):                8
  On-line CPU(s) list:   0-7
  Thread(s) per core:    1
  Core(s) per socket:    1
  Socket(s):             8
  NUMA node(s):          1
  Vendor ID:             GenuineIntel
  CPU family:            6
  Model:                 13
  Model name:            QEMU Virtual CPU version 2.5+
  Stepping:              3
  CPU MHz:               2194.916
  BogoMIPS:              4389.83
  Hypervisor vendor:     KVM
  Virtualization type:   full
  L1d cache:             32K
  L1i cache:             32K
  L2 cache:              4096K
  L3 cache:              16384K
  NUMA node0 CPU(s):     0-7
  Flags:                 fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology pni cx16 x2apic hypervisor lahf_lm
  ```

  并没有带有vmx特性，不支持创建虚拟机，在加载kvm-intel也会报错。

  ```
  modprobe kvm_intel nested=1
  modprobe: ERROR: could not insert 'kvm_intel': Operation not supported
  ```

- host-passthrough模式下的vm

  ```
  [root@localhost ~]# lscpu |grep vmx
  Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
  [root@localhost ~]# lsmod |grep kvm
{'fraction_non_alphanumeric': 0.07331211639008865, 'fraction_numerical': 0.026596953853148444, 'mean_word_length': 3.0567081604426, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3622703', 'n_tokens_mistral': 3752, 'n_tokens_neox': 3898, 'n_words': 981}