<commit_before>---
layout: post
title: Oracle RAC with libvirt on Centos 7
published: true
---

In this post I will go through the different steps to install Oracle grid and set-up a cluster. My host is a fedora and the two guests running oracle will be Centos 7.6 virtual machines. The two small difficulties when setting up a RAC using virtual servers are:

* setting up the networking (it requires a DNS)
* setting up the shared storage
<!--more-->

But before we need to have a Centos 7.6 VM (base install). Once the VM is ready we will clone it. The final set-up will be the following:

* ora01, public IP 192.168.122.10, private IP (for the interconnect): 10.0.0.1
* ora02, public IP 192.168.122.11, private IP (for the interconnect): 10.0.0.2

Scan IP's: 192.168.122.15, 192.168.122.16, 192.168.122.17 (round-robin DNS)

For the private IP's there will be a dedicated virtual network between ora01 and ora02.

For the shared storage, I will use a udev rule to make the disks naming and ownership permanent, the udev rule will use the serial id of the disk which can be assigned when attaching the disk (the disk is just a file on the host) via virt-manager. When attaching the disk to the VM's we will also check the flag "Shareable".

## creating the first guest VM

In order to create the first VM I will use the cloud init image provided by Centos. I made a post here on how to use cloud init: [Clout-init to create centos 7 guests on KVM](http://saule1508.github.io/libvirt-centos-cloud-image/). Basically, 
* create a file ora01.iso using cloud-localds, based on user-data and metadata (take care: on Fedora the module resolv_conf is not working)
* use virt-install to create the VM with 8GB and 4 cpu cores.

```bash
export VMPOOLDIR=/data2/virtpool/ora01

virt-install --name ora01 --memory 8192 --vcpus 4 --disk $VMPOOLDIR/ora01.qcow2,device=disk,bus=virtio --os-type generic --os-variant centos7.0 --virt-type kvm --network=default,model=virtio --cdrom $VMPOOLDIR/ora01.iso
```

where VMPOOLDIR is the directory I created as a storage pool.

```bash
virsh pool-create-as --name ora01 --type dir --target $VMPOOLDIR
```

At this stage I have a base guest VM, I need to add a /u01 file system and to add more swap space.

First let's create then allocate a new disk to the guest

```bash
virsh vol-create-as --pool ora01 --name ora01-disk2.qcow2 --capacity 120G --allocation 20G --format qcow2
virsh attach-disk --domain ora01 --source $VMPOOLDIR/ora01-disk2.qcow2 --target vdb --persistent --driver qemu --subdriver qcow2
```

Now let's get into the VM, create a volume group and the new file system. So inside the VM, as root:

```bash
pvcreate /dev/vdb
vgcreate vg01 /dev/vdb
lvcreate -n lv_swap -L16G vg01
mkswap /dev/vg01/lv_swap
lvcreate -n lv_u01 -l100%FREE vg01
mkfs -t xfs /dev/vg01/lv_u01
mkdir /u01
```

Use the command blkid to get the UUID of swap and lv_u01, then add the entries in /etc/fstab

```
UUID=f41e390f-835b-4223-a9bb-9b45984ddf8d /                       xfs     defaults        0 0
UUID=6748bd37-ac36-456f-ac1d-5f969079ae14 swap                    swap    defaults        0 0
UUID=66aece44-1c8a-4c55-9a17-e554b025e936 /u01                    xfs     defaults        0 0
```

then activate the swap and mount /u01

```bash
swapon -a
mount -a
```

Now is a good time to take a first snapshot of the VM.

```bash
virsh shutdown ora01
virsh snapshot-create-as --domain ora01 --name ora01-baseinstall
virsh start ora01
```

The next step is to take care of all pre-requirements for oracle grid and database: install packages, create users, kernel parameters, etc. We can either do it on the first VM and the clone this VM, or we can first clone the VM then use ansible to provision both servers at the same time. Since I am using ansible I will do the second approach

## clone the VM ora01 to ora02

First make sure ora01 is stopped, because ora02 will initially have the same IP (192.168.122.10). Also create the directory for the storage pool of ora02 and create the pool in virsh.

```bash
virsh shutdown ora01
virt-clone -o ora01 -n ora02  --file /data2/virtpool/ora02/ora02.qcow2 --file  /data2/virtpool/ora02/ora02-disk2.qcow2
```

start ora02, get into the VM and change the IP by editing the file /etc/sysconfig/network-scripts/ifcfg-eth0. Change also the HWADDR field. Change also the hostname with the hostnamectl command.

## configure both servers via ansible

I use ansible to configure the servers: oracle and grid users, linux groups, linux parameters, etc.. I have an ansible role called oracle-grid which I include a playbook. It is in my github project ansible, so the project must be cloned and a playbook must be created that includes the role oracle-grid

```bash
git clone https://github.com/saule1508/ansible.git
```

Edit the inventory, the file is called orarac.inventory

```
[orarac]
ora01 ansible_host=192.168.122.10 ansible_connection=ssh ansible_user=ansible ansible_become=true ansible_become_user=root
ora02 ansible_host=192.168.122.11 ansible_connection=ssh ansible_user=ansible ansible_become=true ansible_become_user=root
```

then I create a playbook, orarac.yml, that set a few variables then include the role.

```
---

- hosts: orarac
  vars:
    - oracle_linux_password: <PASSWORD>
    - oracle_grid_software_url: "/run/media/pierre/b651edce-0f5c-493e-8d05-2f10391f6b68/oa18/LINUX.X64_180000_grid_home.zip"
    - oracle_db_software_url: "/run/media/pierre/b651edce-0f5c-493e-8d05-2f10391f6b68/oa18/LINUX.X64_180000_db_home.zip"

  remote_user: ansible
  become: yes
  become_method: sudo
  roles:
    - oracle-grid
```

And now we can run the playbook with the inventory file referencing both servers.

```bash
ansible-playbook -i orarac.inventory orarac.yml | tee -a ora.log
```


## Networking

### Public network

The RAC requires a DNS to resolve the scan 3 ip addresses in a round-robin fashion. The easiest solution is to use dnsmasq on your host but since dnsmasq is already used by libvirt we can simply use that one. The steps involved are:

* put all the IP's in the /etc/hosts file of the host
* destroy and restart the libvirt default network

So in the /etc/hosts of my host I have put those IP's 

```
172.16.31.10  ora01 ora01.localnet
192.168.122.11  ora02 ora02.localnet
# VIP those IP's are reserved here and will be assigned by grid
192.168.122.12  ora01-vip ora01-vip.localnet
192.168.122.13  ora02-vip ora02-vip.localnet
# SCAN 
192.168.122.14  orarac-scan orarac-scan.localnet
192.168.122.15  orarac-scan orarac-scan.localnet
192.168.122.16  orarac-scan orarac-scan.localnet
```

The let's restart the default network<commit_msg>Update 2019-05-05-oracle-RAC-libvirt.md<commit_after>---
layout: post
title: Oracle RAC with libvirt on Centos 7
published: true
---

In this post I will go through the different steps to install Oracle grid and set-up a cluster. My host is a fedora and the two guests running oracle will be Centos 7.6 virtual machines. The two small difficulties when setting up a RAC using virtual servers are:

* setting up the networking (it requires a DNS)
* setting up the shared storage
<!--more-->

But before we need to have a Centos 7.6 VM (base install). Once the VM is ready we will clone it. The final set-up will be the following:

* ora01, public IP 192.168.122.10, private IP (for the interconnect): 10.0.0.1
* ora02, public IP 192.168.122.11, private IP (for the interconnect): 10.0.0.2

Scan IP's: 192.168.122.15, 192.168.122.16, 192.168.122.17 (round-robin DNS)

For the private IP's there will be a dedicated virtual network between ora01 and ora02.

For the shared storage, I will use a udev rule to make the disks naming and ownership permanent, the udev rule will use the serial id of the disk which can be assigned when attaching the disk (the disk is just a file on the host) via virt-manager. When attaching the disk to the VM's we will also check the flag "Shareable".

## creating the first guest VM

In order to create the first VM I will use the cloud init image provided by Centos. I made a post here on how to use cloud init: [Clout-init to create centos 7 guests on KVM](http://saule1508.github.io/libvirt-centos-cloud-image/). Basically, 
* create a file ora01.iso using cloud-localds, based on user-data and metadata (take care: on Fedora the module resolv_conf is not working)
* use virt-install to create the VM with 8GB and 4 cpu cores.

```bash
export VMPOOLDIR=/data2/virtpool/ora01

virt-install --name ora01 --memory 8192 --vcpus 4 --disk $VMPOOLDIR/ora01.qcow2,device=disk,bus=virtio --os-type generic --os-variant centos7.0 --virt-type kvm --network=default,model=virtio --cdrom $VMPOOLDIR/ora01.iso
```

where VMPOOLDIR is the directory I created as a storage pool.

```bash
virsh pool-create-as --name ora01 --type dir --target $VMPOOLDIR
```

At this stage I have a base guest VM, I need to add a /u01 file system and to add more swap space.

First let's create then allocate a new disk to the guest

```bash
virsh vol-create-as --pool ora01 --name ora01-disk2.qcow2 --capacity 120G --allocation 20G --format qcow2
virsh attach-disk --domain ora01 --source $VMPOOLDIR/ora01-disk2.qcow2 --target vdb --persistent --driver qemu --subdriver qcow2
```

Now let's get into the VM, create a volume group and the new file system. So inside the VM, as root:

```bash
pvcreate /dev/vdb
vgcreate vg01 /dev/vdb
lvcreate -n lv_swap -L16G vg01
mkswap /dev/vg01/lv_swap
lvcreate -n lv_u01 -l100%FREE vg01
mkfs -t xfs /dev/vg01/lv_u01
mkdir /u01
```

Use the command blkid to get the UUID of swap and lv_u01, then add the entries in /etc/fstab

```
UUID=f41e390f-835b-4223-a9bb-9b45984ddf8d /                       xfs     defaults        0 0
UUID=6748bd37-ac36-456f-ac1d-5f969079ae14 swap                    swap    defaults        0 0
UUID=66aece44-1c8a-4c55-9a17-e554b025e936 /u01                    xfs     defaults        0 0
```

then activate the swap and mount /u01

```bash
swapon -a
mount -a
```

Now is a good time to take a first snapshot of the VM.

```bash
virsh shutdown ora01
virsh snapshot-create-as --domain ora01 --name ora01-baseinstall
virsh start ora01
```

The next step is to take care of all pre-requirements for oracle grid and database: install packages, create users, kernel parameters, etc. We can either do it on the first VM and when done clone the VM, or we can first clone the VM then use ansible to provision both servers at the same time. Since I am using ansible I will do the second approach

## clone the VM ora01 to ora02

First make sure ora01 is stopped, because ora02 will initially have the same IP (192.168.122.10). Also create the directory for the storage pool of ora02 and create the pool in virsh.

```bash
virsh shutdown ora01
virt-clone -o ora01 -n ora02  --file /data2/virtpool/ora02/ora02.qcow2 --file  /data2/virtpool/ora02/ora02-disk2.qcow2
```

start ora02, get into the VM and change the IP by editing the file /etc/sysconfig/network-scripts/ifcfg-eth0. Change also the HWADDR field. Change also the hostname with the hostnamectl command.

## configure both servers via ansible

I use ansible to configure the servers: oracle and grid users, linux groups, linux parameters, etc.. I have an ansible role called oracle-grid which I will include in a playbook and then run the playbook against both servers. The role is in my github project ansible, so the project must be cloned and a playbook must be created that includes the role oracle-grid

```bash
git clone https://github.com/saule1508/ansible.git
```

Edit the inventory, the file is called orarac.inventory

```
[orarac]
ora01 ansible_host=192.168.122.10 ansible_connection=ssh ansible_user=ansible ansible_become=true ansible_become_user=root
ora02 ansible_host=192.168.122.11 ansible_connection=ssh ansible_user=ansible ansible_become=true ansible_become_user=root
```

then I create a playbook, orarac.yml, that set a few variables then include the role.

```
---

- hosts: orarac
  vars:
    - oracle_linux_password: <PASSWORD>
    - oracle_grid_software_url: "/run/media/pierre/b651edce-0f5c-493e-8d05-2f10391f6b68/oa18/LINUX.X64_180000_grid_home.zip"
    - oracle_db_software_url: "/run/media/pierre/b651edce-0f5c-493e-8d05-2f10391f6b68/oa18/LINUX.X64_180000_db_home.zip"

  remote_user: ansible
  become: yes
  become_method: sudo
  roles:
    - oracle-grid
```

And now we can run the playbook with the inventory file referencing both servers.

```bash
ansible-playbook -i orarac.inventory orarac.yml | tee -a ora.log
```

## Networking

### Public network

The RAC requires a DNS to resolve the scan 3 ip addresses in a round-robin fashion. The easiest solution is to use dnsmasq on your host but since dnsmasq is already used by libvirt we can simply use that one. The steps involved are:

* put all the IP's in the /etc/hosts file of the host
* destroy and restart the libvirt default network

So in the /etc/hosts of my host I have put those IP's 

```
172.16.31.10  ora01 ora01.localnet
192.168.122.11  ora02 ora02.localnet
# VIP those IP's are reserved here and will be assigned by grid
192.168.122.12  ora01-vip ora01-vip.localnet
192.168.122.13  ora02-vip ora02-vip.localnet
# SCAN 
192.168.122.14  orarac-scan orarac-scan.localnet
192.168.122.15  orarac-scan orarac-scan.localnet
192.168.122.16  orarac-scan orarac-scan.localnet
```

The let's restart the default network
{'fraction_non_alphanumeric': 0.0812411002023533, 'fraction_numerical': 0.06880011991306303, 'mean_word_length': 3.6317250954529676, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 34, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1545196', 'n_tokens_mistral': 5087, 'n_tokens_neox': 4396, 'n_words': 1806}