<commit_before># Troubleshooting

This page recaps the actions you can perform if you have any problems with your XOA.

:::warning
If you have issues with XO installed from GitHub (not XOA), [please go to the dedicated section first!](community.md).
:::

## Deploy error

> Auto deploy failed. - No SR specified and Pool default SR is null

It means you don't have a default SR set on the pool you are importing XOA on. To set a default SR, you must first find the SR UUID you want, with `xe sr-list`. When you have the UUID, you can set the default SR with: `xe pool-param-set uuid=<pool-uuid> default-SR=<sr-uuid>`. For the pool UUID, just press tab after `xe pool-param-set uuid=` and it will autofill your pool UUID. When this is done, re-enter the deploy script command and it will work!

## Unreachable after boot

XOA uses HVM mode. If your physical host doesn't support virtualization extensions, XOA won't work. To check if your XenServer supports hardware assisted virtualization (HVM), you can enter this command in your host: `grep --color vmx /proc/cpuinfo`. If you don't have any result, it means XOA won't work on this hardware.

## Set or recover XOA VM password

As no password is set for the xoa system user by default, you will need to set your own. This can be done via the XenStore data of the VM. The following is to be ran on your XCP-ng host:

```
xe vm-param-set uuid=<UUID> xenstore-data:vm-data/system-account-xoa-password=<password>
```

Where UUID is the uuid of your XOA VM.

Then you need to restart the VM.

## Recover web login password

If you have lost your password to log in to the XOA webpage, you can reset it. From the XOA CLI (for login/access info for the CLI, [see here](xoa.md#first-console-connection)), use the following command and insert the email/account you wish to recover:

`xo-server-recover-account <EMAIL>`

It will prompt you to set a new password. If you provide an email here that does not exist in XOA yet, it will create a new account using it, with admin permissions - you can use that new account to log in as well.

## Empty page after login

This happens when your antivirus or firewall is blocking the websocket protocol. This is what we use to communicate between `xo-server` and `xo-web` (see the [architecture page](architecture.md)).

The solution is to use **HTTPS**. When doing so, websockets will be encapsulated in the secured protocol, avoiding interception from your firewall or antivirus system.

## Migration issues

By default, XOA has a static max memory set to 16GiB. Sometimes you can have trouble migrating with this error message:

```
"Failed","Migrating VM 'XOA' from '<origin_hostname>' to '<destination_hostname>'
Internal error: Xenops_interface.Internal_error("Domain.Xenguest_failure(\"Error while waiting for suspend notification: xenguest: xc_domain_save: [1] Save failed (0 = Success)\")")

```

In this case, it means you need to reduce the static max memory field to a lower value, and try again.

## Boot issues

XOA is configured in HVM. It means you need hardware that supports HVM instructions (almost all hardware since 2011). If that's not the case, the symptom is this:

1. XOA VM starts for few seconds
2. Then it shuts down

Please check that you have enabled virtualization settings in your BIOS or upgrade your hardware.

## Logs

This section will explain how to check the XOA logs, and use them to detect issues.

### From the web interface

Go into Settings/Logs view.

### CLI

All XOA logs are stored in `/var/log/syslog` (on the XO Appliance).

To filter only what you need, you can use `journalctl`. Below is an example to filter only logs for `xo-server`:

```
$ journalctl -u xo-server -f -n 50
```

This will return the 50 last lines and tail the file. If you have an error message in your application, start this command and try to reproduce the issue. You'll see clearly what the problem is.

You can also filter for the updater program:

```
$ journalctl -u xoa-updater -f -n 50
```

## Configuration

XOA is a virtual appliance running Debian with Xen Orchestra installed. If you have any problems, the first thing to do is to use our check service by running the `xoa check` command in a terminal:

```
$ xoa check
✔ Node version
✔ Disk space for /var
✔ Disk space for /
✔ XOA version
✔ xo-server config syntax
✔ Appliance registration
✔ Internet connectivity
```

If you have something completely different than that, or error messages, lost packets etc., it means you have a problem.

### Network issues

You can see your current network configuration by running `ifconfig eth0`. If you have an external firewall, please check that you allow the XOA's IP.

You can modify the IP configuration with `xoa network static` (for a static IP address) or `xoa network dhcp` to use DHCP.

### Memory

Sometimes xo-server runs out of memory, this can be seen in the logs (`journalctl -u xo-server.service`):

```
<--- Last few GCs --->
48734864 ms: Mark-sweep 1359.7 (1422.6) -> 1359.7 (1438.6) MB, 1675.5 / 0.0 ms [allocation failure] [scavenge might not succeed].
48736444 ms: Mark-sweep 1359.7 (1438.6) -> 1359.7 (1438.6) MB, 1579.5 / 0.0 ms [allocation failure] [scavenge might not succeed].
48738329 ms: Mark-sweep 1359.7 (1438.6) -> 1368.7 (1422.6) MB, 1885.0 / 0.0 ms [last resort gc].
48740025 ms: Mark-sweep 1368.7 (1422.6) -> 1377.7 (1422.6) MB, 1695.0 / 0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x12ba820cfb51 <JS Object>
1: stringify(aka stringify) [native json.js:178] [pc=0x12d240955b57] (this=0x12ba82004381 <undefined>,E=0x162cbe055041 <an Object with map 0x8ecabcc95a1>,F=0x12ba82004381 <undefined>,S=0x12ba82004381 <undefined>)
2: arguments adaptor frame: 1->3
3: response [/usr/local/lib/node_modules/xo-server/node_modules/json-rpc-protocol/dist/format.js:~54] [pc=0x12d240cc1301] (this=0x1de505db7...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
```

In that case, you need to increase the memory allocated to the XOA VM (from 2GB to 4GB or 8GB). Note that simply increasing the RAM for the VM is not enough. You must also edit the service file (`/etc/systemd/system/xo-server.service`) to increase the memory allocated to the xo-server process itself.

:::tip
You should leave ~512MB for the debian OS itself. Meaning if your VM has 4096MB total RAM, you should use `3584` for the memory value below.

```diff
- ExecStart=/usr/local/bin/xo-server
+ ExecStart=/usr/local/bin/node --max-old-space-size=3584 /usr/local/bin/xo-server
```

:::

The last step is to refresh and restart the service:

```
$ systemctl daemon-reload
$ systemctl restart xo-server
```

### Behind a transparent proxy

If you're behind a transparent proxy, you'll probably have issues with the updater (SSL/TLS issues).

Run the following commands to allow the updater to work:

```
$ sudo -s
$ echo NODE_TLS_REJECT_UNAUTHORIZED=0 >> /etc/xo-appliance/env
$ npm config -g set strict-ssl=false
$ systemctl restart xoa-updater
```

Now try running an update again.

### Updating SSL self-signed certificate

If the provided certificate is expired, you may want to create a new one.

Connect to your appliance via SSH, then as root execute these commands:

```
$ cd /etc/ssl
$ cp server.crt server.crt.old
$ cp server.key server.key.old
$ openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt -nodes -days 360
$ systemctl restart xo-server.service
```

### Logs

The system logs are visible by using this command:

```
$ tail -f /var/log/syslog
```

You can read more about logs [in the dedicated logs chapter](troubleshooting.md#logs).

### Ghost tasks

If you have ghost tasks accumulating in your Xen Orchestra you can try the following actions in order:<commit_msg>fix(docs): filenames in how to update self-signed cert<commit_after># Troubleshooting

This page recaps the actions you can perform if you have any problems with your XOA.

:::warning
If you have issues with XO installed from GitHub (not XOA), [please go to the dedicated section first!](community.md).
:::

## Deploy error

> Auto deploy failed. - No SR specified and Pool default SR is null

It means you don't have a default SR set on the pool you are importing XOA on. To set a default SR, you must first find the SR UUID you want, with `xe sr-list`. When you have the UUID, you can set the default SR with: `xe pool-param-set uuid=<pool-uuid> default-SR=<sr-uuid>`. For the pool UUID, just press tab after `xe pool-param-set uuid=` and it will autofill your pool UUID. When this is done, re-enter the deploy script command and it will work!

## Unreachable after boot

XOA uses HVM mode. If your physical host doesn't support virtualization extensions, XOA won't work. To check if your XenServer supports hardware assisted virtualization (HVM), you can enter this command in your host: `grep --color vmx /proc/cpuinfo`. If you don't have any result, it means XOA won't work on this hardware.

## Set or recover XOA VM password

As no password is set for the xoa system user by default, you will need to set your own. This can be done via the XenStore data of the VM. The following is to be ran on your XCP-ng host:

```
xe vm-param-set uuid=<UUID> xenstore-data:vm-data/system-account-xoa-password=<password>
```

Where UUID is the uuid of your XOA VM.

Then you need to restart the VM.

## Recover web login password

If you have lost your password to log in to the XOA webpage, you can reset it. From the XOA CLI (for login/access info for the CLI, [see here](xoa.md#first-console-connection)), use the following command and insert the email/account you wish to recover:

`xo-server-recover-account <EMAIL>`

It will prompt you to set a new password. If you provide an email here that does not exist in XOA yet, it will create a new account using it, with admin permissions - you can use that new account to log in as well.

## Empty page after login

This happens when your antivirus or firewall is blocking the websocket protocol. This is what we use to communicate between `xo-server` and `xo-web` (see the [architecture page](architecture.md)).

The solution is to use **HTTPS**. When doing so, websockets will be encapsulated in the secured protocol, avoiding interception from your firewall or antivirus system.

## Migration issues

By default, XOA has a static max memory set to 16GiB. Sometimes you can have trouble migrating with this error message:

```
"Failed","Migrating VM 'XOA' from '<origin_hostname>' to '<destination_hostname>'
Internal error: Xenops_interface.Internal_error("Domain.Xenguest_failure(\"Error while waiting for suspend notification: xenguest: xc_domain_save: [1] Save failed (0 = Success)\")")

```

In this case, it means you need to reduce the static max memory field to a lower value, and try again.

## Boot issues

XOA is configured in HVM. It means you need hardware that supports HVM instructions (almost all hardware since 2011). If that's not the case, the symptom is this:

1. XOA VM starts for few seconds
2. Then it shuts down

Please check that you have enabled virtualization settings in your BIOS or upgrade your hardware.

## Logs

This section will explain how to check the XOA logs, and use them to detect issues.

### From the web interface

Go into Settings/Logs view.

### CLI

All XOA logs are stored in `/var/log/syslog` (on the XO Appliance).

To filter only what you need, you can use `journalctl`. Below is an example to filter only logs for `xo-server`:

```
$ journalctl -u xo-server -f -n 50
```

This will return the 50 last lines and tail the file. If you have an error message in your application, start this command and try to reproduce the issue. You'll see clearly what the problem is.

You can also filter for the updater program:

```
$ journalctl -u xoa-updater -f -n 50
```

## Configuration

XOA is a virtual appliance running Debian with Xen Orchestra installed. If you have any problems, the first thing to do is to use our check service by running the `xoa check` command in a terminal:

```
$ xoa check
✔ Node version
✔ Disk space for /var
✔ Disk space for /
✔ XOA version
✔ xo-server config syntax
✔ Appliance registration
✔ Internet connectivity
```

If you have something completely different than that, or error messages, lost packets etc., it means you have a problem.

### Network issues

You can see your current network configuration by running `ifconfig eth0`. If you have an external firewall, please check that you allow the XOA's IP.

You can modify the IP configuration with `xoa network static` (for a static IP address) or `xoa network dhcp` to use DHCP.

### Memory

Sometimes xo-server runs out of memory, this can be seen in the logs (`journalctl -u xo-server.service`):

```
<--- Last few GCs --->
48734864 ms: Mark-sweep 1359.7 (1422.6) -> 1359.7 (1438.6) MB, 1675.5 / 0.0 ms [allocation failure] [scavenge might not succeed].
48736444 ms: Mark-sweep 1359.7 (1438.6) -> 1359.7 (1438.6) MB, 1579.5 / 0.0 ms [allocation failure] [scavenge might not succeed].
48738329 ms: Mark-sweep 1359.7 (1438.6) -> 1368.7 (1422.6) MB, 1885.0 / 0.0 ms [last resort gc].
48740025 ms: Mark-sweep 1368.7 (1422.6) -> 1377.7 (1422.6) MB, 1695.0 / 0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x12ba820cfb51 <JS Object>
1: stringify(aka stringify) [native json.js:178] [pc=0x12d240955b57] (this=0x12ba82004381 <undefined>,E=0x162cbe055041 <an Object with map 0x8ecabcc95a1>,F=0x12ba82004381 <undefined>,S=0x12ba82004381 <undefined>)
2: arguments adaptor frame: 1->3
3: response [/usr/local/lib/node_modules/xo-server/node_modules/json-rpc-protocol/dist/format.js:~54] [pc=0x12d240cc1301] (this=0x1de505db7...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
```

In that case, you need to increase the memory allocated to the XOA VM (from 2GB to 4GB or 8GB). Note that simply increasing the RAM for the VM is not enough. You must also edit the service file (`/etc/systemd/system/xo-server.service`) to increase the memory allocated to the xo-server process itself.

:::tip
You should leave ~512MB for the debian OS itself. Meaning if your VM has 4096MB total RAM, you should use `3584` for the memory value below.

```diff
- ExecStart=/usr/local/bin/xo-server
+ ExecStart=/usr/local/bin/node --max-old-space-size=3584 /usr/local/bin/xo-server
```

:::

The last step is to refresh and restart the service:

```
$ systemctl daemon-reload
$ systemctl restart xo-server
```

### Behind a transparent proxy

If you're behind a transparent proxy, you'll probably have issues with the updater (SSL/TLS issues).

Run the following commands to allow the updater to work:

```
$ sudo -s
$ echo NODE_TLS_REJECT_UNAUTHORIZED=0 >> /etc/xo-appliance/env
$ npm config -g set strict-ssl=false
$ systemctl restart xoa-updater
```

Now try running an update again.

### Updating SSL self-signed certificate

If the provided certificate is expired, you may want to create a new one.

Connect to your appliance via SSH, then as root execute these commands:

```
$ cd /etc/ssl
$ cp cert.pem cert.pem-old
$ cp key.pem key.pem-old
$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -nodes -days 360
$ systemctl restart xo-server.service
```

### Logs

The system logs are visible by using this command:

```
$ tail -f /var/log/syslog
```

You can read more about logs [in the dedicated logs chapter](troubleshooting.md#logs).

### Ghost tasks

If you have ghost tasks accumulating in your Xen Orchestra you can try the following actions in order:
{'fraction_non_alphanumeric': 0.09283076531261938, 'fraction_numerical': 0.03565516363173309, 'mean_word_length': 4.114620644741127, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '202039', 'n_tokens_mistral': 5228, 'n_tokens_neox': 4723, 'n_words': 2286}