Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/v1.2/troubleshooting-hv/grub-menu.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/v1.2/troubleshooting-hv/vm-is-off.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
50 changes: 25 additions & 25 deletions versioned_docs/version-v1.6/troubleshooting/harvester.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
sidebar_position: 2
sidebar_label: Harvester
title: "Harvester"
sidebar_label: Hypervisor
title: "Hypervisor"
---

<head>
Expand All @@ -10,15 +10,15 @@ title: "Harvester"

## Fail to Deploy a Multi-node Cluster Due to Incorrect HTTP Proxy Setting

### ISO Installation Without a Harvester Configuration File
### ISO Installation Without a Hypervisor Configuration File

#### Configure HTTP Proxy During Harvester Installation
#### Configure HTTP Proxy During Hypervisor Installation

In some environments, you configure [http-proxy](../airgap.md#configure-an-http-proxy-during-installation) of [OS Environment](../install/harvester-configuration.md#osenvironment) during Harvester installation.
In some environments, you configure [http-proxy](../airgap.md#configure-an-http-proxy-during-installation) of [OS Environment](../install/harvester-configuration.md#osenvironment) during Hypervisor installation.

#### Configure HTTP Proxy After First Node is Ready

After the first node is installed successfully, you login into the `Harvester GUI` to configure [http-proxy](../airgap.md#configure-an-http-proxy-in-harvester-settings) of [Harvester System Settings](../install/harvester-configuration.md#system_settings).
After the first node is installed successfully, you login into the `Hypervisor GUI` to configure [http-proxy](../airgap.md#configure-an-http-proxy-in-harvester-settings) of [Hypervisor System Settings](../install/harvester-configuration.md#system_settings).

Then you continue to add more nodes to the cluster.

Expand All @@ -44,28 +44,28 @@ For example, your cluster assigns IPs from CIDR `172.26.50.128/27` to nodes via

After setting this, you can continue to add new nodes to the cluster.

For more details, please refer to [Harvester issue 3091](https://github.com/harvester/harvester/issues/3091).
For more details, please refer to [Hypervisor issue 3091](https://github.com/harvester/harvester/issues/3091).

### ISO Installation With a Harvester Configuration File
### ISO Installation With a Hypervisor Configuration File

When a Harvester configuration file is used in ISO installation, please configure proper `http-proxy` in [Harvester System Settings](../install/harvester-configuration.md#system_settings).
When a Hypervisor configuration file is used in ISO installation, please configure proper `http-proxy` in [Hypervisor System Settings](../install/harvester-configuration.md#system_settings).

### PXE Boot Installation

When [PXE Boot Installation](../install/pxe-boot-install.md) is adopted, please configure proper `http-proxy` in [OS Environment](../install/harvester-configuration.md#osenvironment) and [Harvester System Settings](../install/harvester-configuration.md#system_settings).
When [PXE Boot Installation](../install/pxe-boot-install.md) is adopted, please configure proper `http-proxy` in [OS Environment](../install/harvester-configuration.md#osenvironment) and [Hypervisor System Settings](../install/harvester-configuration.md#system_settings).

## Generate a Support Bundle

Users can generate a support bundle in the Harvester GUI with the following steps:
Users can generate a support bundle in the Hypervisor GUI with the following steps:

- Click the `Support` link at the bottom-left of Harvester Web UI.
![](/img/v1.2/troubleshooting/harvester-sb-support-link.png)
- Click the `Support` link at the bottom-left of Hypervisor Web UI.
![](/img/v1.2/troubleshooting-hv/harvester-sb-support-link.png)

- Click `Generate Support Bundle` button.
![](/img/v1.2/troubleshooting/harvester-sb-support-button.png)
![](/img/v1.2/troubleshooting-hv/harvester-sb-support-button.png)

- Enter a useful description for the support bundle and click `Create` to generate and download a support bundle.
![](/img/v1.2/troubleshooting/harvester-sb-support-modal.png)
![](/img/v1.2/troubleshooting-hv/harvester-sb-support-modal.png)

:::note

Expand All @@ -85,19 +85,19 @@ For information about collecting guest cluster logs and configuration files, see

### Manually Download and Retain a Support Bundle File

By default, a support bundle file is automatically generated, downloaded, and deleted after you click **Create** on the Harvester UI. However, you may want to retain a file for various reasons, including the following:
By default, a support bundle file is automatically generated, downloaded, and deleted after you click **Create** on the Hypervisor UI. However, you may want to retain a file for various reasons, including the following:

- You are unable to download the file because of network connectivity errors and other issues.

- You must use a previously generated file to troubleshoot issues (because generating a support bundle file takes time).

- You want to view information that only exists in a previously generated file.

Even if the file remains in the cluster, the Harvester UI does not provide a download link. Use the following workaround to generate, manually download, and retain a support bundle file:
Even if the file remains in the cluster, the Hypervisor UI does not provide a download link. Use the following workaround to generate, manually download, and retain a support bundle file:

#### Generate the File and Prevent Automatic Downloading

1. On the Harvester UI, click **Generate Support Bundle**.
1. On the Hypervisor UI, click **Generate Support Bundle**.

1. When the progress indicator reaches 20% to 80%, close the browser tab to prevent automatic downloading of the generated file.

Expand Down Expand Up @@ -195,7 +195,7 @@ You can delete the related resources using the following methods:
$ kubectl get supportbundle -A
No resources found
```
- Automatic: Harvester deletes the related resources based on how the following settings are configured:
- Automatic: Hypervisor deletes the related resources based on how the following settings are configured:

- [support-bundle-expiration](../advanced/index#support-bundle-expiration): Defines the time allowed for retaining a support bundle file

Expand All @@ -213,7 +213,7 @@ kubectl cp harvester-system/supportbundle-manager-bundle-dtl2k-69dcc69b59-w64vl:

### Manually Collect Data for Support Bundle

Harvester is unable to collect data and generate a support bundle when the node is inaccessible or not ready. The workaround is to run a script and compress the generated files.
Hypervisor is unable to collect data and generate a support bundle when the node is inaccessible or not ready. The workaround is to run a script and compress the generated files.

1. Prepare the environment.
```sh
Expand Down Expand Up @@ -319,7 +319,7 @@ Harvester is unable to collect data and generate a support bundle when the node

For more information, see [Issue 3383](https://github.com/harvester/harvester/issues/3383).

## Access Embedded Rancher and Longhorn Dashboards
<!-- ## Access Embedded Rancher and Longhorn Dashboards

_Available as of v1.1.0_

Expand All @@ -332,17 +332,17 @@ You can now access the embedded Rancher and Longhorn dashboards directly on the
We only support using the embedded Rancher and Longhorn dashboards for debugging and validation purposes.
For Rancher's multi-cluster and multi-tenant integration, please refer to the docs [here](../rancher/rancher-integration.md).

:::
::: -->

## I can't access Harvester after I changed SSL/TLS enabled protocols and ciphers
## I can't access Hypervisor after I changed SSL/TLS enabled protocols and ciphers

If you changed
[SSL/TLS enabled protocols and ciphers settings](../advanced/settings.md#ssl-parameters)
and you no longer have access to Harvester GUI and API,
and you no longer have access to Hypervisor GUI and API,
it's highly possible that NGINX Ingress Controller has stopped working due to the misconfigured SSL/TLS protocols and ciphers.
Follow these steps to reset the setting:

1. Following [FAQ](../faq.md) to SSH into Harvester node and switch to `root` user.
1. Following [FAQ](../faq.md) to SSH into Hypervisor node and switch to `root` user.
```
$ sudo -s
```
Expand Down
24 changes: 12 additions & 12 deletions versioned_docs/version-v1.6/troubleshooting/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ title: "Installation"

The following sections contain tips to troubleshoot or get assistance with failed installations.

## Logging into the Harvester Installer (a live OS)
## Logging into the Hypervisor Installer (a live OS)

Users can press the key combination `CTRL + ALT + F2` to switch to another TTY and log in with the following credentials:

Expand Down Expand Up @@ -65,7 +65,7 @@ token: 'ThisIsTheCorrectOne'
To ensure the change is persistent across reboots, update the `token` value of the OS configuration file `/oem/90_custom.yaml`:

```yaml
name: Harvester Configuration
name: Hypervisor Configuration
stages:
...
initramfs:
Expand Down Expand Up @@ -106,13 +106,13 @@ $ sudo yq eval .token /etc/rancher/rancherd/config.yaml

:::

## Check the status of Harvester components
## Check the status of Hypervisor components

Before checking the status of Harvester components, obtain a copy of the Harvester cluster's kubeconfig file following the [guide](../faq.md#how-can-i-access-the-kubeconfig-file-of-the-harvester-cluster).
Before checking the status of Hypervisor components, obtain a copy of the Hypervisor cluster's kubeconfig file following the [guide](../faq.md#how-can-i-access-the-kubeconfig-file-of-the-harvester-cluster).

After you obtain a copy of the kubeconfig file, run the following script against the cluster to check the readiness of each component.

- Harvester components script
- Hypervisor components script
```shell
#!/bin/bash

Expand Down Expand Up @@ -160,29 +160,29 @@ Please include the following information in a bug report when reporting a failed
- System information and logs.
- Available as of v1.0.2

Please follow the guide in [Logging into the Harvester Installer (a live OS)](#logging-into-the-harvester-installer-a-live-os) to log in. And run the command to generate a tarball that contains troubleshooting information:
Please follow the guide in [Logging into the Hypervisor Installer (a live OS)](#logging-into-the-harvester-installer-a-live-os) to log in. And run the command to generate a tarball that contains troubleshooting information:

```
supportconfig -k -c
```

The command output messages contain the generated tarball path. For example the path is `/var/loq/scc_aaa_220520_1021 804d65d-c9ba-4c54-b12d-859631f892c5.txz` in the following example:

![](/img/v1.2/troubleshooting/installation-support-config-example.png)
![](/img/v1.2/troubleshooting-hv/installation-support-config-example.png)

:::note

A failure PXE Boot installation automatically generates a tarball if the [`install.debug`](../install/harvester-configuration.md#installdebug) field is set to `true` in the Harvester configuration file.
A failure PXE Boot installation automatically generates a tarball if the [`install.debug`](../install/harvester-configuration.md#installdebug) field is set to `true` in the Hypervisor configuration file.

:::

## Harvester Console shows "Setting up Harvester" after day 0 Installation
## Hypervisor Console shows "Setting up Hypervisor" after day 0 Installation

### Issue Description

After a successful installation, the Harvester console persistently shows `Setting up Harvester`. While most UI and CLI operations remain unaffected, attempts to [start an upgrade](../upgrade/automatic.md#start-an-upgrade) are blocked.
After a successful installation, the Hypervisor console persistently shows `Setting up Hypervisor`. While most UI and CLI operations remain unaffected, attempts to [start an upgrade](../upgrade/automatic.md#start-an-upgrade) are blocked.

![](/img/v1.6/troubleshooting/setting-up-harvester-after-day-0.png)
![](/img/v1.2/troubleshooting-hv/setting-up-harvester-after-day-0.png)

The following information is displayed after you run the command `kubectl get managedchart -n fleet-local harvester -oyaml`:

Expand All @@ -205,7 +205,7 @@ status:

### Root Cause

The Harvester console runs the following command to determine if the status of the `harvester` ManagedChart (in the `fleet-local` namespace) is `Ready`.
The Hypervisor console runs the following command to determine if the status of the `harvester` ManagedChart (in the `fleet-local` namespace) is `Ready`.

```
cmd := exec.Command("/bin/sh", "-c", kubectl -n fleet-local get ManagedChart harvester -o jsonpath='{.status.conditions}' |
Expand Down
39 changes: 20 additions & 19 deletions versioned_docs/version-v1.6/troubleshooting/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,22 @@
sidebar_position: 4
sidebar_label: Monitoring
title: "Monitoring"
draft: true
---

<head>
<link rel="canonical" href="https://docs.harvesterhci.io/v1.6/troubleshooting/monitoring"/>
</head>

The following sections contain tips to troubleshoot Harvester Monitoring.
The following sections contain tips to troubleshoot Hypervisor Monitoring.

## Monitoring is unusable

When the Harvester Dashboard is not showing any monitoring metrics, it can be caused by the following reasons.
When the Hypervisor Dashboard is not showing any monitoring metrics, it can be caused by the following reasons.

### Monitoring is unusable due to Pod being stuck in `Terminating` status

Harvester Monitoring pods are deployed randomly on the cluster Nodes. When the Node hosting the pods accidentally goes down, the related pods may become stuck in the `Terminating` status rendering the Monitoring unusable from the WebUI.
Hypervisor Monitoring pods are deployed randomly on the cluster Nodes. When the Node hosting the pods accidentally goes down, the related pods may become stuck in the `Terminating` status rendering the Monitoring unusable from the WebUI.

```shell
$ kubectl get pods -n cattle-monitoring-system
Expand Down Expand Up @@ -73,13 +74,13 @@ rancher-monitoring-grafana-d9c56d79b-cp86w 3/3 Running 0

## Expand PV/Volume Size

`Harvester` integrates `Longhorn` as the default storage provider.
`Hypervisor` integrates `Longhorn` as the default storage provider.

Harvester `Monitoring` uses `Persistent Volume (PV)` to store running data. When a cluster has been running for a certain time, the `Persistent Volume` may need to expand its size.
Hypervisor `Monitoring` uses `Persistent Volume (PV)` to store running data. When a cluster has been running for a certain time, the `Persistent Volume` may need to expand its size.

Based on the `Longhorn` `Volume` expansion guide, `Harvester` illustrates how to [expand the volume size](https://longhorn.io/docs/1.3.2/volumes-and-nodes/expansion/).
Based on the `Longhorn` `Volume` expansion guide, `Hypervisor` illustrates how to [expand the volume size](https://longhorn.io/docs/1.3.2/volumes-and-nodes/expansion/).

### View Volume
<!-- ### View Volume

#### From Embedded Longhorn WebUI

Expand Down Expand Up @@ -185,15 +186,15 @@ The `Volume` is attached to the new POD.

![](/img/v1.2/troubleshooting/6-after-scale-up.png)

To now, the `Volume` is expanded to the new size and the POD is using it smoothly.
To now, the `Volume` is expanded to the new size and the POD is using it smoothly. -->

## Fail to Enable `rancher-monitoring` Addon

You may encounter this when you install the Harvester v1.3.0 or higher version cluster with the minimal 250 GB disk per [hardware requirements](../install/requirements.md#hardware-requirements).
You may encounter this when you install the Hypervisor v1.3.0 or higher version cluster with the minimal 250 GB disk per [hardware requirements](../install/requirements.md#hardware-requirements).

### Reproduce Steps

1. Install the Harvester v1.3.0 cluster.
1. Install the Hypervisor v1.3.0 cluster.

1. Enable the `rancher-monitoring` [addon](../advanced/addons.md), you will observe:

Expand Down Expand Up @@ -275,13 +276,13 @@ You may encounter this when you install the Harvester v1.3.0 or higher version c
alertmanager-rancher-monitoring-alertmanager-db-alertmanager-rancher-monitoring-alertmanager-0 Bound pvc-cea6316e-f74f-4771-870b-49edb5442819 5Gi RWO harvester-longhorn 16m
```

1. On the **Addons** screen of the Harvester UI, select **⋮** (menu icon) and then select **Edit YAML**.
1. On the **Addons** screen of the Hypervisor UI, select **⋮** (menu icon) and then select **Edit YAML**.

![](/img/v1.3/troubleshooting/edit-rancher-monitoring.png)
![](/img/v1.3/troubleshooting-hv/edit-rancher-monitoring.png)

1. As indicated below, change the two occurrences of the number `50` to `30` under prometheusSpec, and then save. The `prometheus` feature will use a 30GiB disk to store data.

![](/img/v1.3/troubleshooting/edit-rancher-monitoring-yaml.png)
![](/img/v1.3/troubleshooting-hv/edit-rancher-monitoring-yaml.png)

Alternatively, you can use `kubectl` to edit the object.

Expand Down Expand Up @@ -382,7 +383,7 @@ fleet-local mcc-rancher-logging-crd 1/1
fleet-local mcc-rancher-monitoring-crd 0/1 Modified(1) [Cluster fleet-local/local]; clusterrole.rbac.authorization.k8s.io rancher-monitoring-crd-manager missing; clusterrolebinding.rbac.authorization.k8s.io rancher-monitoring-crd-manager missing; configmap.v1 cattle-monitoring-system/rancher-monitoring-crd-manifest missing; serviceaccount.v1 cattle-monitoring-system/rancher-monitoring-crd-manager missing
```

When the issue exists and you [start an upgrade](../upgrade/automatic.md#start-an-upgrade), Harvester may return the following error message: `admission webhook "validator.harvesterhci.io" denied the request: managed chart rancher-monitoring-crd is not ready, please wait for it to be ready`.
When the issue exists and you [start an upgrade](../upgrade/automatic.md#start-an-upgrade), Hypervisor may return the following error message: `admission webhook "validator.harvesterhci.io" denied the request: managed chart rancher-monitoring-crd is not ready, please wait for it to be ready`.

Also, when you search for the objects marked as `missing`, you will find that they exist in the cluster.

Expand Down Expand Up @@ -638,7 +639,7 @@ The `prometheus` CRD object includes the `storage-network.settings.harvesterhci.
creationTimestamp: "2025-05-20T06:40:25Z"
```

The Harvester pod logs ('harvester-system/harvester' deployment) indicate that the attempt to change the `storage-network` setting was blocked.
The Hypervisor pod logs ('harvester-system/harvester' deployment) indicate that the attempt to change the `storage-network` setting was blocked.

```
...
Expand All @@ -652,21 +653,21 @@ The Harvester pod logs ('harvester-system/harvester' deployment) indicate that t

### Root Cause

When you make changes to the [storage-network](../advanced/storagenetwork.md#storage-network-setting) setting, the Harvester controller waits for the attached volumes to be detached before applying the changes. In addition, the controller automatically terminates the pods related to Prometheus, Alertmanager, and Grafana because those pods use volumes to store data.
When you make changes to the [storage-network](../advanced/storagenetwork.md#storage-network-setting) setting, the Hypervisor controller waits for the attached volumes to be detached before applying the changes. In addition, the controller automatically terminates the pods related to Prometheus, Alertmanager, and Grafana because those pods use volumes to store data.

This process usually takes a short time to complete, but can be disrupted when the following occur:

- Attached volumes prevent the Harvester controller from applying the changes to the setting.
- Attached volumes prevent the Hypervisor controller from applying the changes to the setting.
- A user or the `monitoring-operator` attempts to enable the `rancher-monitoring` add-on.
- The Harvester controller terminates the pods.
- The Hypervisor controller terminates the pods.

### Workaround

1. Disable the `rancher-monitoring` add-on.

1. Check if the [storage-network](../advanced/storagenetwork.md#storage-network-setting) setting is enabled or disabled.

1. Check for error indicators in the Harvester pod logs. If volumes are still attached, stop the related virtual machines until no errors appear after the `storage network change` message.
1. Check for error indicators in the Hypervisor pod logs. If volumes are still attached, stop the related virtual machines until no errors appear after the `storage network change` message.

1. Enable the `rancher-monitoring` add-on.

Expand Down
Loading
Loading