Adding Hosts¶

Additional hosts can be added at any time to provide more capacity for guest VMs. For requirements and instructions, see “Adding a Host”.

Scheduled Maintenance and Maintenance Mode for Hosts¶

You can place a host into maintenance mode. When maintenance mode is activated, the host becomes unavailable to receive new guest VMs, and the guest VMs already running on the host are seamlessly migrated to another host not in maintenance mode. This migration uses live migration technology and does not interrupt the execution of the guest.

Disabling and Enabling Zones, Pods, and Clusters¶

You can enable or disable a zone, pod, or cluster without permanently removing it from the cloud. This is useful for maintenance or when there are problems that make a portion of the cloud infrastructure unreliable. No new allocations will be made to a disabled zone, pod, or cluster until its state is returned to Enabled. When a zone, pod, or cluster is first added to the cloud, it is Disabled by default.

To disable and enable a zone, pod, or cluster:

1. Log in to the CloudStack UI as administrator
2. In the left navigation bar, click Infrastructure.
3. In Zones, click View More.
4. If you are disabling or enabling a zone, find the name of the zone in the list, and click the Enable/Disable button.
5. If you are disabling or enabling a pod or cluster, click the name of the zone that contains the pod or cluster.
6. Click the Compute tab.
7. In the Pods or Clusters node of the diagram, click View All.
8. Click the pod or cluster name in the list.
9. Click the Enable/Disable button.

Removing Hosts¶

Hosts can be removed from the cloud as needed. The procedure to remove a host depends on the hypervisor type.

Re-Installing Hosts¶

You can re-install a host after placing it in maintenance mode and then removing it. If a host is down and cannot be placed in maintenance mode, it should still be removed before the re-install.

Maintaining Hypervisors on Hosts¶

When running hypervisor software on hosts, be sure all the hotfixes provided by the hypervisor vendor are applied. Track the release of hypervisor patches through your hypervisor vendor’s support channel, and apply patches as soon as possible after they are released. CloudStack will not track or notify you of required hypervisor patches. It is essential that your hosts are completely up to date with the provided hypervisor patches. The hypervisor vendor is likely to refuse to support any system that is not up to date with patches.

(XenServer) For more information, see Highly Recommended Hotfixes for XenServer in the CloudStack Knowledge Base.

Changing Host Password¶

The password for a XenServer Node, KVM Node, or vSphere Node may be changed in the database. Note that all Nodes in a Cluster must have the same password.

To change a Node’s password:

1. Identify all hosts in the cluster.

2. Change the password on all hosts in the cluster. Now the password for the host and the password known to CloudStack will not match. Operations on the cluster will fail until the two passwords match.

3. if the password in the database is encrypted, it is (likely) necessary to encrypt the new password using the database key before adding it to the database.

   java -classpath /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar \
   org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI \
   encrypt.sh input="newrootpassword" \
   password="databasekey" \
   verbose=false
   

4. Get the list of host IDs for the host in the cluster where you are changing the password. You will need to access the database to determine these host IDs. For each hostname “h” (or vSphere cluster) that you are changing the password for, execute:

   mysql> SELECT id FROM cloud.host WHERE name like '%h%';
   

5. This should return a single ID. Record the set of such IDs for these hosts. Now retrieve the host_details row id for the host

   mysql> SELECT * FROM cloud.host_details WHERE name='password' AND host_id={previous step ID};
   

6. Update the passwords for the host in the database. In this example, we change the passwords for hosts with host IDs 5 and 12 and host_details IDs 8 and 22 to “password”.

   mysql> UPDATE cloud.host_details SET value='password' WHERE id=8 OR id=22;
   

Over-Provisioning and Service Offering Limits¶

(Supported for XenServer, KVM, and VMware)

CPU and memory (RAM) over-provisioning factors can be set for each cluster to change the number of VMs that can run on each host in the cluster. This helps optimize the use of resources. By increasing the over-provisioning ratio, more resource capacity will be used. If the ratio is set to 1, no over-provisioning is done.

The administrator can also set global default over-provisioning ratios in the cpu.overprovisioning.factor and mem.overprovisioning.factor global configuration variables. The default value of these variables is 1: over-provisioning is turned off by default.

Over-provisioning ratios are dynamically substituted in CloudStack’s capacity calculations. For example:

Capacity = 2 GB Over-provisioning factor = 2 Capacity after over-provisioning = 4 GB

With this configuration, suppose you deploy 3 VMs of 1 GB each:

Used = 3 GB Free = 1 GB

The administrator can specify a memory over-provisioning ratio, and can specify both CPU and memory over-provisioning ratios on a per-cluster basis.

In any given cloud, the optimum number of VMs for each host is affected by such things as the hypervisor, storage, and hardware configuration. These may be different for each cluster in the same cloud. A single global over-provisioning setting can not provide the best utilization for all the different clusters in the cloud. It has to be set for the lowest common denominator. The per-cluster setting provides a finer granularity for better utilization of resources, no matter where the CloudStack placement algorithm decides to place a VM.

The overprovisioning settings can be used along with dedicated resources (assigning a specific cluster to an account) to effectively offer different levels of service to different accounts. For example, an account paying for a more expensive level of service could be assigned to a dedicated cluster with an over-provisioning ratio of 1, and a lower-paying account to a cluster with a ratio of 2.

When a new host is added to a cluster, CloudStack will assume the host has the capability to perform the CPU and RAM over-provisioning which is configured for that cluster. It is up to the administrator to be sure the host is actually suitable for the level of over-provisioning which has been set.

VLAN Provisioning¶

CloudStack automatically creates and destroys interfaces bridged to VLANs on the hosts. In general the administrator does not need to manage this process.

CloudStack manages VLANs differently based on hypervisor type. For XenServer or KVM, the VLANs are created on only the hosts where they will be used and then they are destroyed when all guests that require them have been terminated or moved to another host.

For vSphere the VLANs are provisioned on all hosts in the cluster even if there is no guest running on a particular Host that requires the VLAN. This allows the administrator to perform live migration and other functions in vCenter without having to create the VLAN on the destination Host. Additionally, the VLANs are not removed from the Hosts when they are no longer needed.

You can use the same VLANs on different physical networks provided that each physical network has its own underlying layer-2 infrastructure, such as switches. For example, you can specify VLAN range 500 to 1000 while deploying physical networks A and B in an Advanced zone setup. This capability allows you to set up an additional layer-2 physical infrastructure on a different physical NIC and use the same set of VLANs if you run out of VLANs. Another advantage is that you can use the same set of IPs for different customers, each one with their own routers and the guest networks on different physical NICs.

Out-of-band Management¶

CloudStack provides Root admins the ability to configure and use supported out-of-band management interface (e.g. IPMI, iLO, DRAC, etc.) on a physical host to manage host power operations such as on, off, reset etc. By default, IPMI 2.0 baseboard controller are supported out of the box with IPMITOOL out-of-band management driver in CloudStack that uses ipmitool for performing IPMI 2.0 management operations.

Following are some global settings that control various aspects of this feature.

A change in outofbandmanagement.sync.poolsize settings requires restarting of management server(s) as the thread pool and a background (power state) sync thread are configured during load time when CloudStack management server starts. Rest of the global settings can be changed without requiring restarting of management server(s).

The outofbandmanagement.sync.poolsize is the maximum number of ipmitool background power state scanners that can run at a time. Based on the maximum number of hosts you’ve, you can increase/decrease the value depending on how much stress your management server host can endure. It will take atmost number of total out-of-band-management enabled hosts in a round * outofbandmanagement.action.timeout / outofbandmanagement.sync.poolsize seconds to complete a background power-state sync scan in a single round.

In order to use this feature, the Root admin needs to first configure out-of-band management for a host using either the UI or the configureOutOfBandManagement API. Next, the Root admin needs to enable it. The feature can be enabled or disabled across a zone or a cluster or a host,

Once out-of-band management is configured and enabled for a host (and provided not disabled at zone or cluster level), Root admins would be able to issue power management actions such as on, off, reset, cycle, soft and status.

If a host is in maintenance mode, Root admins are still allowed to perform power management actions but in the UI a warning is displayed.

Server Address Usage¶

Historically, when multiple management servers are used a tcp-LB is used on port 8250 (default) of the management servers and the VIP/LB-IP is used as the host setting to be used by various CloudStack agents such as the KVM, CPVM, SSVM agents, who connect to the host on port 8250. However, starting CloudStack 4.11+ the host setting can accept comma separated list of management server IPs to which new CloudStack hosts/agents will get a shuffled list of the same to which they can cycle reconnections in a round-robin way.

Securing Process¶

Agents while making connections/reconnections to management server will only validate server certificate and be able to present client certificate (issued to them) when cloud.jks is accessible to them. On older hosts that are setup prior to this feature the keystore won’t be available, however, they can still connect to management server(s) if ca.plugin.root.auth.strictness is set to false. Management server(s) will check and setup their own cloud.jks keystore on startup, this keystore will be used for connecting to peer management server(s).

When a new host is being setup, such as adding a KVM host or starting a systemvm host, the CA framework kicks in and uses ssh to execute keystore-setup to generate a new keystore file cloud.jks.new, save a random passphrase of the keystore in the agent’s properties file and a CSR cloud.csr file. The CSR is then used to issue certificate for that agent/host and ssh is used to execute keystore-cert-import to import the issued certificate along with the CA certificate(s), the keystore is that renamed as cloud.jks replacing an previous keystore in-use. During this process, keys and certificates files are also stored in cloud.key, cloud.crt, cloud.ca.crt in the agent’s configuration directory.

When hosts are added out-of-band, for example a KVM host that is setup first outside of CloudStack and added to a cluster, the keystore file will not be available however the keystore and security could be setup by using keystore utility scripts manually. The keystore-setup can be ran first to generate a keystore and a CSR, then CloudStack CA can be used to issue certificate by providing the CSR to the issueCertificate API, and finally issued certificate and CA certificate(s) can be imported to the keystore using keystore-cert-import script.

Following lists the usage of these scripts, when using these script use full paths, use the final keystore filename as cloud.jks, and the certificate/key content need to be encoded and provided such that newlines are replace with ^ and space are replaced with ~:

keystore-setup <properties file> <keystore file> <passphrase> <validity> <csr file>

keystore-cert-import <properties file> <keystore file> <mode: ssh|agent> <cert file> <cert content> <ca-cert file> <ca-cert content> <private-key file> <private key content:optional>

Starting 4.11.1, a KVM host is considered secured when it has its keystore and certificates setup for both the agent and libvirtd process. A secured host will only allow and initiate TLS enabled live VM migration. This requires libvirtd to listen on default port 16514, and the port to be allowed in the firewall rules. Certificate renewal (using the provisionCertificate API) will restart both the libvirtd process and agent after deploying new certificates.