4.4. Managing Virtual Machines

Each virtual machine (VM) is an independent system with an independent set of virtual hardware. Its main features are the following:

  • A virtual machine resembles and works like a regular computer. It has its own virtual hardware. Software applications can run in virtual machines without any modifications or adjustment.
  • Virtual machine configuration can be changed easily, e.g., by adding new virtual disks or memory.
  • Although virtual machines share physical hardware resources, they are fully isolated from each other (file system, processes, sysctl variables) and the compute node.
  • A virtual machine can run any supported guest operating system.

The following table lists the current virtual machine configuration limits:

Table 4.4.1 Virtual machine hardware
Resource Limit
RAM 1 TiB
CPU 48 logical CPUs
Storage 15 volumes, 512 TiB each
Network 15 NICs

A logical CPU is a core (thread) in a multicore (multithreading) processor.

4.4.1. Supported Guest Operating Systems

The following guest operating systems have been tested and are supported in virtual machines:

Table 4.4.1.1 Windows guest operating systems
Operating System Edition Architecture
Windows Server 2019 Essentials, Standard, Datacenter x64
Windows Server 2016 Essentials, Standard, Datacenter x64
Windows Server 2012 R2 Essentials, Standard, Datacenter x64
Windows Server 2012 Standard, Datacenter x64
Windows Server 2008 R2 Standard, Datacenter x64
Windows Server 2008 Standard, Datacenter x64
Windows 10 Home, Professional, Enterprise, Enterprise 2016 LTSB x64
Windows 8.1 Home, Professional, Enterprise x64
Windows 7 Home, Professional, Enterprise x64
Table 4.4.1.2 Linux guest operating systems
Operating System Architecture
CentOS 8.x x64
CentOS 7.x x64
CentOS 6.x x64
RHEL 8.x x64
RHEL 7.x x64
Debian 9.x x64
Ubuntu 20.04.x x64
Ubuntu 18.04.x x64
Ubuntu 16.04.x x64

4.4.2. Creating Virtual Machines

Before you proceed to creating VMs, check that you have these:

  • A guest OS source (see Managing Images):

    • a distribution ISO image of a guest OS to install in the VM, or

    • a template that is a boot volume in the QCOW2 format, or

    • a boot volume

      Note

      To obtain a boot volume, create a volume as described in Managing Compute Volumes, attach it to a VM, install an operating system in it, then delete the VM.

  • A storage policy for volumes (see Managing Storage Policies)

  • A flavor (see Managing Flavors)

Note

Virtual machines are created with the host CPU model by default. Having compute nodes with different CPUs may lead to live migration issues. To avoid them, you can manually set CPU model for all new VMs as described in Setting Virtual Machines CPU Model.

To create a VM, do the following:

  1. On the COMPUTE > Virtual machines > VIRTUAL MACHINES tab, click Create virtual machine. A window will open where you will need to specify VM parameters.

    ../_images/stor_image70_ac.png
  2. Specify a name for the new VM.

  3. In Deploy from, choose Volume if you have a boot volume or want to create one. Otherwise, choose Image.

  4. Depending on your choice, click the pencil icon in the Volumes or Image section and do one of the following:

    • In the Images window, select the ISO image or template and click Done.
    ../_images/stor_image71_ac.png
    • In the Volumes window, do one of the following:
      • If you have prepared a volume with an installed guest OS, click Attach, find and select the volume, and click Done.
    ../_images/stor_image72_ac.png ../_images/stor_image73_ac.png
  5. Optionally, in the Volumes window, click Add or Attach to create or attach any other volumes you need. To select a volume as bootable, place it first in the list by clicking the up arrow button next to it.

  6. After you select an image or a volume, the Placement drop-down list is displayed. Placements are created by the administrator to group nodes or VMs sharing a distinctive feature, like a special license. Select the placement corresponding to the VM characteristics. For more information, see Managing Placements.

  7. In the Flavor window, choose a flavor and click Done.

    ../_images/stor_image74_ac.png
  8. In the network window, click Add, select a virtual network interface and click Add. It will appear in the Network interfaces list.

    ../_images/stor_image75_ac.png

    You can edit additional parameters of newly added network interfaces, like IP and MAC addresses and spoofing protection. To do this, click interface’s ellipsis icon, then Edit, and set parameters in the Edit network interface window.

    ../_images/stor_image75_1_ac.png

    You will not be able to edit these parameters later. Instead, you will be able to delete the old network interface and replace it with a new one.

    ../_images/stor_image75_2_ac.png

    Click Done.

  9. (Optional) If you are deploying the VM from a template or boot volume (not an ISO image), you can specify the following:

    • An SSH key to be injected into the VM. To do it, select an SSH key in the Select an SSH key window, and click Done.

      ../_images/stor_image76_ac.png

      Note

      To be able to connect to the VM via SSH, make sure the VM template or boot volume has cloud-init and OpenSSH installed (see Preparing Templates).

    • User data to customize the VM after launch. You can specify user data in one of two formats: cloud-config or shell script. To do it, write a script in the Customization script field or browse a file on your local server to load the script from.

      ../_images/stor_image77_ac.png

      Note

      For the guest OS to be customizable, make sure the VM template or boot volume has cloud-init installed (see Preparing Templates).

      To inject a script in a Windows VM, refer to the Cloudbase-Init documentation. For example, you can set a new password for the account using the following script:

      #ps1
      net user <username> <new_password>
      
  10. Back in the Create virtual machine window, click Deploy to create and boot the VM.

  11. If you are deploying the VM from an ISO image (not a boot volume template or a volume with a pre-installed guest OS), select the VM, click Console, and install the guest OS using the built-in VNC console.

  12. (Optional) If you are deploying the VM from a prepared template with an injected SSH key, you can connect to it via SSH using the username and the VM IP address:

    • For Linux templates, enter the username that is default for the cloud image OS (for example, for a CentOS cloud image, the default login is centos).
    • For Windows templates, enter the username that you specified during Cloudbase-Init installation.

    For example:

    # ssh myuser@10.10.10.10
    

4.4.3. Virtual Machine Actions Overview

After you create a virtual machine, you can manage it using the actions available for its current state. To see the full list of available actions, click the ellipsis button next to a VM or on top of its panel. Actions include:

  • Run powers up a VM.

  • Console connects to running VMs via the built-in VNC console. In the console browser window, you can send a key combination to a VM, take a screenshot of the console window, and download the console log.

  • Reboot soft-reboots a running VM.

  • Shut down gracefully shuts down a running VM.

  • Hard reboot cuts off and restores power, then starts a VM.

  • Power off forcibly cuts off power from a VM.

  • Shelve unbinds a stopped VM from the node it is hosted on and releases its reserved resources such as CPU and RAM. A shelved VM remains bootable and retains its configuration, including the IP addresses.

    Virtual machines in other states can be shelved by clicking Shut down or Power off and selecting the checkbox Shelve virtual machine in the confirmation window.

  • Unshelve spawns a shelved VM on a node with enough resources to host it.

  • Suspend saves the current VM state to a file.

    This may prove useful, for example, if you need to restart the host but do not want to quit the applications currently running in the VM or restart its guest OS.

  • Resume restores a VM from suspended state.

  • Download console log downloads the console log. Make sure logging is enabled inside the VM, otherwise the log will be empty (for more information, see Enabling Logging inside Virtual Machines).

    Examining console logs may be useful in troubleshooting failed virtual machines.

  • Reset state resets the VM stuck in a failed or transitional state to its last stable state: active, shut down or shelved.

  • Delete removes a VM from the compute cluster.

  • Migrate moves a VM to another node in the compute cluster (for more information, see Migrating Virtual Machines).

4.4.4. Enabling Logging inside Virtual Machines

VM’s console log will contain log messages only if the TTY1 and TTYS0 logging levels are enabled inside the VM. For example, you can enable them as follows in Linux VMs:

  1. Add the line GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0" to the file /etc/default/grub.

  2. Depending on the boot loader, run either

    # grub-mkconfig -o /boot/grub/grub.cfg
    

    or

    # grub2-mkconfig -o /boot/grub2/grub.cfg
    
  3. Reboot the VM.

In Windows VMs, you can enable Emergency Management Services (EMS) console redirection for this purpose. Do the following:

  1. Start Windows PowerShell with administrator privileges.

  2. In the PowerShell console, set the COM port and baud rate for EMS console redirection. As Windows VMs have only the COM1 port with the transmission rate of 9600 bps, run:

    bcdedit /emssettings EMSPORT:1
    
  3. Enable EMS for the current boot entry:

    bcdedit /ems on
    

You may also enable driver status logging to see the list of loaded drivers. This can be useful for troubleshooting a faulty driver or long boot process. You can do this as follows:

  1. Start System Configuration with administrator privileges.
  2. In the System Configuration windows, open the Boot tab, select the checkboxes OS boot information and Make all boot settings permanent.
  3. Confirm the changes and restart the system.

4.4.5. Migrating Virtual Machines

VM migration helps facilitate cluster upgrades and workload balancing between compute nodes. Acronis Cyber Infrastructure allows you to perform two types of migration:

  • Cold migration for stopped and suspended virtual machines
  • Hot migration for running virtual machines (allows you to avoid VM downtime)

For both migration types, a virtual machine is migrated between compute nodes using shared storage, so no block device migration takes place.

Hot migration consists of the following steps:

  1. All VM memory is copied to the destination node while the virtual machine keeps running on the source node. If a VM memory page changes, it is copied again.
  2. When only a few memory pages are left to copy, the VM is stopped on the source node, the remaining pages are transferred, and the VM is restarted on the destination node.

Large virtual machines with write-intensive workloads write to memory faster than memory changes can be transferred to the destination node, thus preventing migration from converging. For such VMs, the auto-converge mechanism is used. When a lack of convergence is detected during live migration, VM’s vCPU execution speed is throttled down, which also slows down writing to VM memory. Initially, virtual machine’s vCPU is throttled by 20% and then by 10% during each iteration. This process continues until writing to VM memory slows down enough for migration to complete or the VM vCPU is throttled by 99%.

Note

Virtual machines are created with the host CPU model by default. Having compute nodes with different CPUs may lead to live migration issues. To avoid them, you can manually set CPU model for all new VMs as described in Setting Virtual Machines CPU Model.

To migrate a VM, do the following:

  1. On the COMPUTE > Virtual machines > VIRTUAL MACHINES tab, click a VM to migrate, click the ellipsis button and choose Migrate.

    ../_images/stor_image104_ac.png
  2. In the new window, specify the destination node:

    • Auto. Automatically select the optimal destination among cluster nodes based on available CPU and RAM resources.
    • Select the destination node manually from the drop-down list.
    ../_images/stor_image105_ac.png
  3. By default, running VMs are migrated live. You can change the migration mode to offline by ticking the Cold migration checkbox. A VM will be stopped and restarted on the destination node after migration.

  4. Click Migrate to reserve resources on the destination node and start migration.

The admin panel will show the migration progress.

4.4.6. Reconfiguring and Monitoring Virtual Machines

To monitor virtual machine’s CPU, storage, and network usage, select the VM and open the Monitoring tab.

The default time interval for the charts is 12 hours. To zoom into a particular time interval, select the internal with the mouse; to reset zoom, double click any chart.

The following performance charts are available:

CPU / RAM
CPU and RAM usage by the VM.
Network
Incoming and outgoing network traffic.
Storage read/write
Amount of data read and written by the VM.
Read/write latency
Read and write latency. Hovering the mouse cursor over a point on the chart, you can also see the average and maximum latency for that moment as well as the 95 and 99 percentiles.

To reconfigure a VM, select it and, on the Overview tab, click the pencil icon next to a parameter you need to change. You cannot do the following:

  • Change, detach, or delete the boot volume
  • Manage non-boot volumes except attaching and detaching
  • Modify previously added network interfaces
  • Attach and detach network interfaces to and from shelved VMs
  • Change the flavor for running and shelved VMs

4.4.7. Configuring Virtual Machine High Availability

High availability keeps virtual machines operational if the node they are located on fails due to kernel crash, power outage and such or becomes unreachable over the network. Graceful shutdown is not considered a failure event.

Important

The compute cluster can survive the failure of only one node.

In the event of failure, the system will attempt to evacuate affected VMs automatically, that is, migrate them offline with auto-scheduling to other healthy compute nodes in the following order:

  • VMs with the “Active” status are evacuated first and automatically started.
  • VMs with the “Shut down” status are evacuated next and remain stopped.
  • All other VMs are ignored and left on the failed node.

If something blocks the evacuation, for example, destination compute nodes lack resources to host the affected VMs, these VMs remain on the failed node and receive the “Error” status. You can evacuate them manually after solving the issue (providing sufficient resources, joining new nodes to the cluster, etc.). To do this, click the ellipsis button next to such a VM or open its panel and click Evacuate.

../_images/stor_image155_ac.png

When the failed node becomes available again, it is fenced from scheduling new VMs on it and can be returned to operation manually. To do it, click the ellipsis button next to the fenced node or open its panel and then click Return to operation.

../_images/stor_image154_ac.png

By default, high availability for virtual machines is enabled automatically after creating the compute cluster. If required, you can disable it manually as follows:

  1. Click the VM for which you wish to disable HA.
  2. On the VM panel, click the pencil icon next to the High availability parameter.
  3. In the High availability window, disable HA for the VM and click Save.
../_images/stor_image156_ac.png

Virtual machines with disabled HA will not be evacuated to healthy nodes in case of failover.

4.4.8. Managing Guest Tools

This section explains how to install and uninstall the guest tools. This functionality is required for Running Commands in Virtual Machines without Network Connectivity, as well as for creating consistent snapshots of a running VM’s disks (refer to Managing Volume Snapshots).

4.4.8.1. Installing Guest Tools

Installation of the guest tools inside a virtual machine includes steps performed by users with different roles: a system administrator and a VM user.

As a system administrator, you need to do the following:

  1. Upload the guest tools ISO files located in the /usr/share/vz-guest-tools/ directory on any compute node to a network share or FTP server:
    • for a Windows guest, upload vz-guest-tools-win.iso
    • for a Linux guest, upload vz-guest-tools-lin.iso
  2. Provide access to the uploaded ISO file to a VM user.

As a VM user, log in to the virtual machine and do the following:

  • Inside a Windows VM:

    1. Download the Windows guest tools ISO image provided by your system administrator.
    2. Mount the image inside the VM.
      • On Windows 8 or Windows Server 2012 or newer, you can natively mount an ISO image. To do this, right-click the guest tools ISO image and select Mount.
      • On Windows 7 and Windows Server 2008, you need a third-party application to mount ISO images.
    3. Go to the mounted optical drive in Explorer and install the guest tools by running setup.exe.
    4. After the installation is complete, restart the VM.
  • Inside a Linux VM:

    1. Download the Linux guest tools ISO image provided by your system administrator.

    2. Create a mount point for the optical drive with the guest tools image and run the installer:

      # mkdir /mnt/cdrom
      # mount <path_to_guest_tools_iso> /mnt/cdrom
      # bash /mnt/cdrom/install
      

Note

Guest tools rely on the QEMU guest agent that is installed alongside the tools. The agent service must be running for the tools to work.

4.4.8.2. Uninstalling Guest Tools

If you find out that the guest tools are incompatible with some software inside a virtual machine, you can uninstall them as follows:

  • Inside a Windows VM:

    1. Remove the QEMU device drivers from the device manager.

      Important

      Do not remove the VirtIO/SCSI hard disk driver and NetKVM network driver. Without the former, the VM will not boot; without the latter, the VM will lose network connectivity.

    2. Uninstall the QEMU guest agent and guest tools from the list of installed applications.

    3. Stop and delete Guest Tools Monitor:

      > sc stop VzGuestToolsMonitor
      > sc delete VzGuestToolsMonitor
      
    4. Unregister Guest Tools Monitor from Event Log:

      > reg delete HKLM\SYSTEM\CurrentControlSet\services\eventlog\Application\\
      VzGuestToolsMonitor
      
    5. Delete the autorun registry key for RebootNotifier:

      > reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run /v \
      VzRebootNotifier
      
    6. Delete the C:\Program Files\Qemu-ga\ directory.

      If VzGuestToolsMonitor.exe is locked, close all the Event Viewer windows. If it remains locked, restart the eventlog service:

      > sc stop eventlog
      > sc start eventlog
      

    After removing the guest tools, restart the virtual machine.

  • Inside a Linux VM:

    1. Remove the packages:

      1. On RPM-based systems (CentOS and other):

        # yum remove dkms-vzvirtio_balloon prl_nettool qemu-guest-agent-vz vz-guest-udev
        
      2. On DEB-based systems (Debian and Ubuntu):

        # apt-get remove vzvirtio-balloon-dkms prl-nettool qemu-guest-agent-vz vz-guest-udev
        

        If any of the packages listed above are not installed on your system, the command will fail. In this case, exclude these packages from the command and run it again.

    2. Remove the files:

      # rm -f /usr/bin/prl_backup /usr/share/qemu-ga/VERSION /usr/bin/install-tools \
      /etc/udev/rules.d/90-guest_iso.rules /usr/local/bin/fstrim-static /etc/cron.weekly/fstrim
      
    3. Reload the udev rules:

      # udevadm control --reload
      

    After removing guest tools, restart the virtual machine.