2.3. Planning Node Hardware Configurations¶

2.3.3.1. Storage Cluster Composition Recommendations¶

Designing an efficient storage cluster means finding a compromise between performance and cost that suits your purposes. When planning, keep in mind that a cluster with many nodes and few disks per node offers higher performance while a cluster with the minimum number of nodes (3) and a lot of disks per node is cheaper. See the following table for more details.

Take note of the following:

1. These considerations only apply if failure domain is host.
2. The speed of rebuilding in the replication mode does not depend on the number of nodes in the cluster.
3. Acronis Cyber Infrastructure supports hundreds of disks per node. If you plan to use more than 36 disks per node, contact sales engineers who will help you design a more efficient cluster.

2.3.3.2. General Hardware Recommendations¶

• At least five nodes are required for a production environment. This is to ensure that the cluster can survive failure of two nodes without data loss.
• One of the strongest features of Acronis Cyber Infrastructure is scalability. The bigger the cluster, the better Acronis Cyber Infrastructure performs. It is recommended to create production clusters from at least ten nodes for improved resilience, performance, and fault tolerance in production scenarios.
• Even though a cluster can be created on top of varied hardware, using nodes with similar hardware in each node will yield better cluster performance, capacity, and overall balance.
• Any cluster infrastructure must be tested extensively before it is deployed to production. Such common points of failure as SSD drives and network adapter bonds must always be thoroughly verified.
• It is not recommended for production to run Acronis Cyber Infrastructure on top of SAN/NAS hardware that has its own redundancy mechanisms. Doing so may negatively affect performance and data availability.
• To achieve best performance, keep at least 20% of cluster capacity free.
• During disaster recovery, Acronis Cyber Infrastructure may need additional disk space for replication. Make sure to reserve at least as much space as any single storage node has.
• It is recommended to have the same CPU models on each node to avoid VM live migration issues. For more details, see the Administrator’s Command Line Guide.
• If you plan to use Backup Gateway to store backups in the cloud, make sure the local storage cluster has plenty of logical space for staging (keeping backups locally before sending them to the cloud). For example, if you perform backups daily, provide enough space for at least 1.5 days’ worth of backups. For more details, see the Administrator’s Guide.
• It is recommended to use UEFI instead of BIOS if supported by hardware. In particular, if you use NVMe drives.

2.3.3.3. Storage Hardware Recommendations¶

• It is possible to use disks of different size in the same cluster. However, keep in mind that, given the same IOPS, smaller disks will offer higher performance per terabyte of data compared to bigger disks. It is recommended to group disks with the same IOPS per terabyte in the same tier.
• Using the recommended SSD models may help you avoid loss of data. Not all SSD drives can withstand enterprise workloads and may break down in the first months of operation, resulting in TCO spikes.
  • SSD memory cells can withstand a limited number of rewrites. An SSD drive should be viewed as a consumable that you will need to replace after a certain time. Consumer-grade SSD drives can withstand a very low number of rewrites (so low, in fact, that these numbers are not shown in their technical specifications). SSD drives intended for storage clusters must offer at least 1 DWPD endurance (10 DWPD is recommended). The higher the endurance, the less often SSDs will need to be replaced, improving TCO.
  • Many consumer-grade SSD drives can ignore disk flushes and falsely report to operating systems that data was written while it, in fact, was not. Examples of such drives include OCZ Vertex 3, Intel 520, Intel X25-E, and Intel X-25-M G2. These drives are known to be unsafe in terms of data commits, they should not be used with databases, and they may easily corrupt the file system in case of a power failure. For these reasons, use enterprise-grade SSD drives that obey the flush rules (for more information, see http://www.postgresql.org/docs/current/static/wal-reliability.html). Enterprise-grade SSD drives that operate correctly usually have the power loss protection property in their technical specification. Some of the market names for this technology are Enhanced Power Loss Data Protection (Intel), Cache Power Protection (Samsung), Power-Failure Support (Kingston), Complete Power Fail Protection (OCZ).
  • It is highly recommended to check the data flushing capabilities of your disks as explained in Checking Disk Data Flushing Capabilities.
  • Consumer-grade SSD drives usually have unstable performance and are not suited to withstand sustainable enterprise workloads. For this reason, pay attention to sustainable load tests when choosing SSDs.
  • Performance of SSD disks may depend on their size. Lower-capacity drives (100 to 400 GB) may perform much slower (sometimes up to ten times slower) than higher-capacity ones (1.9 to 3.8 TB). Consult drive performance and endurance specifications before purchasing hardware.
• Using NVMe or SAS SSDs for write caching improves random I/O performance and is highly recommended for all workloads with heavy random access (e.g., iSCSI volumes). In turn, SATA disks are best suited for SSD-only configurations but not write caching.
• Using shingled magnetic recording (SMR) HDDs is strongly not recommended, even for backup scenarios. Such disks have unpredictable latency that may lead to unexpected temporary service outages and sudden performance degradations.
• Running metadata services on SSDs improves cluster performance. To also minimize CAPEX, the same SSDs can be used for write caching.
• If capacity is the main goal and you need to store non-frequently accessed data, choose SATA disks over SAS ones. If performance is the main goal, choose NVMe or SAS disks over SATA ones.
• The more disks per node the lower the CAPEX. As an example, a cluster created from ten nodes with two disks in each will be less expensive than a cluster created from twenty nodes with one disk in each.
• Using SATA HDDs with one SSD for caching is more cost effective than using only SAS HDDs without such an SSD.
• Create hardware or software RAID1 volumes for system disks using RAID or HBA controllers, respectively, to ensure its high performance and availability.
• Use HBA controllers as they are less expensive and easier to manage than RAID controllers.
• Disable all RAID controller caches for SSD drives. Modern SSDs have good performance that can be reduced by a RAID controller’s write and read cache. It is recommend to disable caching for SSD drives and leave it enabled only for HDD drives.
• If you use RAID controllers, do not create RAID volumes from HDDs intended for storage. Each storage HDD needs to be recognized by Acronis Cyber Infrastructure as a separate device.
• If you use RAID controllers with caching, equip them with backup battery units (BBUs) to protect against cache loss during power outages.
• Disk block size (e.g., 512b or 4K) is not important and has no effect on performance.

2.3.3.4. Network Hardware Recommendations¶

• Use separate networks (and, ideally albeit optionally, separate network adapters) for internal and public traffic. Doing so will prevent public traffic from affecting cluster I/O performance and also prevent possible denial-of-service attacks from the outside.
• Network latency dramatically reduces cluster performance. Use quality network equipment with low latency links. Do not use consumer-grade network switches.
• Do not use desktop network adapters like Intel EXPI9301CTBLK or Realtek 8129 as they are not designed for heavy load and may not support full-duplex links. Also use non-blocking Ethernet switches.
• To avoid intrusions, Acronis Cyber Infrastructure should be on a dedicated internal network inaccessible from outside.
• Use one 1 Gbit/s link per each two HDDs on the node (rounded up). For one or two HDDs on a node, two bonded network interfaces are still recommended for high network availability. The reason for this recommendation is that 1 Gbit/s Ethernet networks can deliver 110-120 MB/s of throughput, which is close to sequential I/O performance of a single disk. Since several disks on a server can deliver higher throughput than a single 1 Gbit/s Ethernet link, networking may become a bottleneck.
• For maximum sequential I/O performance, use one 1Gbit/s link per each hard drive, or one 10Gbit/s link per node. Even though I/O operations are most often random in real-life scenarios, sequential I/O is important in backup scenarios.
• For maximum overall performance, use one 10 Gbit/s link per node (or two bonded for high network availability).
• It is not recommended to configure 1 Gbit/s network adapters to use non-default MTUs (e.g., 9000-byte jumbo frames). Such settings require additional configuration of switches and often lead to human error. 10+ Gbit/s network adapters, on the other hand, need to be configured to use jumbo frames to achieve full performance.
• The currently supported Fibre Channel host bus adapters (HBAs) are QLogic QLE2562-CK and QLogic ISP2532.
• It is recommended to use Mellanox ConnectX-4 and ConnectX-5 InfiniBand adapters. Mellanox ConnectX-2 and ConnectX-3 cards are not supported.
• Adapters using the BNX2X driver, such as Broadcom Limited BCM57840 NetXtreme II 10/20-Gigabit Ethernet / HPE FlexFabric 10Gb 2-port 536FLB Adapter, are not recommended. They limit MTU to 3616, which affects the cluster performance.

2.3.6.1. HDD Only¶

This basic configuration requires a dedicated disk for each metadata server.

2.3.6.2. HDD + System SSD (No Cache)¶

This configuration is good for creating capacity-oriented clusters.

2.3.6.3. HDD + SSD¶

This configuration is good for creating performance-oriented clusters.

2.3.6.4. SSD Only¶

This configuration does not require SSDs for cache.

When choosing hardware for this configuration, have in mind the following:

• Each Acronis Cyber Infrastructure client will be able to obtain up to about 40K sustainable IOPS (read + write) from the cluster.
• If you use the erasure coding redundancy scheme, each erasure coding file, e.g., a single VM HDD disk, will get up to 2K sustainable IOPS. That is, a user working inside a VM will have up to 2K sustainable IOPS per virtual HDD at their disposal. Multiple VMs on a node can utilize more IOPS, up to the client’s limit.
• In this configuration, network latency defines more than half of overall performance, so make sure that the network latency is minimal. One recommendation is to have one 10Gbps switch between any two nodes in the cluster.

2.3.6.5. HDD + SSD (No Cache), 2 Tiers¶

In this configuration example, tier 1 is for HDDs without cache and tier 2 is for SSDs. Tier 1 can store cold data (e.g., backups), tier 2 can store hot data (e.g., high-performance virtual machines).

2.3.6.6. HDD + SSD, 3 Tiers¶

In this configuration example, tier 1 is for HDDs without cache, tier 2 is for HDDs with cache, and tier 3 is for SSDs. Tier 1 can store cold data (e.g., backups), tier 2 can store regular virtual machines, and tier 3 can store high-performance virtual machines.