.. _Exporting Data via S3: Exporting Data via S3 --------------------- .. include:: /includes/about-s3-clusters-part1.inc .. include:: /includes/exporting-data-via-s3-part1.inc .. _S3 Storage Infrastructure Overview: S3 Storage Infrastructure Overview ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part2.inc .. image:: /images/stor_image42.png :align: center :class: align-center .. include:: /includes/exporting-data-via-s3-part3.inc .. _Planning the S3 Cluster: Planning the S3 Cluster ~~~~~~~~~~~~~~~~~~~~~~~ Before creating an S3 cluster, do the following: #. Define which nodes of the storage cluster will run the S3 storage access point services. It is recommended to have all nodes available in |product_name| run these services. #. Configure the network so that the following is achieved: - All components of the S3 cluster communicate with each other via the S3 private network. All nodes of an S3 cluster must be connected to the S3 private network. |product_name| internal network can be used for this purpose. - The nodes running S3 gateways must have access to the public network. - The public network for the S3 gateways must be balanced by an external DNS load balancer. For more details on network configuration, refer to the *Installation Guide*. #. All components of the S3 cluster should run on multiple nodes for high-availability. Name server and object server components in the S3 cluster are automatically balanced and migrated between S3 nodes. S3 gateways are not automatically migrated; their high availability is based on DNS records. You need to maintain the DNS records manually when adding or removing S3 gateways. .. _Sample S3 Storage: Sample S3 Storage ~~~~~~~~~~~~~~~~~ This section shows a sample object storage deployed on top of a storage cluster of five nodes that run various services. The final setup is shown on the figure below. .. image:: /images/stor_image43.png :align: center :class: align-center .. _Creating the S3 Cluster: Creating the S3 Cluster ~~~~~~~~~~~~~~~~~~~~~~~ To set up object storage services on a cluster node, do the following: #. On the **INFRASTRUCTURE** > **Networks** screen, make sure that the **OSTOR private** and **S3 public** traffic types are added to your networks. #. In the left menu, click **STORAGE SERVICES** > **S3**. #. Select one or more nodes and click **Create S3 cluster** in the right menu. To create a highly available S3 cluster, select at least three nodes. It is also recommended to enable HA for the management node prior to creating the S3 cluster. See :ref:`Enabling High Availability` for more details. #. Make sure the correct network interface is selected in the corresponding drop-down list. .. include:: /includes/connecting-abc-via-abgw-part8_1.inc .. only:: ac .. image:: /images/stor_image45_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image45_vz.png :align: center :class: align-center Click **Proceed**. #. In **Tier**, select the storage tier that will be used for the object storage. #. In **Failure domain**, choose a placement policy for replicas. #. In **Data redundancy**, select the redundancy mode that the object storage will use. .. only:: ac .. image:: /images/stor_image46_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image46_vz.png :align: center :class: align-center You can change the redundancy mode later on the **S3** > **OVERVIEW** > **Settings** panel. Click **Proceed**. #. Specify the external (publicly resolvable) DNS name for the S3 endpoint that will be used by the end users to access the object storage. For example, ``s3.example.com``. Click **Proceed**. .. include:: /includes/connecting-abc-via-abgw-part9_2_2.inc #. From the drop-down list, select an S3 endpoint protocol: HTTP, HTTPS or both. .. only:: ac .. image:: /images/stor_image46_1_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image46_1_vz.png :align: center :class: align-center It is recommended to use only HTTPS for production deployments. If you have selected HTTPS, do one of the following: - Check **Generate self-signed certificate** to get a self-signed certificate for HTTPS evaluation purposes. Take note of the following: - S3 geo-replication requires a certificate from a trusted authority. It does not work with self-signed certificates. - To access the data in the S3 cluster via a browser, add the self-signed certificate to browser's exceptions. - Acquire a key and a trusted wildcard SSL certificate for endpoint's bottom-level domain. For example, the endpoint ``s3.storage.example.com`` would need a wildcard certificate for ``*.s3.storage.example.com`` with the subject alternative name ``s3.storage.example.com``. Upload the certificate, and, depending on the certificate type, do one of the following: - in case the certificate is contained in a PKCS#12 file, specify the passphrase; - upload the SSL key. You can change the redundancy mode later on the **S3** > **OVERVIEW** > **Protocol settings** panel. Click **Proceed**. #. If required, click **Configure Acronis Notary** and specify **Notary DNS name** and **Notary user key**. #. Click **Done** to create an S3 cluster. After the S3 cluster is created, open the **S3 Overview** screen to view cluster status, hostname, used disk capacity, the number of users, I/O activity, and the state of S3 services. To check if the S3 cluster is successfully deployed and can be accessed by users, visit \https:// or \http:// in your browser. You should receive the following XML response: :: AccessDenied To start using the S3 storage, you will also need to create at least one S3 user. .. _Managing S3 Users: Managing S3 Users ~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part12.inc .. _Adding S3 users: Adding S3 Users *************** To add an S3 user, do the following: #. On the **STORAGE SERVICES** > **S3** > **Users** screen, click **Add user**. #. Specify a valid email address as login for the user and click **ADD**. .. only:: ac .. image:: /images/stor_image48_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image48_vz.png :align: center :class: align-center .. _Managing S3 Access Key Pairs: Managing S3 Access Key Pairs **************************** Each S3 user has one or two key pairs (access key and secret key) for accessing the S3 cloud. You can think of the access key as login and the secret key as password. (For more information about S3 key pairs, refer to the `Amazon documentation `__.) The access keys are generated and stored locally in the storage cluster on S3 name servers. Each user can have up to two key pairs. It is recommended to periodically revoke old and generate new access key pairs. To view, add, or revoke the S3 access key pairs for an S3 user, do the following: #. Select a user in the list and click **Keys**. .. only:: ac .. image:: /images/stor_image49_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image49_vz.png :align: center :class: align-center #. The existing keys will be shown on the **Keys** panel. - To revoke a key, click **Revoke**. - To add a new key, click **Generate access key**. To access a bucket, a user will need the following information: - admin panel IP address, - DNS name of the S3 cluster specified during configuration, - S3 access key ID, - S3 secret access key, - SSL certificate if the HTTPS protocol was chosen during configuration. The certificate file can be found in the ``/etc/nginx/ssl/`` directory on any node hosting the S3 gateway service. To automatically log in to S3 with user credentials using the generated keys, select a user and click **Browse**. To **Browse** using an SSL certificate, make sure it is valid or, in case of a self-signed one, add it to browser's exceptions. .. _Managing S3 Buckets: Managing S3 Buckets ~~~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part13.inc In the current version of |product_name|, you can enable and disable Acronis Notary for object storage buckets and monitor the space used by them on the **STORAGE SERVICES** > **S3** > **Buckets** screen. You cannot create and manage object storage buckets from |product_name| admin panel. However, you can do it via the |product_name| user panel or by using a third-party application. For example, the applications listed below allow you to perform the following actions: - CyberDuck: create and manage buckets and their contents. - MountainDuck: mount object storage as a disk drive and manage buckets and their contents. - Backup Exec: store backups in the object storage. .. _Listing S3 Bucket Contents: Listing S3 Bucket Contents ************************** .. include:: /includes/exporting-data-via-s3-part14.inc .. _Managing Acronis Notary in S3 Buckets: Managing Acronis Notary in S3 Buckets ************************************* |product_name| offers integration with the Acronis Notary service to leverage blockchain notarization and ensure the immutability of data saved in object storage clusters. To use Acronis Notary in user buckets, you need to set it up in the S3 cluster and enable it for said buckets. To set up Acronis Notary, do the following: #. Get the DNS name and the user key for the notary service from your sales contact. #. On the **STORAGE SERVICES** > **S3** screen, click **Notary settings**. #. On the **Notary Settings** screen, specify the DNS name and user key in the respective fields and click **Done**. .. only:: ac .. image:: /images/stor_image66_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image66_vz.png :align: center :class: align-center To enable or disable blockchain notarization for a bucket, select a bucket on the **STORAGE SERVICES** > **S3** > **Buckets** screen and click **Enable Notary** or **Disable Notary**, respectively. Notarization is disabled for new buckets by default. Once you enable notarization for a bucket, certificates are created automatically only for the newly uploaded files. The previously uploaded files are left unnotarized. Once a file was notarized, it will remain notarized even if you disable notarization later. .. _Best Practices for Using S3 in |product_name|: Best Practices for Using S3 in |product_name| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section offers recommendations on how to best use the S3 feature of |product_name|. .. _S3 Bucket and Key Naming Policies: S3 Bucket and Key Naming Policies ********************************* .. include:: /includes/exporting-data-via-s3-part4.inc .. _Improving Performance of PUT Operations: Improving Performance of PUT Operations *************************************** .. include:: /includes/exporting-data-via-s3-part5.inc .. _Replicating S3 Data Between Datacenters: Replicating S3 Data Between Datacenters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |product_name| can store replicas of S3 cluster data and keep them up-to-date in multiple geographically distributed datacenters with S3 clusters based on |product_name|. Geo-replication reduces the response time for local S3 users accessing the data in a remote S3 cluster or remote S3 users accessing the data in a local S3 cluster as they do not need to have an Internet connection. Geo-replication schedules the update of the replicas as soon as any data is modified. Geo-replication performance depends on the speed of Internet connection, the redundancy mode, and cluster performance. If you have multiple datacenters with enough free space, it is recommended to set up geo-replication between S3 clusters residing in these datacenters. .. important:: Each cluster must have its own SSL certificate signed by a global certificate authority. To set up geo-replication between S3 clusters, exchange tokens between datacenters as follows: #. In the admin panel of a remote datacenter, open the **STORAGE SERVICES** > **S3** > **GEO-REPLICATION** screen. .. only:: ac .. image:: /images/stor_image66_1_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image66_1_vz.png :align: center :class: align-center #. In the section of the home S3 cluster, click **TOKEN** and, on the **Get token** panel, copy the token. #. In the admin panel of the local datacenter, open the **STORAGE SERVICES** > **S3** > **GEO-REPLICATION** screen and click **ADD DATACENTER**. .. only:: ac .. image:: /images/stor_image66_2_ac.png :align: center :class: align-center .. only:: vz .. image:: /images/stor_image66_2_vz.png :align: center :class: align-center #. Enter the copied token and click **Done**. #. Configure the remote S3 cluster the same way. .. _Monitoring S3 Access Points: Monitoring S3 Access Points ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The S3 monitoring screen enables you to inspect the availability of each S3 component as well as the performance of NS and OS services (which are highly available). If you see that some of the NS or OS services are offline, it means that the S3 access point does not function properly, and you should contact support consult the CLI guide for low-level troubleshooting. S3 gateways are not highly available, but DNS load balancing should be enough to avoid downtime if the gateway fails. The performance charts represent the number of operations that the OS/NS services are performing. .. _Releasing Nodes from S3 Clusters: Releasing Nodes from S3 Clusters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before releasing a node, make sure that the cluster has enough nodes running name and object servers as well as gateways left. .. warning:: When the last node in the S3 cluster is removed, the cluster is destroyed, and all the data is deleted. To release a node from an S3 cluster, do the following: #. On the **STORAGE SERVICES** > **S3 Nodes** screen, check the box of the node to release. #. Click **Release**. .. _Supported Amazon S3 Features: Supported Amazon S3 Features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part6.inc .. _Supported Amazon S3 REST Operations: Supported Amazon S3 REST Operations *********************************** .. include:: /includes/exporting-data-via-s3-part7.inc .. _Supported Amazon Request Headers: Supported Amazon Request Headers ******************************** .. include:: /includes/exporting-data-via-s3-part8.inc .. _Supported Amazon Response Headers: Supported Amazon Response Headers ********************************* .. include:: /includes/exporting-data-via-s3-part9.inc .. _Supported Amazon Error Response Headers: Supported Amazon Error Response Headers *************************************** .. include:: /includes/exporting-data-via-s3-part10.inc .. _Supported Authentication Scheme and Methods: Supported Authentication Scheme and Methods ******************************************* .. include:: /includes/exporting-data-via-s3-part11.inc