.. _Exporting Data via S3: Exporting Data via S3 --------------------- .. include:: /includes/about-s3-clusters-part1.inc .. include:: /includes/exporting-data-via-s3-part1.inc .. _S3 Storage Infrastructure Overview: S3 Storage Infrastructure Overview ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part2.inc .. image:: /images/stor_image42.png :align: center :class: align-center .. include:: /includes/exporting-data-via-s3-part3.inc .. _Planning the S3 Cluster: Planning the S3 Cluster ~~~~~~~~~~~~~~~~~~~~~~~ Before creating an S3 cluster, do the following: #. Define which nodes of the |product_name| cluster will run the S3 storage access point services. It is recommended to have all nodes available in |product_name| run these services. #. Configure the network so that the following is achieved: - All components of the S3 cluster communicate with each other via the S3 private network. All nodes of an S3 cluster must be connected to the S3 private network. |product_name| internal network can be used for this purpose. - The nodes running S3 gateways must have access to the public network. - The public network for the S3 gateways must be balanced by an external DNS load balancer. For more details on network configuration, refer to the `Installation Guide `__. #. All components of the S3 cluster should run on multiple nodes for high-availability. Name server and object server components in the S3 cluster are automatically balanced and migrated between S3 nodes. S3 gateways are not automatically migrated; their high availability is based on DNS records. You should maintain the DNS records manually when adding or removing the S3 gateways. .. _Sample S3 Storage: Sample S3 Storage ~~~~~~~~~~~~~~~~~ This section shows a sample object storage deployed on top of a storage cluster of five nodes that run various services. The final setup is shown on the figure below. .. image:: /images/stor_image43.png :align: center :class: align-center .. _Creating the S3 Cluster: Creating the S3 Cluster ~~~~~~~~~~~~~~~~~~~~~~~ To set up object storage services on a cluster node, do the following: #. Make sure that S3 private network is configured on each node that will run object storage services. #. On the **SERVICES** > **Nodes** screen, check the box of each cluster node where object storage services will run. .. image:: /images/stor_image44.png :align: center :class: align-center #. Click **Create S3 cluster**. #. Make sure a network interface with an **Objest Storage private** role is selected in the drop-down list. The corresponding interfaces with S3 public roles will be selected automatically. .. Note:: If necessary, click the cogwheel icon and, on the **Network Configuration** screen, configure S3 roles. .. image:: /images/stor_image45.png :align: center :class: align-center #. Click **Proceed**. #. In **Tier**, select the storage tier that will be used for the object storage. For information about storage tiers, consult the `Installation Guide `__. #. In **Failure domain**, choose a placement policy for replicas. For more details, see the `Installation Guide `__. #. In **Data redundancy**, select the redundancy mode that the object storage will use. For more details, see the `Installation Guide `__. .. image:: /images/stor_image46.png :align: center :class: align-center .. note:: You can later change the redundancy mode on the **S3** > **Settings** panel. #. Click **Proceed**. #. Specify the external (publicly resolvable) DNS name for the S3 endpoint that will be used by the end users to access the object storage. For example, ``mys3storage.example.com``. Click **Proceed**. .. important:: Configure your DNS server according to the example suggested in the management panel. #. From the drop-down list, select an S3 endpoint protocol: HTTP, HTTPS or both. .. image:: /images/stor_image46_1.png :align: center :class: align-center .. note :: It is recommended to use only HTTPS for production deployments. If you have selected HTTPS, do one of the following: - Check **Generate self-signed certificate** to get a self-signed certificate for HTTPS evaluation purposes. .. note:: #. S3 geo-replication requires a certificate from a trusted authority. It does not work with self-signed certificates. #. To access the data in the S3 cluster via a browser, add the self-signed certificate to browser's exceptions. - Acquire a key and a trusted wildcard SSL certificate for endpoint's bottom-level domain. For example, the endpoint ``s3.storage.example.com`` would need a wildcard certificate for ``*.s3.storage.example.com`` with the subject alternative name ``s3.storage.example.com``. Upload the certificate, and, depending on the certificate type, do one of the following: - in case the certificate is contained in a PKCS#12 file, specify the passphrase; - upload the SSL key. #. If required, click **Configure Acronis Notary** and specify **Notary DNS name** and **Notary user key**. For more information on Acronis Notary, see **Managing Acronis Notary in S3 Buckets**. #. Click **Done** to create an S3 cluster. After the cluster is created, on the **S3 Overview** screen, you can view cluster status, hostname, used disk capacity, the number of users, I/O activity, and the state of S3 services. .. image:: /images/stor_image67.png :align: center :class: align-center To check if the S3 cluster is successfully deployed and can be accessed by users, visit https:// or http:// in your browser. You should receive the following XML response: :: AccessDenied To start using the S3 storage, you will also need to create at least one S3 user. .. _Managing S3 Users: Managing S3 Users ~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part12.inc .. _Adding S3 users: Adding S3 Users *************** To add an S3 user, do the following: #. On the **SERVICES** > **S3 Users** screen, click **Add user**. .. image:: /images/stor_image47.png :align: center :class: align-center #. Specify a valid email address as login for the user and click **Done**. .. image:: /images/stor_image48.png :align: center :class: align-center .. _Managing S3 Access Key Pairs: Managing S3 Access Key Pairs **************************** Each S3 user has one or two key pairs (access key and secret key) for accessing the S3 cloud. You can think of the access key as login and the secret key as password. (For more information about S3 key pairs, refer to the `Amazon documentation `__.) The access keys are generated and stored locally in the |product_name| cluster on S3 name servers. Each user can have up to two key pairs. It is recommended to periodically revoke old and generate new access key pairs. To view, add, or revoke the S3 access key pairs for an S3 user, do the following: #. Select a user in the list and click **Keys**. .. image:: /images/stor_image49.png :align: center :class: align-center #. The existing keys will be shown on the **Keys** panel. - To revoke a key, click **Revoke**. - To add a new key, click **Generate access key**. To access a bucket, a user will need the following information: - management panel IP address, - DNS name of the S3 cluster specified during configuration, - S3 access key ID, - S3 secret access key, - SSL certificate if the HTTPS protocol was chosen during configuration. .. note:: The certificate file can be found in the ``/etc/nginx/ssl/`` directory on any node hosting the S3 gateway service. To automatically log in to S3 with user credentials using the generated keys, select a user and click **Browse**. .. note:: To **Browse** using an SSL certificate, make sure it is valid or, in case of a self-signed one, add it to browser's exceptions. .. _Managing S3 Buckets: Managing S3 Buckets ~~~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part13.inc In the current version of |product_name|, you can enable and disable Acronis Notary for object storage buckets and monitor the space used by them on the **SERVICES** > **S3** > **Buckets** screen. You cannot create and manage object storage buckets from |product_name| management panel. However, you can do it via the |product_name| user panel or by using a third-party application. For example, the applications listed below allow you to perform the following actions: - CyberDuck: create and manage buckets and their contents. - MountainDuck: mount object storage as a disk drive and manage buckets and their contents. - Backup Exec: store backups in the object storage. .. _Listing S3 Bucket Contents: Listing S3 Bucket Contents ************************** .. include:: /includes/exporting-data-via-s3-part14.inc .. _Managing Acronis Notary in S3 Buckets: Managing Acronis Notary in S3 Buckets ************************************* |product_name| offers integration with the Acronis Notary service to leverage blockchain notarization and ensure the immutability of data saved in object storage clusters. To use Acronis Notary in user buckets, you need to set it up in the S3 cluster and enable it for said buckets. .. _Setting Up Acronis Notary: Setting Up Acronis Notary ^^^^^^^^^^^^^^^^^^^^^^^^^ To set up Acronis Notary, do the following: #. Get the DNS name and the user key for the notary service from your sales contact. #. On the **SERVICES** > **S3** screen, click **Notary settings**. .. image:: /images/stor_image67.png :align: center :class: align-center #. On the **Notary Settings** screen, specify the DNS name and user key in the respective fields and click **Done**. .. image:: /images/stor_image66.png :align: center :class: align-center .. _Enabling and Disabling Acronis Notary: Enabling and Disabling Acronis Notary ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To enable or disable blockchain notarization for a bucket, select a bucket on the **SERVICES** > **S3** > **Buckets** screen and click **Enable Notary** or **Disable Notary**, respectively. Notarization is disabled for new buckets by default. .. note:: Once you enable notarization for a bucket, certificates are created automatically only for the newly uploaded files. The previously uploaded files are left unnotarized. Once a file was notarized, it will remain notarized even if you disable notarization later. .. _Best Practices for Using S3 in |product_name|: Best Practices for Using S3 in |product_name| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section offers recommendations on how to best use the S3 feature of |product_name|. .. _S3 Bucket and Key Naming Policies: S3 Bucket and Key Naming Policies ********************************* .. include:: /includes/exporting-data-via-s3-part4.inc .. _Improving Performance of PUT Operations: Improving Performance of PUT Operations *************************************** .. include:: /includes/exporting-data-via-s3-part5.inc .. _Replicating S3 Data Between Datacenters: Replicating S3 Data Between Datacenters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |product_name| can store replicas of S3 cluster data and keep them up-to-date in multiple geographically distributed datacenters with S3 clusters based on |product_name|. Geo-replication reduces the response time for local S3 users accessing the data in a remote S3 cluster or remote S3 users accessing the data in a local S3 cluster as they do not need to have an Internet connection. Geo-replication schedules the update of the replicas as soon as any data is modified. Geo-replication performance depends on the speed of Internet connection, the redundancy mode, and cluster performance. If you have multiple datacenters with enough free space, it is recommended to set up geo-replication between S3 clusters residing in these datacenters. .. important:: Each cluster must have its own SSL certificate signed by a global certificate authority. To set up geo-replication between S3 clusters, exchange tokens between datacenters as follows: #. In the management panel of a remote datacenter, open the **SERVICES** > **S3** > **GEO-REPLICATION** screen. .. image:: /images/stor_image66_1.png :align: center :class: align-center #. In the section of the home S3 cluster, click **TOKEN** and, on the **Get token** panel, copy the token. #. In the management panel of the local datacenter, open the **SERVICES** > **S3** > **GEO-REPLICATION** screen and click **ADD DATACENTER**. .. image:: /images/stor_image66_2.png :align: center :class: align-center #. Enter the copied token and click **Done**. #. Configure the remote |product_name| S3 cluster the same way. .. _Monitoring S3 Access Points: Monitoring S3 Access Points ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The S3 monitoring screen enables you to inspect the availability of each S3 component as well as the performance of NS and OS services (which are highly available). If you see that some of the NS or OS services are offline, it means that the S3 access point does not function properly, and you should contact support consult the CLI guide for low-level troubleshooting. S3 gateways are not highly available, but DNS load balancing should be enough to avoid downtime if the gateway fails. The performance charts represent the number of operations that the OS/NS services are performing. .. _Releasing Nodes from S3 Clusters: Releasing Nodes from S3 Clusters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before releasing a node, make sure that the cluster has enough nodes running name and object servers as well as gateways left. .. warning:: When the last node in the S3 cluster is removed, the cluster is destroyed, and all the data is deleted. To release a node from an S3 cluster, do the following: #. On the **SERVICES** > **S3 Nodes** screen, check the box of the node to release. #. Click **Release**. .. _Supported Amazon S3 Features: Supported Amazon S3 Features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. include:: /includes/exporting-data-via-s3-part6.inc .. _Supported Amazon S3 REST Operations: Supported Amazon S3 REST Operations *********************************** .. include:: /includes/exporting-data-via-s3-part7.inc .. _Supported Amazon Request Headers: Supported Amazon Request Headers ******************************** .. include:: /includes/exporting-data-via-s3-part8.inc .. _Supported Amazon Response Headers: Supported Amazon Response Headers ********************************* .. include:: /includes/exporting-data-via-s3-part9.inc .. _Supported Amazon Error Response Headers: Supported Amazon Error Response Headers *************************************** .. include:: /includes/exporting-data-via-s3-part10.inc .. _Supported Authentication Scheme and Methods: Supported Authentication Scheme and Methods ******************************************* .. include:: /includes/exporting-data-via-s3-part11.inc