Oracle Cloud Infrastructure Documentation

Moving Data to and from Object Storage

Learn how to move your data between Object Storage and other data hosting systems.

Use the Rclone tool to synchronize files and directories between an Object Storage bucket and other cloud storage providers, or between an Object Storage bucket and on-premises systems. Rclone is known for its simplicity, efficiency, and wide support for many cloud services, including Object Storage. For more information, see Rclone.

Prerequisites

Installation and use of Rclone has the following prerequisites:

  • You must have permissions to create and edit files, create directories and folder, run commands, and install software on your laptop or workstation
  • You must have access to your Object Storage and to your target cloud storage.

Installing Rclone

If you already have a networking environment set up in your tenancy with VCN and networks and can launch compute instances, you can run this lab on an OCI Compute instance. We recommend using Oracle Linux.

Linux and BSD Systems

Open a terminal and run the following command:

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Windows

Download the correct installation file for your processor type:

Extract the file and add the final location of the Rclone executable file (.exe) to your Windows system PATH environment variable for easy command line access.

Mac OS

Open a terminal window and install Rclone on Mac OS X using Homebrew:

brew update && brew install rclone

Installing OCI CLI

Use of the OCI command line interface (CLI) is required to use Rclone to move your data to and from Object Storage. To set up the OCI CLI on your computer, see Installing the CLI.

Configuring Rclone

  1. Record the full path to your OCI CLI configuration file, which is named config.
    This configuration file's location is typically the in .oci directory under your home directory. For example: 
    <home>/opc/.oci/config

    On Windows, the CLI configuration file would reside in the following location:

    %HOMEDRIVE%%HOMEPATH%\.oci
  2. Open the CLI configuration file and record the region name and the tenancy OCID.
    The following example shows how the CLI configuration file would look with the user OCID, API Key fingerprint, tenancy OCID, home region, and the private API Key_file path:
    [DEFAULT]
user=ocid1.user.oc1..exampleuniqueID
fingerprint=60:09:0a:00:ab:60:a0:00:0c:00:0c:16:d0:00:09:0a
tenancy=ocid1.tenancy.oc1..exampleuniqueID
region=us-ashburn-1
key_aile=/Users/<username>/.oci/oci_api_key.pem
  3. Run the following CLI command to get your tenancy namespace: 
    oci os ns get

    You need this namespace later in the steps.

  4. Create the directory that will contain the Rclone configuration file (rclone.conf) that you will create later.
    For example:
    <home>/.config/rclone/
    On Windows, create the following directory as:
    C:\Users\<username>\config\rclone
  5. Navigate to the Rclone directory you created.
  6. Using your favorite text editor, create a file named rclone.conf containing the following content: 
    [oci]
type = oracleobjectstorage
provider = user_principal_auth
namespace = <namespace>
compartment = <tenancy_or_compartment_OCID>
region = <oci_region>
config_file = <path_to_cli_configuration_file>

    The first line in square brackets ([oci]) is the name of remote. In this example, oci is used to identify the remote as Oracle Cloud Infrastructure. You reference this remote name when making Rclone commands.

    The following example shows what the rclone.conf file might look like with real data:

    [oci]
type = oracleobjectstorage
provider = user_principal_auth
namespace = ns1
compartment = ocid1.tenancy.oc1..exampleuniqueID
region = us-ashburn-1
config_file = /Users/username/.oci/config
  7. Run the following Rclone command to get information on the prospective target providers for the data you're moving out of OCI Object Storage: 
    rclone config

    Running rclone config manages the configuration of remotes, allowing you to connect and interact with various cloud storage services and other locations. It helps you to set up, update, and manage the connections to these services by storing the necessary credentials and settings in a configuration file.

  8. Add another entry to the rclone.conf file to specify the target provider of the Object Storage data being moved using the information obtained from running rclone config.

Copy Data from OCI Object Storage

Copy the data from the Object Storage bucket containg your data into the target location.

Other Cloud Providers

Contact the cloud provider or check https://rclone.org/ for information on how to copy data from an Object Storage bucket into the bucket of another cloud.

On-Premises

Run the following to copy data to a local filesystem on-premises:

rclone copy --progress --transfers=20 oci:<oci_bucket> /<local_filesystem_or_nfs_mount>