Oracle Cloud Infrastructure Documentation

Storage Mounts

Similar to ML Jobs, Pipelines enable customers to mount File Storage and Object Storage as local folders for pipeline steps.

This simplifies the process of accessing data stored in File or Object Storage, removing the need for installing extra libraries or writing specialized code. Using standard OSs or language libraries, users can access shared file systems and object storage buckets in the same way they access local files and folders.

You can also override storage mounts in pipeline runs, enabling one-off changes for a specific run. This is essential for SaaS customers who use pipelines within ML Applications and need to override storage mounts for a particular SaaS customer (tenant).

Configuring Storage Mounts

Ensure you have the appropriate Policies applied.

Storage mounts can be configured at two levels:

  • Pipeline level: The storage mount configuration is applied to all steps, unless overridden.
  • Step level: The storage mount configuration is applied only to the specific step.
Note

Storage mounts apply only to the following step types:
  • Script step type. Steps defined by a script.
  • BYOC step type. Steps defined by a container image.

The configuration is managed in the Storage Mounts configuration panel.

Note

You can configure no more than five storage mounts for a pipeline or a step.
  1. To use File storage mounts, select +Add file storage mount.
    1. Select a compartment that contains the storage resource that you want to mount.
    2. The mount target and export path you want to use. You must use a custom network to use file storage.
    3. Enter the path under which the storage is to be mounted.

      Storage is mounted under the specified mount path. The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The permitted characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

      You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

    4. Repeat these steps to add up to five storage mounts for a pipeline or a step.

    Note

    Custom Networking must be used to use a file storage mount.
  2. To use Object storage mounts, select +Add object storage mount.
    1. Select a compartment that contains the storage resource that you want to mount.
    2. Specify the bucket you want to use. You can add an object name prefix. The prefix must start with an alphanumeric character. The permitted characters are alphanumerics, slash ( / ), hyphen ( - ) and underscore ( _ ).
    3. Enter the path under which the storage is to be mounted.

      Storage is mounted under the specified mount path. The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The permitted characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

      You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

    4. Repeat these steps to add up to five storage mounts for a pipeline or a step.

Storage Mount Overrides

You can specify storage mount overrides for Pipeline Runs the same way as for Pipelines. For details about storage mounts, see Storage Mounts.

When creating a pipeline run, you can define the override at the pipeline level or for a pipeline step.