Creating a Pipeline

Create a pipeline in a project or folder in Data Integration. A pipeline is a design-time resource for connecting tasks and activities in one or more sequences or in parallel from start to finish to orchestrate data processing.

Data Integration includes one default project to get you started. To create other projects or folders, see Projects and Folders.

You can add published tasks and design-time tasks in a pipeline. The published tasks can be from any application in the current workspace or another workspace in the same compartment or a different compartment.

Ensure that you review Support Information and Considerations When Using Pipelines before you create or edit a pipeline. For example, pay attention to the nested depth level and the task limit per level.

1. Open the project or folder in which you want to create the pipeline
  
  For the steps to open the details page of a project or folder, see Viewing the Details of a Project or Viewing the Details of a Folder.
2. On the project or folder details page, click Pipelines.
3. In the Pipelines section, click Create pipeline.
  
  Data Integration displays the designer in a tab. The Operators panel and Properties panel are open, and a start operator and an end operator are placed on the canvas for you. For information about these operators, see Start Operator and End Operator.
4. On the Details tab in the Properties panel, enter a name and an optional description for the pipeline.
  
  The identifier is a system-generated value based on the name. You can change the value, but after you create and save the pipeline, you can't update the identifier.
5. (Optional) For Project or folder field, click Select and select a different project or folder to save the pipeline in.
6. Drag pipeline operators from the Operators panel onto the canvas to design the pipeline.
  
  To duplicate a task or expression operator, right-click the operator icon and select Duplicate. Then select the duplicated operator and rename the identifier in the Properties panel.
  
  Consider the following restrictions when using the duplicate action:
  
  The duplicate action isn't available for the start, end, and merge operators.
  
  If the original operator is connected to other operators, the connections aren't copied to the duplicated operator.
  
  If the original operator has configured outputs from the previous operator, the output configurations aren't copied to the duplicated operator.
7. To save the pipeline for the first time, click one of the following buttons:
  
  Create: Creates and saves the pipeline. You can continue to create and edit the pipeline in the designer.
  
  Create and close: Creates and saves the pipeline, closes the designer, and returns you to the Pipelines list on the project or folder details page.
  
  Save periodically while you work in the designer by clicking one of the following buttons:
  
  Save: Commits changes since the last save. You can continue editing after saving.
  
  Save and close: Commits changes, closes the designer, and returns you to the Pipelines list on the project or folder details page.
  
  Save as: Commits changes (since the last save) and saves to a copy instead of overwriting the current pipeline. You can provide a name for the copy and select a different project or folder for the copy, or save the copy in the same project or folder as the current pipeline.
8. On the Configuration tab in the Properties panel, complete the configuration of a task operator, and the rule on the end operator.
  
  Configure any incoming parameter values in the tasks or associated data flows. You can assign a value directly, use a user-defined or system parameter, or use a value of an output from a previous operator.
  
  Configure task run options.
  
  Configure the rule on the end operator to specify a rule for displaying the pipeline task run status.
9. On the Parameters tab in the Properties panel for the pipeline, view and edit the pipeline parameters.
10. Validate the pipeline to check for warnings or errors that could cause issues during runtime. In the designer toolbar, click Validate.
  
  Data Integration displays the Global validation panel. If warnings or errors are found, click an identifier name in the list of issues to bring the operator with that warning or error into focus on the canvas.
11. When you finish working in the pipeline, click Create and close or Save and close.
To run the pipeline, create a pipeline task. See Creating a Pipeline Task.
Use the oci data-integration pipeline create command and required parameters to create a pipeline:
oci data-integration pipeline create [OPTIONS]
For a complete list of flags and variable options for CLI commands, see the Command Line Reference.
Run the CreatePipeline operation to create a pipeline.

Oracle Cloud Infrastructure Documentation

Creating a Pipeline