Scheduling Data Science Job Runs

In this tutorial, you use Data Integration to schedule job runs for your Data Science jobs.

Key tasks include how to:

Create a job with a Data Science job artifact.
Set up a REST task to create a job with the same specifics as the job created with the artifact.
Set up a schedule and assign it to the REST task.
Have the task scheduler create the Data Science jobs.

A diagram of a user connected from a local machine to an Oracle Cloud Infrastructure compartment called data-science-work compartment. The user creates a job artifact, hello_world_job.py, and sends the job to a Data Science project. The Data Science project is called DS Project and the job is called, hello_world_job. In another workflow, from a Data Integration workspace called hello_world_workspace, a hello_world_REST_task is published to the Scheduler Application of the workspace. Scheduler Application contains hello_world_task_schedule that sends hello_world_job instances to the DS Project. The hello_world_task_schedule contains a hello_world_task and a hello_world_schedule, suggesting that the schedule for the task comes from the hello_world_schedule. The DS Project displays scheduled job runs coming from the Scheduler application, called HELLO WORLD JOB RUN.

Before You Begin

To successfully perform this tutorial, you must have the following:

Requirements

1. Prepare

Create and set up dynamic groups, policies, a compartment, and a Data Science project for your tutorial.

Set Up Resources

Perform the Manually Configuring a Data Science Tenancy tutorial with the following specifics:

Note

If you have performed the Manually Configuring a Data Science Tenancy before, ensure that you read the next steps and incorporate the policies that apply to this tutorial.

Perform all the steps in Step 1. Creating a User Group, and name your group, data-scientists.
Perform all the steps in Step 2. Creating a Compartment, and name the compartment for your work data-science-work.
From the detail page of data-science-work compartment, copy the <data-science-work-compartment-ocid>.
Follow all the steps in Step 3. Creating a VCN and Subnet. This step is required for this tutorial. In the data-science-work compartment, use the wizard to create a VCN with the name, datascience-vcn.
In Step 4. Creating Policies, create a policy in the data-science-work compartment called data-science-policy, and only add the following policies:
```
allow group data-scientists to manage all-resources in compartment data-science-work 
allow service datascience to use virtual-network-family in compartment data-science-work
```
The first policy gives you administrative rights to your compartment, in which you can manage all the resources of OCI services.
In Step 5. Creating a Dynamic Group with Policies, create a dynamic group called data-science-dynamic-group with the following three matching rules:
Replace <data-science-work-compartment-ocid> with the OCID that you copied in step 3.
```
ALL {resource.type='datasciencenotebooksession', resource.compartment.id='<data-science-work-compartment-ocid>'}
```
```
ALL {resource.type='datasciencemodeldeployment', resource.compartment.id='<data-science-work-compartment-ocid>'}
```
```
ALL {resource.type='datasciencejobrun', resource.compartment.id='<data-science-work-compartment-ocid>'}
```
Note

You only need the last matching rule for the datasciencejobrun resource used in this tutorial. Add the other Data Science resources to be prepared for working with notebook sessions and model deployments.

For Step 5, create a policy called data-science-dynamic-group-policy in the root (tenancy) compartment. Click Show manual editor and add the following policies for the data-science-dynamic-group:

allow dynamic-group data-science-dynamic-group to manage all-resources in compartment data-science-work
allow dynamic-group data-science-dynamic-group to read compartments in tenancy
allow dynamic-group data-science-dynamic-group to read users in tenancy

For Step 6. Creating a Notebook Session, create a project in the data-science-work compartment called DS Project and skip creating a notebook session.

Note

In this tutorial, you name your Data Science project, DS Project, and later you name your Data Integration project DI Project. Don't name your project Initial Project as you are instructed in Step 6.

Add Data Integration Policy

Allow the Data Integration service to create workspaces.

Open the navigation menu and click Identity & Security. Under Identity, click Policies.
In the left navigation, under List Scope, click data-science-work for the compartment.
Click data-science-policy that you created in the Set Up Resources step.
Click Edit Policy Statements.
Click Advanced.

In a new line, add the following statement:

allow service dataintegration to use virtual-network-family in compartment data-science-work

Click Save Changes.

Note

The preceding policy allows the Create workspace dialog of the Data Integration service to list the VCNs in the data-science-work compartment, allowing you to assign a VCN to your workspace when you create it. The workspace then uses this VCN for its resources.

Add Data Integration to Dynamic Group

In this step, you add Data Integration workspaces to the data-science-dynamic-group. The data-science-dynamic-group-policy allows all members of this dynamic group to manage the data-science-family. This way, the workspace resources such tasks schedules can create your Data Science jobs.

Open the navigation menu and click Identity & Security. Under Identity, click Dynamic Groups.
In the list of Dynamic Groups, click the data-science-dynamic-group that you created in the Set Up Resources step.
Click Edit All Matching Rules.
Add the following matching rule:
```
ALL {resource.type='disworkspace', resource.compartment.id='<data-science-work-compartment-ocid>'}
```
Replace <data-science-work-compartment-ocid> with the OCID for datascience-work compartment.

Tip

You can copy the <data-science-work-compartment-ocid> from another rule in the data-science-dynamic-group matching rules, because they all point to the datascience-work compartment.

The preceding matching rule means that all Data Integration workspaces created in your compartment are added to data-science-dynamic-group. The data-science-dynamic-group-policy created for data-science-dynamic-group now applies to the workspaces in this compartment.

2. Set Up a Job Run

Create a Job Artifact

Create a hello world Python job artifact to use in your job and job runs:

Copy the following Python code into a text file.
```
# simple job
print("Hello world job!")
```
Save the code as hello_world_job.py on your local machine.

Create a Job

Start a Job Run

Gather Job Info

3. Set Up the Task

For a visual relationship of the components, refer to the scheduler diagram.

Create a Workspace

Update Project Name

Create a REST Task in the Workspace

Create a task and define the REST API parameters for creating a job run.

In the trail that displays the current page, go back to the hello_world_workspace workspace.
In the Quick actions panel of the hello_world_workspace, click Create REST task.
Name the task, hello_world_REST_task.
For Project or Folder, select DI Project.
Configure your REST API details:
- HTTP method: POST
- URL: Find the API endpoint and path for your URL:
  - From the Data Science API, copy the API endpoint for your region. The endpoint must include the <region-identifier> you copied in the Gather Job Info section.
    https://datascience.<region-identifier>.oci.oraclecloud.com
  - From POST /<REST_API_version>/jobRuns, copy the POST command for CreateJobRun.
    /<REST_API_version>/jobRuns
  - Put the two sections together:
    https://datascience.<region-identifier>.oci.oraclecloud.com/<REST_API_version>/jobRuns
    Example:
    https://datascience.us-phoenix-1.oci.oraclecloud.com/20190101/jobRuns
- Request: Click the Request link to activate it and then construct the body with the following attributes from CreateJobRunDetails Reference:
```
{
    "projectId": "<data-science-project-ocid>",
    "compartmentId": "<data-science-work-compartment-ocid>",
    "jobId": "<data-science-hello-world-job-ocid>",
    "definedTags": {},
    "displayName": "HELLO WORLD JOB RUN",
    "freeformTags": {},
    "jobConfigurationOverrideDetails": {
    "jobType": "DEFAULT"
    }
}
```
  In the Request body, replace the fields with brackets with the information from the Gather Job Info section.
  Example:
```
{
    "projectId": "ocid1.datascienceproject.oc1....",
    "compartmentId": "ocid1.compartment.oc1.....",
    "jobId": "ocid1.datasciencejob.oc1....",
    "definedTags": {},
    "displayName": "HELLO WORLD JOB RUN",
    "freeformTags": {},
    "jobConfigurationOverrideDetails": {
    "jobType": "DEFAULT"
    }
}
```
- Click Next, review the default conditions, and keep their default options:
  Success condition: SYS.RESPONSE_STATUS >= 200 AND SYS.RESPONSE_STATUS < 300
Click Configure.
For Authentication, Configure the following options:
- Authentication: OCI resource principal
- Authentication source: Workspace
Click Configure.
Skip configuring the Parameters (Optional) panel.
Click Validate task.
After you get Validation: Successful, click Create.

After the workspace shows that REST task created successfully, click Save and Close.

Note

In the Request body of your REST task, you assign values to the parameters needed for creating a job run. You use the same values as the hello_world_job you created in Data Science in the Create a Job section of this tutorial.

References:

Create an Application