Enabling Oracle Cloud Infrastructure Logging Spark Diagnostic Logs

Follow these steps to enable Oracle Cloud Infrastructure Logging Spark diagnostic logs.

Follow these steps in the Console to use Oracle Cloud Infrastructure Logging for Spark diagnostic logs in Data Flow.

Note

Spark Diagnostic Logs (Oracle Cloud Infrastructure Logging Service Logs) can be enabled only at the Data Flow Application level and can't be overridden.
1. After you have created an Application that uses Oracle Cloud Infrastructure Logging, open the details page for that Application. If you need help finding the Data Flow page, see Viewing an Application's Details.
2. Under Resources, select Logs.
3. Select Diagnostic logs.
4. Toggle Enable log. The Enable Log panel is displayed.
5. Select a Compartment.
6. Select a Log group.
7. Enter the Log name.
8. Select the amount of time for Log retention.
9. Select Enable log.
  Wait for the Status of the Diagnostic logs to be Active. After it's active, any runs using this Application have the diagnostic logs feature turned on for the Spark Driver and Executor logs. The Driver and Executor logs are technically 'Service logs' as they come from an Oracle Cloud Infrastructure native service such as Data Flow.

Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging Spark diagnostic logs in Data Flow.

Create a Data Flow Application.

Request

oci data-flow application create \
--compartment-id $COMPARTMENT_ID \
--display-name "Sample CSV Processing App with OCI Logging Service Logs" \
--driver-shape "VM.Standard2.1" \
--executor-shape "VM.Standard2.1" \
--file-uri "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar" \
--language "JAVA" \
--num-executors 1 \
--spark-version "3.0.2" \
--class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"

Response

{
  "data": {
    "archive-uri": "",
    "arguments": null,
    "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample",
    "compartment-id": "<compartment-ocid>",
    "configuration": null,
    "defined-tags": {
      "Dataflow_Resource_Category": {
        "category": "DEV_QA"
      }
    },
    "description": null,
    "display-name": "Sample CSV Processing App with OCI Logging Service Logs",
    "driver-shape": "VM.Standard2.1",
    "execute": null,
    "executor-shape": "VM.Standard2.1",
    "file-uri": "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar",
    "freeform-tags": {},
    "id": "<data-flow-application-ocid>",
    "language": "JAVA",
    "lifecycle-state": "ACTIVE",
    "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/",
    "metastore-id": null,
    "num-executors": 1,
    "owner-principal-id": "<user-ocid>",
    "owner-user-name": "<user-name>",
    "parameters": null,
    "private-endpoint-id": "",
    "spark-version": "3.0.2",
    "time-created": "2022-04-08T19:50:57.795000+00:00",
    "time-updated": "2022-04-08T19:50:57.795000+00:00",
    "warehouse-bucket-uri": null
  }
}

Create a log group.

Request

oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dataflow-logs"

Response

{
  "opc-work-request-id": "<log-workrequest-ocid>"
}

Retrieve the log group ID.

Request

export LOG_GROUP_WR_ID=<log-workrequest-ocid>
oci logging work-request get --work-request-id $LOG_GROUP_WR_ID

Response

{
  "data": {
    "compartment-id": "<compartment-ocid>",
    "id": "<log-workrequest-ocid>",
    "operation-type": "CREATE_LOG_GROUP",
    "percent-complete": 100.0,
    "resources": [
      {
        "action-type": "CREATED",
        "entity-type": "loggroup",
        "entity-uri": "/logGroups/<log-group-ocid>",
        "identifier": "<log-group-ocid>"
      }
    ],
    "status": "SUCCEEDED",
    "time-accepted": "2022-04-08T19:53:57.065000+00:00",
    "time-finished": "2022-04-08T19:53:57.065000+00:00",
    "time-started": "2022-04-08T19:53:57.065000+00:00"
  }
}

Enable diagnostic (service) logs on the Data Flow Application.

Request

export LOG_GROUP_ID=<log-group-ocid>
oci logging log create --display-name "sample_csv_processing_diagnostic" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json

config.json

{
    "compartment-id": "<compartment-ocid>",
    "source": {
        "resource": "<data-flow-application-ocid>",
        "service": "dataflow",
        "source-type": "OCISERVICE",
        "category": "diagnostic"
    }
}

Response

{
  "opc-work-request-id": "<log-workrequest-ocid>"
}

Retrieve the log ID for the diagnostic (service) logs.

Request

export LOG_WR_ID=<log-workrequest-ocid>
oci logging work-request get --work-request-id $LOG_WR_ID

Response

{
  "data": {
    "compartment-id": "<compartment-ocid>",
    "id": "<log-workrequest-ocid>",
    "operation-type": "CREATE_LOG",
    "percent-complete": 100.0,
    "resources": [
      {
        "action-type": "CREATED",
        "entity-type": "log",
        "entity-uri": "/logs/<log-ocid>",
        "identifier": "<log-ocid>"
      }
    ],
    "status": "SUCCEEDED",
    "time-accepted": "2022-04-08T20:12:29.660000+00:00",
    "time-finished": "2022-04-08T20:12:33.356000+00:00",
    "time-started": "2022-04-08T20:12:29.660000+00:00"
  }
}

Follow these steps in the API to use Oracle Cloud Infrastructure Logging Spark diagnostic logs in Data Flow.

Create a Data Flow Application.

Request

oci data-flow application create \
--compartment-id $COMPARTMENT_ID \
--display-name "Sample CSV Processing App with OCI Logging Service Logs" \
--driver-shape "VM.Standard2.1" \
--executor-shape "VM.Standard2.1" \
--file-uri "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar" \
--language "JAVA" \
--num-executors 1 \
--spark-version "3.0.2" \
--class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"

Response

{
  "data": {
    "archive-uri": "",
    "arguments": null,
    "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample",
    "compartment-id": "<compartment-ocid>",
    "configuration": null,
    "defined-tags": {
      "Dataflow_Resource_Category": {
        "category": "DEV_QA"
      }
    },
    "description": null,
    "display-name": "Sample CSV Processing App with OCI Logging Service Logs",
    "driver-shape": "VM.Standard2.1",
    "execute": null,
    "executor-shape": "VM.Standard2.1",
    "file-uri": "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar",
    "freeform-tags": {},
    "id": "data-flow-application-ocid>",
    "language": "JAVA",
    "lifecycle-state": "ACTIVE",
    "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/",
    "metastore-id": null,
    "num-executors": 1,
    "owner-principal-id": "<user-ocid>",
    "owner-user-name": "<user-name>",
    "parameters": null,
    "private-endpoint-id": "",
    "spark-version": "3.0.2",
    "time-created": "2022-04-08T19:50:57.795000+00:00",
    "time-updated": "2022-04-08T19:50:57.795000+00:00",
    "warehouse-bucket-uri": null
  }
}

Create a log group.

Request

oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dataflow-logs"

Response

{
  "opc-work-request-id": "<log-workrequest-ocid"
}

Retrieve the log group ID.

Request

export LOG_GROUP_WR_ID=<log-workrequest-ocid
oci logging work-request get --work-request-id $LOG_GROUP_WR_ID

Response

{
  "data": {
    "compartment-id": "<compartment-ocid>",
    "id": "<log-workrequest-ocid",
    "operation-type": "CREATE_LOG_GROUP",
    "percent-complete": 100.0,
    "resources": [
      {
        "action-type": "CREATED",
        "entity-type": "loggroup",
        "entity-uri": "/logGroups/<log-group-ocid>",
        "identifier": "<log-group-ocid>"
      }
    ],
    "status": "SUCCEEDED",
    "time-accepted": "2022-04-08T19:53:57.065000+00:00",
    "time-finished": "2022-04-08T19:53:57.065000+00:00",
    "time-started": "2022-04-08T19:53:57.065000+00:00"
  }
}

Enable diagnostic (service) logs on the Data Flow Application.

Request

export LOG_GROUP_ID=<log-group-ocid>
oci logging log create --display-name "sample_csv_processing_diagnostic" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json

config.json

{
    "compartment-id": "<compartment-ocid>",
    "source": {
        "resource": "data-flow-application-ocid>",
        "service": "dataflow",
        "source-type": "OCISERVICE",
        "category": "diagnostic"
    }
}

Response

{
  "opc-work-request-id": "<log-workrequest-ocid"
}

Retrieve the log ID for the diagnostic (service) logs.

Request

export LOG_WR_ID=<log-workrequest-ocid
oci logging work-request get --work-request-id $LOG_WR_ID

Response

{
  "data": {
    "compartment-id": "<compartment-ocid>",
    "id": "<log-workrequest-ocid",
    "operation-type": "CREATE_LOG",
    "percent-complete": 100.0,
    "resources": [
      {
        "action-type": "CREATED",
        "entity-type": "log",
        "entity-uri": "/logs/<log-ocid>",
        "identifier": "<log-ocid>"
      }
    ],
    "status": "SUCCEEDED",
    "time-accepted": "2022-04-08T20:12:29.660000+00:00",
    "time-finished": "2022-04-08T20:12:33.356000+00:00",
    "time-started": "2022-04-08T20:12:29.660000+00:00"
  }
}

Oracle Cloud Infrastructure Documentation

Enabling Oracle Cloud Infrastructure Logging Spark Diagnostic Logs