Oracle Cloud Infrastructure Documentation

Details for Data Flow

Logging details for Data Flow Spark diagnostic logs.

Resources

  • applications

Log Categories

API value (ID): Console (Display Name) Description
all Diagnostic Includes all logs generated by the Apache Spark framework (driver and executors).

Availability

Data Flow logging is available in all the regions of the commercial realms.

Comments

Spark diagnostic logs can be enabled only at the Data Flow application level, and can't be overridden.

If you enable logging for a Data Flow application, Spark diagnostic logs are streamed for any new Data Flow run submission. Already accepted or in-progress runs aren't updated.

Contents of a Data Flow Spark Diagnostic Log

Property Description
specversion Oracle Cloud Infrastructure Logging schema version of the log.
type Log category that follows the com.oraclecloud.{service}.{resource-type}.{category} convention.
com.oraclecloud.dataflow.run.driver
com.oraclecloud.dataflow.run.executor
source Name of the resource that generated the message.
subject A specific subresource that generated the event.
id A source-unique identifier for this batch ingestion.
time Time the function output was generated, in RFC 3339 timestamp format.
oracle.logid OCID of the Oracle Cloud Infrastructure Logging log object.
oracle.loggroupid OCID of the Oracle Cloud Infrastructure Logging log group.
oracle.compartmentid OCID of the compartment the Oracle Cloud Infrastructure Logging log group is in.
oracle.tenantid OCID of the tenant.
oracle.ingestedtime Time the log line was ingested by Oracle Cloud Infrastructure Logging, in RFC 3339 timestamp format.
data[i].id Unique Identifier for this log event.
data[i].time Time when this specific log entry was generated. Must adhere to the format specified in RFC 3339.
data[i].data Non-empty data representing a log event.
data.data[i].level The logging level of the logging event.
data.data[i].message A message describing the event details.
data.data[i].opcRequestId A unique Oracle-assigned request ID generated when the Data Flow run was submitted and included in the createRun response.
data.data[i].runId The OCID of the Data Flow run whose resource (a Spark driver or executor) generated this message.
data.data[i].thread The name of the thread that generated the logging event.

Example Data Flow Spark Diagnostic Log

{
  "datetime": 1687551602245,
  "logContent": {
    "data": {
      "logLevel": "INFO",
      "message": "Execution complete.",
      "opcRequestId": "<unique_ID>",
      "runId": "ocid1.dataflowrun.oc1.ca-toronto-1.<unique_ID>",
      "thread": "shaded.dataflow.oracle.dfcs.spark.wrapper.DataflowWrapper"
    },
    "id": "<unique_ID>",
    "oracle": {
      "compartmentid": "ocid1.tenancy.oc1..<unique_ID>",
      "ingestedtime": "2023-06-23T20:20:06.974Z",
      "loggroupid": "ocid1.loggroup.oc1.ca-toronto-1.<unique_ID>",
      "logid": "ocid1.log.oc1.ca-toronto-1.<unique_ID>",
      "tenantid": "ocid1.tenancy.oc1..<unique_ID>"
    },
    "source": "Sample CSV Processing App",
    "specversion": "1.0",
    "subject": "spark-driver",
    "time": "2023-06-23T20:20:02.245Z",
    "type": "com.oraclecloud.dataflow.run.driver"
  },
  "regionId": "ca-toronto-1"
}

Using the CLI

See Enable Oracle Cloud Infrastructure Logging Spark Diagnostic Logs for an example command to enable Data Flow Spark diagnostic logging.