Enabling Oracle Cloud Infrastructure Logging Applications Logs
Follow these steps to use Oracle Cloud Infrastructure Logging for applications logs in Data Flow.
Follow these steps in the Console to use Oracle Cloud Infrastructure Logging for applications logs in Data Flow.
Note
Application Logs (Oracle Cloud Infrastructure Logging Custom Logs) can be enabled at the Data Flow Application level and the Data Flow Run level, and can be overridden.- After you have created an Application that uses Oracle Cloud Infrastructure Logging, open the details page for that Application. If you need help finding the Data Flow page, see Viewing an Application's Details.
- Under Resources, select Logs.
- Select Application logs.
- Toggle Enable log. The Enable Log panel is displayed.
- Select a Compartment.
- Select a Log group.
- Enter the Log name.
- Select the amount of time for Log retention.
-
Select Enable log.
Wait for the Status of the Application logs to be
Active
. After it's active, any runs using this Application have the Application Logs feature turned on for the Application stdout and stderr logs.
Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging application logs in Data Flow.
-
Create a Data Flow Application.
- Request
-
oci data-flow application create \ --compartment-id $COMPARTMENT_ID \ --display-name "Sample CSV Processing App with OCI Logging Service Logs" \ --driver-shape "VM.Standard2.1" \ --executor-shape "VM.Standard2.1" \ --file-uri "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar" \ --language "JAVA" \ --num-executors 1 \ --spark-version "3.0.2" \ --class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"
- Response
-
{ "data": { "archive-uri": "", "arguments": null, "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample", "compartment-id": "<compartment-ocid>", "configuration": null, "defined-tags": { "Dataflow_Resource_Category": { "category": "DEV_QA" } }, "description": null, "display-name": "Sample CSV Processing App with OCI Logging Service Logs", "driver-shape": "VM.Standard2.1", "execute": null, "executor-shape": "VM.Standard2.1", "file-uri": "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar", "freeform-tags": {}, "id": "<data-flow-application-ocid>", "language": "JAVA", "lifecycle-state": "ACTIVE", "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/", "metastore-id": null, "num-executors": 1, "owner-principal-id": "<user-ocid>", "owner-user-name": "<user-name>", "parameters": null, "private-endpoint-id": "", "spark-version": "3.0.2", "time-created": "2022-04-08T19:50:57.795000+00:00", "time-updated": "2022-04-08T19:50:57.795000+00:00", "warehouse-bucket-uri": null } }
-
Create an Application (custom) log.
- Request
-
export LOG_GROUP_ID=<log-group-ocid> oci logging log create --display-name "sample_csv_processing_application" --log-group-id $LOG_GROUP_ID --log-type CUSTOM --is-enabled true
- Response
-
{ "opc-work-request-id": "<log-workrequest-ocid>" }
-
Retrieve the log ID for the Application (service) log.
- Request
-
export LOG_WR_ID=<log-workrequest-ocid> oci logging work-request get --work-request-id $LOG_WR_ID
- Response
-
{ "data": { "compartment-id": "<compartment-ocid>", "id": "<log-workrequest-ocid>", "operation-type": "CREATE_LOG", "percent-complete": 100.0, "resources": [ { "action-type": "CREATED", "entity-type": "log", "entity-uri": "/logs/<log-ocid>", "identifier": "<log-ocid>" } ], "status": "SUCCEEDED", "time-accepted": "2022-06-13T18:04:54.587000+00:00", "time-finished": "2022-06-13T18:04:54.587000+00:00", "time-started": "2022-06-13T18:04:54.587000+00:00" } }
-
Submit a Data Flow Run.
- Request
-
export DF_APP_ID=<data-flow-application-ocid> export COMPARTMENT_ID=<tenancy-ocid> oci data-flow run create \ --compartment-id $COMPARTMENT_ID \ --display-name "Run Sample CSV Processing App with OCI Logging Diagnostic and Application Logs" \ --application-id $DF_APP_ID \ --application-log-config file://application_log.config
-
where
application_log.config
is:{ "logGroupId": "<log-group-ocid>", "logId": "<log-ocid>" }
- Response
-
{ "data": { "application-id": "<data-flow-application-ocid>", "application-log-config": { "log-group-id": "<log-group-ocid>", "log-id": "<log-ocid>" }, "archive-uri": "", "arguments": [], "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample", "compartment-id": "<compartment-ocid>", "configuration": { "spark.ui.retainedJobs": "1000000" }, "data-read-in-bytes": 0, "data-written-in-bytes": 0, "defined-tags": { "Dataflow_Resource_Category": { "category": "DEV_QA" } }, "display-name": "Run Sample CSV Processing App with OCI Logging Diagnostic and Application Logs", "driver-shape": "VM.Standard2.1", "driver-shape-config": { "memory-in-gbs": 15.0, "ocpus": 1.0 }, "execute": "", "executor-shape": "VM.Standard2.1", "executor-shape-config": { "memory-in-gbs": 15.0, "ocpus": 1.0 }, "file-uri": "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar", "freeform-tags": {}, "id": "<data-flow-run-ocid>", "language": "JAVA", "lifecycle-details": null, "lifecycle-state": "ACCEPTED", "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/", "metastore-id": "", "num-executors": 1, "opc-request-id": "1C23A27499A641CA9B30E554E20135FF/99E2C4439977AFE32FB068DC9390018F", "owner-principal-id": "<user-ocid>", "owner-user-name": "<user-name>", "parameters": [], "private-endpoint-dns-zones": null, "private-endpoint-id": null, "private-endpoint-max-host-count": null, "private-endpoint-nsg-ids": null, "private-endpoint-subnet-id": null, "run-duration-in-milliseconds": 0, "spark-version": "3.0.2", "time-created": "2022-06-13T18:14:31.630000+00:00", "time-updated": "2022-06-13T18:14:31.630000+00:00", "total-o-cpu": 2, "type": "BATCH", "warehouse-bucket-uri": "" } }
-
Search the logs of the Data Flow Run.
- Request
-
export LOG_GROUP_ID=<log-group-ocid> export LOG_ID=<log-ocid> export LOG_COMPARTMENT_ID=<compartment-ocid> export RUN_ID=<data-flow-run-ocid> export SEARCH_QUERY="search \"${LOG_COMPARTMENT_ID}/${LOG_GROUP_ID}/${LOG_ID}\" | data.runId='${RUN_ID}' and subject='spark-driver' and data.logLevel='INFO' | sort by datetime desc" export START_TIME=2022-04-08T20:20:00Z export END_TIME=2022-04-08T20:40:00Z oci logging-search search-logs --search-query $SEARCH --time-start $START_TIME --time-end $END_TIME
- Response
-
{ "data": { "fields": null, "results": [ { "data": { "datetime": 1649449421045, "logContent": { "data": { "logLevel": "INFO", "message": "Getting security token from the auth server", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "<data-flow-run-ocid>", "thread": "com.oracle.bmc.auth.internal.X509FederationClient" }, "id": "837a9de0-2bdd-4eae-ad06-1e9a78a0a23e", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:41.045Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420975, "logContent": { "data": { "logLevel": "INFO", "message": "No Progressable passed, not reporting progress.", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "<data-flow-run-ocid>", "thread": "com.oracle.bmc.hdfs.store.RequestBuilder" }, "id": "696fe929-12a7-45a7-834a-0cbf7eacf9cf", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.975Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420645, "logContent": { "data": { "logLevel": "INFO", "message": "No Progressable passed, not reporting progress.", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "<data-flow-run-ocid>", "thread": "com.oracle.bmc.hdfs.store.RequestBuilder" }, "id": "bb0fd88b-d925-457b-9194-6d5cd64d3d38", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.645Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420582, "logContent": { "data": { "logLevel": "INFO", "message": "Refreshing session keys.", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "<data-flow-run-ocid>", "thread": "com.oracle.bmc.auth.internal.X509FederationClient" }, "id": "31d028d6-0227-49c0-a94a-6e9ce78c4e81", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.582Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420504, "logContent": { "data": { "logLevel": "INFO", "message": "DynamicSslContextProviderConfig is not configured. Attempting to use tlsConfig", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "<data-flow-run-ocid>", "thread": "com.oracle.pic.telemetry.overlay.metrics.OverlayHttpClientBuilder" }, "id": "e4b361ec-700c-464e-b32d-6c79e1a46827", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.504Z", "type": "com.oraclecloud.dataflow.run.driver" } } } ], "summary": { "field-count": null, "result-count": 5 } }, "opc-next-page": "AP36-PX4-PX4_vz5_PjpjP75_vz59Pz0-P77-fj89Pz0-PWM_fr49fj49fj-_Pn8-IzMzMzM6Q==" }
-
Create a Data Flow Application.
Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging application logs in Data Flow.
-
Create a Data Flow Application.
- Request
-
oci data-flow application create \ --compartment-id $COMPARTMENT_ID \ --display-name "Sample CSV Processing App with OCI Logging Service Logs" \ --driver-shape "VM.Standard2.1" \ --executor-shape "VM.Standard2.1" \ --file-uri "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar" \ --language "JAVA" \ --num-executors 1 \ --spark-version "3.0.2" \ --class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"
- Response
-
{ "data": { "archive-uri": "", "arguments": null, "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample", "compartment-id": "<compartment-ocid>", "configuration": null, "defined-tags": { "Dataflow_Resource_Category": { "category": "DEV_QA" } }, "description": null, "display-name": "Sample CSV Processing App with OCI Logging Service Logs", "driver-shape": "VM.Standard2.1", "execute": null, "executor-shape": "VM.Standard2.1", "file-uri": "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar", "freeform-tags": {}, "id": "data-flow-application-ocid>", "language": "JAVA", "lifecycle-state": "ACTIVE", "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/", "metastore-id": null, "num-executors": 1, "owner-principal-id": "<user-ocid>", "owner-user-name": "<user-name>", "parameters": null, "private-endpoint-id": "", "spark-version": "3.0.2", "time-created": "2022-04-08T19:50:57.795000+00:00", "time-updated": "2022-04-08T19:50:57.795000+00:00", "warehouse-bucket-uri": null } }
-
Create an Application (custom) log.
- Request
-
export LOG_GROUP_ID=<log-group-ocid> oci logging log create --display-name "sample_csv_processing_application" --log-group-id $LOG_GROUP_ID --log-type CUSTOM --is-enabled true
- Response
-
{ "opc-work-request-id": "<log-workrequest-ocid" }
-
Retrieve the log ID for the Application (service) log.
- Request
-
export LOG_WR_ID=<log-workrequest-ocid oci logging work-request get --work-request-id $LOG_WR_ID
- Response
-
{ "data": { "compartment-id": "<compartment-ocid>", "id": "<log-workrequest-ocid", "operation-type": "CREATE_LOG", "percent-complete": 100.0, "resources": [ { "action-type": "CREATED", "entity-type": "log", "entity-uri": "/logs/<log-ocid>", "identifier": "<log-ocid>" } ], "status": "SUCCEEDED", "time-accepted": "2022-06-13T18:04:54.587000+00:00", "time-finished": "2022-06-13T18:04:54.587000+00:00", "time-started": "2022-06-13T18:04:54.587000+00:00" } }
-
Submit a Data Flow Run.
- Request
-
export DF_APP_ID=data-flow-application-ocid> export COMPARTMENT_ID=<tenancy-ocid> oci data-flow run create \ --compartment-id $COMPARTMENT_ID \ --display-name "Run Sample CSV Processing App with OCI Logging Diagnostic and Application Logs" \ --application-id $DF_APP_ID \ --application-log-config file://application_log.config
-
where
application_log.config
is:{ "logGroupId": "<log-group-ocid>", "logId": "<log-ocid>" }
- Response
-
{ "data": { "application-id": "data-flow-application-ocid>", "application-log-config": { "log-group-id": "<log-group-ocid>", "log-id": "<log-ocid>" }, "archive-uri": "", "arguments": [], "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample", "compartment-id": "<compartment-ocid>", "configuration": { "spark.ui.retainedJobs": "1000000" }, "data-read-in-bytes": 0, "data-written-in-bytes": 0, "defined-tags": { "Dataflow_Resource_Category": { "category": "DEV_QA" } }, "display-name": "Run Sample CSV Processing App with OCI Logging Diagnostic and Application Logs", "driver-shape": "VM.Standard2.1", "driver-shape-config": { "memory-in-gbs": 15.0, "ocpus": 1.0 }, "execute": "", "executor-shape": "VM.Standard2.1", "executor-shape-config": { "memory-in-gbs": 15.0, "ocpus": 1.0 }, "file-uri": "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar", "freeform-tags": {}, "id": "data-flow-run-ocid", "language": "JAVA", "lifecycle-details": null, "lifecycle-state": "ACCEPTED", "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/", "metastore-id": "", "num-executors": 1, "opc-request-id": "1C23A27499A641CA9B30E554E20135FF/99E2C4439977AFE32FB068DC9390018F", "owner-principal-id": "<user-ocid>", "owner-user-name": "<user-name>", "parameters": [], "private-endpoint-dns-zones": null, "private-endpoint-id": null, "private-endpoint-max-host-count": null, "private-endpoint-nsg-ids": null, "private-endpoint-subnet-id": null, "run-duration-in-milliseconds": 0, "spark-version": "3.0.2", "time-created": "2022-06-13T18:14:31.630000+00:00", "time-updated": "2022-06-13T18:14:31.630000+00:00", "total-o-cpu": 2, "type": "BATCH", "warehouse-bucket-uri": "" } }
-
Search the logs of the Data Flow Run.
- Request
-
export LOG_GROUP_ID=<log-group-ocid> export LOG_ID=<log-ocid> export LOG_COMPARTMENT_ID=<compartment-ocid> export RUN_ID=data-flow-run-ocid export SEARCH_QUERY="search \"${LOG_COMPARTMENT_ID}/${LOG_GROUP_ID}/${LOG_ID}\" | data.runId='${RUN_ID}' and subject='spark-driver' and data.logLevel='INFO' | sort by datetime desc" export START_TIME=2022-04-08T20:20:00Z export END_TIME=2022-04-08T20:40:00Z oci logging-search search-logs --search-query $SEARCH --time-start $START_TIME --time-end $END_TIME
- Response
-
{ "data": { "fields": null, "results": [ { "data": { "datetime": 1649449421045, "logContent": { "data": { "logLevel": "INFO", "message": "Getting security token from the auth server", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "data-flow-run-ocid", "thread": "com.oracle.bmc.auth.internal.X509FederationClient" }, "id": "837a9de0-2bdd-4eae-ad06-1e9a78a0a23e", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:41.045Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420975, "logContent": { "data": { "logLevel": "INFO", "message": "No Progressable passed, not reporting progress.", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "data-flow-run-ocid", "thread": "com.oracle.bmc.hdfs.store.RequestBuilder" }, "id": "696fe929-12a7-45a7-834a-0cbf7eacf9cf", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.975Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420645, "logContent": { "data": { "logLevel": "INFO", "message": "No Progressable passed, not reporting progress.", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "data-flow-run-ocid", "thread": "com.oracle.bmc.hdfs.store.RequestBuilder" }, "id": "bb0fd88b-d925-457b-9194-6d5cd64d3d38", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.645Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420582, "logContent": { "data": { "logLevel": "INFO", "message": "Refreshing session keys.", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "data-flow-run-ocid", "thread": "com.oracle.bmc.auth.internal.X509FederationClient" }, "id": "31d028d6-0227-49c0-a94a-6e9ce78c4e81", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.582Z", "type": "com.oraclecloud.dataflow.run.driver" } } }, { "data": { "datetime": 1649449420504, "logContent": { "data": { "logLevel": "INFO", "message": "DynamicSslContextProviderConfig is not configured. Attempting to use tlsConfig", "opcRequestId": "6C9C39FF02364BF19358EADD6D32C599/BF1FF4D2FFC25A70020354AEF43C075D", "runId": "data-flow-run-ocid", "thread": "com.oracle.pic.telemetry.overlay.metrics.OverlayHttpClientBuilder" }, "id": "e4b361ec-700c-464e-b32d-6c79e1a46827", "oracle": { "compartmentid": "<compartment-ocid>", "ingestedtime": "2022-04-08T20:23:42.873Z", "loggroupid": "<log-group-ocid>", "logid": "<log-ocid>", "tenantid": "<tenancy-ocid>" }, "source": "Run Sample CSV Processing App with OCI Logging Service Logs", "specversion": "1.0", "subject": "spark-driver", "time": "2022-04-08T20:23:40.504Z", "type": "com.oraclecloud.dataflow.run.driver" } } } ], "summary": { "field-count": null, "result-count": 5 } }, "opc-next-page": "AP36-PX4-PX4_vz5_PjpjP75_vz59Pz0-P77-fj89Pz0-PWM_fr49fj49fj-_Pn8-IzMzMzM6Q==" }
-
Create a Data Flow Application.