Suported Namespaces

Stack Monitoring supports the following OCI metric namespaces:

Importing an OCI Resource using the UI

Follow the steps below to import an OCI resource:

1. Within Stack Monitoring navigate to the Import Resources page and click Import resources.

2. Select OCI Native Services as the Source.

OCI Service Import Input

Import UI Work Requests

Importing an OCI Resource using the Command Line

Use the import-telemetry-resources or create OCI CLI command syntax to import OCI native resources into stack-monitoring:

• Using import command:

  oci stack-monitoring resource-task import-telemetry-resources 
          --compartment-id "<Compartment_OCID>" 
          --name <Request_name> 
          --namespace <OCI_monitoring_namespace>
          --source <source-of-resource>

• Using create command:

  oci stack-monitoring resource-task create
      --compartment-id  "<Compartment_OCID>"     
      --name <Request_name> 
      --task-details file://<JSON_INPUT_FILE>

Example JSON for importing an OCI Resource using the Command Line

{    
    "type": "IMPORT_OCI_TELEMETRY_RESOURCES",    
    "source": "<source-of-the-resource>"    
    "namespace": "<OCI-Monitoring-namespace>"
}

JSON Input Parameters:

Stack Monitoring supports monitoring for resources and metrics that have a Prometheus format. This is a two step process: configure agent to scrape and upload to telemetry and then import the resource into Stack Monitoring.

Prerequisites

Create required policies to allow importing a Prometheus-based resource within a compartment. For more information, see Create Policies for Monitoring Prometheus-based Resources.

The availability status of a Prometheus-based resource is determined as follows:

• A resource is considered up if at least one of the selected metrics under Availability Proxy Metric input has a datapoint in telemetry (including 0) during the specified interval.
• If the Availability Proxy Metric Interval is less than what is configured in the Management Agent (scheduleMins), the UI will show the resource as Not Reporting. It is recommended to enter the same value (converted to seconds) that was specified for scheduleMins in the agent configuration file.
  • For example, when configuring the Management Agent, if the value of scheduleMins is set to 5 (minutes), the availability proxy metric interval must be >= 300 (seconds, or 5 minutes). If the availability proxy metric is set to less than 300 seconds, the resource will show a status of Not Reporting.

Importing Prometheus-based Resource using the UI

• Configuring the Management Agent
• Importing Prometheus-based Resource

Importing Prometheus-based Resource using the CLI

• Configuring the Management Agent
• Importing Prometheus-based Resource

Stack Monitoring now supports monitoring Telegraf-based resources, which is a three step process:

1. Enable metric receiver on agent to make the agent ready to receive metrics from Telegraf third party collectors.
2. Configure Telegraf to upload metrics to the metric receiver listening on the agent, which in turn upload metrics to telemetry.
3. Import Telegraf resources from telemetry to Stack Monitoring.

Prerequisites

• Minimum Version of Management Agent: 250214.1645
• Telegraf is configured and receives metric information for various targets.

• Create required policies to allow importing a Telegraf-based resource within a compartment. For more information, see Create Policies for Monitoring Telegraf-based Resources.

Enable metric receiver and handler for accepting Telegraf metrics

Under Import Resources menu, select Import Resources, at the top left corner of the table. Once the inputs are filled, click on Import Resources. The panel will close and the newly created work request will appear on the list. As part of the work request, receiver will be set to start listening on the agent for the metrics posted by Telegraf. As part of enablement agent will generate a certificate at agent directory <agent_state_directory>/extension_state/StackMonitoringMetricReceiver/certs/clientCerts/stackmonitoring-ca.crt, which needs to be copied to a path which is accessible by Telegraf.

Select the Status link for the work request to see the logs.

At the end of work request completion, two new resources will be created in Stack Monitoring

1. Receiver Resource:
   • Name: <agent_name>_receiver
   • Type: stackmon_receiver
2. Telegraf handler resource:
   • Name: <agent_name>_handler
   • Type: stackmon_handler

These resources will have related handler metrics, rejected metrics count etc. from telegraf

As part of the work request an entry will be made for the scheduler to import telegraf-based resources from telemetry for this agent every 24 hours even if the user does not perform Import Resources manually.

Default metric namespace for Telegraf-based resources is oracle_appmgmt_telegraf.

Configure Telegraf

Add receiver URL for uploading metrics and certificate path which was generated by the receiver on enablement. These configurations needs to be added in telegraf.conf.

[[outputs.http]]
## URL is the address to send metrics to
url = "https://<agent-host-name>:3872/receiver/telegraf?id=<telegraf_instance_id>"
 
#   TLS Config
insecure_skip_verify = true
tls_ca="/home/mcollector/sar_test/certs/stackmonitoring-ca.crt"

Stack Monitoring now supports monitoring collectd-based resources, which is a three step process:

1. Enable metric receiver on agent to make the agent ready to receive metrics from collectd third party collectors.
2. Configure collectd to upload metrics to the metric receiver listening on the agent, which in turn upload metrics to telemetry.
3. Import collectd resources from telemetry to Stack Monitoring.

Prerequisites

• Minimum Version of Management Agent: 250214.1645
• CollectD is configured and receives metric information for various targets.
• Create required policies to allow importing a collectd-based resource within a compartment. For more information, see Create Policies for Monitoring collectd-based Resources.

Enable metric receiver and handler for accepting collectd metrics

Under Import Resources menu, select Import Resources, at the top left corner of the table. Once the inputs are filled, click on Import Resources. The panel will close and the newly created work request will appear on the list. As part of the work request, receiver will be set to start listening on the agent for the metrics posted by collectd. As part of enablement agent will generate a certificate at agent directory <agent_state_directory>/extension_state/StackMonitoringMetricReceiver/certs/clientCerts/stackmonitoring-ca.crt, which needs to be copied to a path which is accessible by collectd.

Select the Status link for the work request to see the logs.

At the end of work request completion, two new resources will be created in Stack Monitoring

1. Receiver Resource:
   • Name: <agent_name>_receiver
   • Type: stackmon_receiver
2. Telegraf handler resource:
   • Name: <agent_name>_handler
   • Type: stackmon_handler

These resources will have related handler metrics, rejected metrics count etc. from collectD

As part of the work request an entry will be made for the scheduler to import collectd-based resources from telemetry for this agent every 24 hours even if the user does not perform Import Resources manually.

Default metric namespace for collectd-based resources is oracle_appmgmt_collectd.

Configure collectd

Add receiver URL for uploading metrics and certificate path which was generated by the receiver on enablement. These configurations needs to be added in collectd.conf.

LoadPlugin write_http
<Plugin write_http>
      <Node "example">
                URL "https://<agent-host-name>:3872/receiver/collectd?id=<collectd_instance_id>"
                CACert "/home/mcollector/sar_test/certs/stackmonitoring-ca.crt"
                VerifyHost false
                VerifyPeer false
                Header "Content-Type: application/json; charset=utf-8"
                Format "JSON"
                Metrics true
        </Node>
</Plugin>

Associating an imported resources with existing resources

To associate an imported resource, to another resource within Stack Monitoring, follow the steps below:

1. From the Resource home page, click on Topology.
2. Click on the Add association button.
3. In the Add related resources popup, fill-in the fields:
   1. Relation - Select on of the two associations possible : Uses and Used by. The selected Related resource will be source or destination accordingly. If the association is Uses then the related resource becomes the destination in the association pair. If the association is Used by then the related resource becomes the source in the association pair
   2. Resource type - Select the resource type you want to associate.
   3. Resource - Select the resource to be associated. Based on the resource type selected, the Resource list will be populated with available option.
4. Click Add resources

Once the association is added, the associated resource will show in the Topology table. If an incorrect association was created, click the delete icon to remove the relationship.

To create associations using CLI, see Application Topology.

Availability Status Calculation for Imported Resource

The availability status of an imported resource is determined as follows:

• A resource is considered up (available) if there is metric data for the OCI Service. It is considered down if there is no metric data.
• If an OCI Service that is down continues to emit metric data to OCI Monitoring, then its Lifecycle state attribute will be used to determine its availability status.

In other cases, such as MySql DB system, the availability status is calculated based on the presence of a predetermined metric. If the metric is present in the OCI monitoring namespace during the given time period, the imported resource is considered Up. If the metric data is absent during this period, the resource is considered down.

Viewing the OCI Resource in Stack Monitoring Enterprise Summary

Stack Monitoring allows you to monitor the availability status and performance metrics of the imported OCI resources in Enterprise Summary.

For more information on monitoring the status of your resources in Enterprise Summary, see Monitor the Status and Performance of Your Enterprise.

You can monitor any custom resource, such as an application or application infrastructure, running on a host by identifying the processes that make up the resource and the hosts on which the resource is running. Once defined, Stack Monitoring creates a new resource instance for your custom resource which you can monitor using a built-in homepage. If this custom resource is used by an application monitored in Stack Monitoring, you can easily create an association between this custom resource and the application, therefore enriching your application topology.

The following metrics are collected for each Process Set that corresponds to your custom resource. These metrics are available as an aggregate over the entire process set, but also separately for each identified process name and owner, or label, if provided.

• Monitoring status
• Memory used (physical/virtual)
• Memory utilization
• CPU usage
• CPU utilization
• Total number of processes

To identify the specific set of processes that make up the custom resource, define a Process Set. When defining a Process Set, specify filters that consist of either a process name, process owner, or process line regular expression, which are then used to match the resource's running processes. Each filter needs to have at least one of the fields present. It is possible to add an optional label that serves as a symbolic reference to the processes matched by the filter. For example, to monitor the nginx application server as custom resource, create a Process Set called nginx processes that has a filter specifying the process name nginx, with the process line regular expression nginx: main, and a label nginx main process.

Processes used as input

The following command-line command is used on Linux to retrieve the processes that are used:

ps -ww -e -o comm:32,user:32,args

Piping the output from this command to a search tool such as grep allows you to isolate the processes you're interested in, and determine how to construct the Process Set. The following command shows process lines that contain the keyword nginx in them.

ps -ww -e -o comm:32,user:32,args | grep nginx

Executing this command outputs the following table, with three columns called: COMMAND, USER, and again COMMAND. The mapping from the table columns to the fields of a Process Set filter are as follows. The processCommand field of the filter is applied to the first COMMAND column, the processUser field is applied to the USER column, and the regular expression is applied to the last COMMAND column. If one of the fields is not provided, the column is ignored for that specific filter.

Suppose you wish to monitor the master and worker processes separately, the Process Set payload could look as follows. The filter with label nginx-main-process matches the first process in the table, the filter with label nginx-worker-processes the others.

{
    "compartmentId": "<Compartment_OCID>",
    "displayName": "nginx processes",
    "specification": {
        "items": [
            {
                "label": "nginx-main-process",
                "processCommand": "nginx",
                "processLineRegexPattern": "nginx: master.*"
            },
            {
                "label": "nginx-worker-processes",
                "processCommand": "nginx",
                "processLineRegexPattern": "nginx: worker.*"
            }
        ]
    }
}

The same can be achieved via the Monitor processes page, by creating a Process Set with 2 filters.

Create a Process Set Using the UI

A Process Set represents a specification for identifying processes running on a host that make up a custom resource, and consists of one or more filters that specify a set of processes running on a host.

A host process gets matched to a Process Set if it meets the criteria defined in one of the filters of the Process Set.

1. From the Stack Monitoring menu, select Monitor processes

2. Click Create Process Set. The Create Process Set UI panel displays.

3. Complete the required fields, for a detailed description of each field see UI Discovery Input.
4. Create and map or Create the Process Set:
   • Create and map allows you to create the Process Set and subsequently starts monitoring the selected hosts, in a single step.
   • Create creates the Process Set without mapping it to any hosts.

Updating a process set

1. From the Stack Monitoring menu, select Monitor processes

2. Choose the Process Set from the list of Process Sets.

3. Make the necessary adjustments to the Process Set properties or specification.
4. Select additional hosts to monitor under Target hosts.
5. Save changes will:
   1. Update the Process Set on existing process-based custom resources.
   2. Map the updated Process Set to the selected hosts.

Creating a Process Set Using OCI-CLI

A Process Set represents a specification for identifying processes running on a host that make up a custom resource, and consists of one or more filters that specify a set of processes running on a host.

A host process gets matched to a Process Set if it meets the criteria defined in one of the filters of the Process Set.

A Process Set is created using an OCI-CLI command with a JSON payload. The JSON file contains a list of filter definitions, which is constructed using strings and regular expressions (regex) to filter out the processes that you're interested in. A Process Set is then defined using the OCI-CLI command: oci stack-monitoring process-set.Once you've created the Process Set, you can then run OCI-CLI commands that perform p

Once you've created the Process Set, you can then run OCI-CLI commands that perform process-based custom resource setup and configuration operations.

The payload of a Process Set has the following mandatory fields:

• compartmentId - the OCID of the compartment
• displayName - a symbolic name for the Process Set
• specification - a list of filters.

Each filter item has the following optional fields. However, at least one of processCommand, processUser, and processLineRegexPattern must be specified. Filters are used to select which processes are part of the process set. A process must match all specified fields of one filter to be selected.

• label is a descriptive label for processes matching this filter.
• processCommand specifies the command name. Only processes whose command name exactly matches this value are included. If this value is not specified, no filtering on command name is done.
• processUser specifies the name of the user running the process. Only processes whose user name exactly matches this value are included. If this value is not specified, no filtering on invoking user is done.

• processLineRegexPattern specifies a regex pattern which is matched against the processes command line. The command line typically includes both the command name and any arguments passed when starting the process. If the regex matches any substring of the command line (using javascript regex rules), the process is included. If this regex is not specified, no filtering on command line is done (but filtering on command name is still done, if specified).

  Note
  
  When a process matches multiple filters (for example, because the regular expressions are overlapping), it will only be matched against on the first matching filter.
  Note
  
  To report the status of the resource as up, at least one process should be matched on each filter item.

Create a process set using the one of the below nginx payloads using the following OCI-CLI create command.

Example of a CLI create command:

oci stack-monitoring process-set create --compartment-id "<Compartment_OCID>" --from-json file://<JSON_INPUT_FILE>

• {
      "compartmentId": "<Compartment_OCID>",
      "displayName": "nginx processes",
      "specification": {
          "items": [
              {
                  "processCommand": "nginx",
              }
          ]
      }
  }

  This will monitor if at least one process is up, but it won't be checking if all required processes from nginx are up, nor have metrics for those specifically (only total summed on the nginx process Command dimension).

• {
      "compartmentId": "<Compartment_OCID>",
      "displayName": "nginx processes",
      "specification": {
          "items": [
              {
                  "processCommand": "nginx",
                  "processLineRegexPattern": "nginx: master."
              },
              {
                  "processCommand": "nginx",
                  "processLineRegexPattern": "nginx: worker."
              }
          ]
      }
  }

  This will make sure all the processes required for nginx are monitored, and the MonitoringStatus is only up when there's the master process and at least one worker. The amount of granularity in metrics is the same as in the first example, as the dimensions ProcessCommand is the same and the regex is not reported as a dimension in metrics.

• {
      "compartmentId": "<Compartment_OCID>",
      "displayName": "nginx processes",
      "specification": {
          "items": [
              {
                  "label": "nginx-main-process",
                  "processCommand": "nginx",
                  "processLineRegexPattern": "nginx: master."
              },
              {
                  "label": "nginx-worker-processes",
                  "processCommand": "nginx",
                  "processLineRegexPattern": "nginx: worker."
              }
          ]
      }
  }

  Adding a label will allow the user to differentiate the master and worker processes in the metrics.

  Since in the nginx case the program is always nginx, we add a label to differentiate workers and master (as regex nor complete process line are being reported as dimensions).

Note that in the examples above the process user field processUser is omited to match on all processes regardless of which user owns the process.

The response contains the Process Set, and the property id will contain the process_set_id.

Identify the Hosts on Which the Custom Resource Should be Monitored

Now that a Process Set has successfully been created a custom resource can be created, to monitor this process set on a specific host. This is done by starting a Stack Monitoring Discovery Job, and creating a resource of type CUSTOM_RESOURCE, via the Stack Monitoring DiscoveryJobs API. See the following payload for an example.

In this example, specify the agent and host on which the custom resource will be monitored.

In the JSON payload, the license must be ENTERPRISE_EDITION.

{
    "discoveryType": "ADD",
    "discoveryClient": "APPMGMT",
    "compartmentId": "<Compartment_OCID>",
    "discoveryDetails": {
        "agentId": "<Agent_OCID>",
        "resourceType": "CUSTOM_RESOURCE",
        "resourceName": "nginx processes",
        "properties": {
            "propertiesMap": {
                 "host_ocid": "<Host_OCID>",
                 "process_set_id": "<ProcessSet_OCID>"
            }
        },
        "license": "ENTERPRISE_EDITION"
    },
    "freeformTags": {},
    "definedTags": {}
}

Once you've defined the JSON file, run the following command to create the custom resource discovery job:

oci stack-monitoring discovery-job create --compartment-id "<Compartment_OCID>" --from-json file://<JSON_INPUT_FILE>

Once the Discovery Job has completed, a Stack Monitoring resource of type CUSTOM_RESOURCE named nginx processes and its corresponding homepage will be available.

Updating a process set

Updating a Process Set is similar to creating one. Determine the OCID of the Process Set you wish to update and perform an update using a payload structured similarly to the create payload. Once the update has been completed you will notice that the revision of the Process Set has been incremented. This is used to track which version of the Process Set specification is currently present on a monitored Host. Note that updating a Process Set via the OCI-CLI or via a REST call does not update the specification used to monitor the processes on a host.

Update a Process Ser using the following CLI command:

oci stack-monitoring process-set update --process-set-id <ProcessSet_OCID> --from-json file://<JSON_INPUT_FILE>

Using a raw request or a PUT request to /processSets/<ProcessSet_OCID>:

{
    "compartmentId": "<Compartment_OCID>",
    "displayName": "nginx processes",
    "specification": {
        "items": [
            {
                "processCommand": "nginx",
                "label": "all nginx processes filter"
            }
        ]
    }
}

Refresh a Custom Resource

When a process set is changed, update the process set on a custom resource using a REFRESH Discovery Job command:

oci stack-monitoring discovery-job create  --compartment-id "<Compartment_OCID>" --from-json file://<JSON_INPUT_FILE>

REFRESH JSON payload example:

{
    "discoveryType": "REFRESH",
    "discoveryClient": "APPMGMT",
    "compartmentId": "<Compartment_OCID>",
    "discoveryDetails": {
        "agentId": "<Agent_OCID>",
        "resourceType": "CUSTOM_RESOURCE",
        "resourceName": "nginx processes",
        "properties": {
            "propertiesMap": {
                 "resource_id": "<Resource_OCID>",
                 "host_ocid": "<Host_OCID>",
                 "process_set_id": "<ProcessSet_OCID>"
            }
        },
        "license": "ENTERPRISE_EDITION"
    }
}

Checking the Status of the Discovery Job

Once the command has been executed, the job's status and logs can be viewed within Stack Monitoring under the Resource Discovery page. From the Resource Discovery page, search for the submitted job with a Resource Name matching the name of the Custom Resource, a Resource Type of Custom Resource, and with a job type of Add. Once the Job Status has Succeeded, click the name of the Custom Resource under Resource Name to navigate to the resource’s homepage.