Manage Storage

The log data ingested into Oracle Logging Analytics is available in the active storage for analysis. You can perform the following storage related activities based on your need:

  • Archive Logs: If you want to use your old logs for analysis in the future, then enable archiving and specify the number of days from the log's timestamp after which the log data must be automatically moved from active storage to archive storage which is available at a lesser cost. You can also recall the archived log data for active use. See Archive Log Data.

    • Recall Archived Logs: After the log data is archived, you can recall the selected log data for active use. The logs are selected for recall by specifying the time range in which the timestamps of the logs are present. You can release the recalled logs back to the archive pool after active use. Note that the recalled data will count towards your active storage usage until you release it. See Recall Archived Logs.

    • Release Recalled Logs: Use this option for releasing the recalled logs back into the archive storage to optimize your storage cost. See step 8 in Recall Archived Logs.

  • Purge Logs: You can purge the unused or old log data to reduce the size of the active storage that you are consuming. You can perform purge on-demand or create a purge policy. See Purge Log Data.

  • View Storage Activity Report: Use this single-pane window to keep track of all your storage management activities and to perform more management tasks. See View Storage Activity Report.

Note

Your archive policy and recall activity may not complete if the time lines overlap with the purge policy. Make sure to review your purge policy and archival setting to avoid losing log data that must be archived.

Archive Log Data

If you're using only the recent logs for your search and analysis tasks in Oracle Logging Analytics, then enable archiving so that you can optimize the storage cost.

Note

  • You can enable archiving only after you have the minimum specified size of data in active storage. Currently, this is 1 TB.

  • The minimum Active Storage Duration (Days) for logs before they can be archived is 30 days.

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  3. Click Enable Archiving. In the Enable Archiving dialog box, enter the count of the days after which the log data in the active storage must be archived in the field Active Storage Duration (Days), and click Enable.

    The count is calculated based on the timestamp of the logs. For example, if your logs have the timestamp November 4, 2020 23:43:12, and you've specified the Active Storage Duration as 30, then the logs will be typically moved to archive storage on December 3, 2020.

    Note

    It must be noted that even if you specify the Active Storage Duration of the logs to determine the logs that must be moved to Archive storage, the log index structure is based on the buckets that are used for storing the logs. In a typical scenario, an entire bucket is moved to the archive storage when all the logs in it are older than the specified criterion.

    For example, consider that the field Active Storage Duration is set to 30 days:

    • Bucket_1 has logs of age 40 - 80 days: The log data is eligible and is moved to archive storage.
    • Bucket_2 has logs of age 25 - 40 days: Although some of the log data is eligible for archiving, it is not archived until all the logs are suitable for the specified age.
    • Bucket_3 has logs of age 0 - 25 days: None of the logs are suitable for archiving. The entire bucket is archived when all the logs become eligible.

    In the above scenario, after Bucket_1 logs are archived, if more logs are collected which are older than 40 days, then they are typically appended to Bucket_2.

  4. If you have enabled archiving already, and want to modify the archiving settings, then click Modify Archiving Settings. You can perform any of the following tasks:

    • You can change the value of the count of the days specified for archiving under Active Storage Duration (Days).
    • Click Disable Archiving to stop archiving.

    Click Save Changes.

Recall Archived Logs

If you want to use the logs that are archived for viewing and analysis, then you can recall the logs. The recalled data will count towards your active storage usage until you release it.

You can recall and release your selected set of logs multiple times. However, the recall feature is enabled only if you already have archived logs.

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  3. In the Storage page, on the left panel under Resources, click Archiving Recall Requests.

    The Archiving Recall Requests page displays the previously initiated recall requests.

  4. Click Create Recall Request. The Create Recall Request dialog box opens.

  5. Specify the Purpose of recall. This can help you to identify your recall request.

  6. Optionally, if you have defined log set, then you can specify one or more Log Sets to filter the recalled data. To specify multiple log sets, use comma separation.

  7. Select the time range of the logs that you want to recall, by specifying the User-defined start time and User-defined end time.

  8. Click Estimate Recall Log Size. The Data set recommended for analysis section opens. The size of the logs that you've selected for recall is displayed adjacent to the heading Maximum recalled data size before filtering.

    Note that the start time and end time are extended to align with the log index structure based on buckets. So, when you view the list of active recalls or visit the activity tab, you may get the start and end time extended beyond your chosen time range.

    If your current recall time specifications overlap with another recall activity, then they can possibly get merged into a single recall activity and the resulting start and end time can get extended.

  9. An alternative time range is recommended based on the availability of data. To select the time range you specified earlier instead of the recommended time range, enable the check box Do not use recommended data set for recall

  10. Specify the Query to filter the data set. Exclude the time and log set from the query.

    Note that applying the filter does not impact the size of the data set estimated based on the time range.

  11. Click Create Recall Request to proceed with the recall of the selected logs.

    The recall activity is listed in the Archiving Recall Requests page. The table specifies the status, time range, data size, and request date and time of recall activity, user who initiated the recall, and the purpose of recall. The individual recalls that have overlapping data are combined into a single collection. In such cases, the table displays the data size of the collection instead of the data size of the underlying recalls.

    Note

    If you keep the recommended and default data set for each recall, then the collection time range is the super set of the time ranges of the individual recalls. Otherwise, the collection time range may not be the super set of the individual recall time ranges.

    Watch the status of the recall activity. You can use the recalled logs for viewing and analysis after the recall activity is complete.

    If the data size icon for a collection is displayed in orange, then new additional log data is available for recall. Click the data icon new data icon and click Recall new data to initiate the recall of the new data. The Recall new data dialog box opens. The query to filter the data set and the time range for data recall are predefined. Specify the purpose of recall and click Create Recall Request.

  12. After active use of the recalled logs, if you want to release them back to the archive pool, click the actions menu icon Actions icon in the row corresponding to your recalled logs, and select Release.

    The recalled logs will then be released back into the archive pool. This will enable you to optimize your storage size and cost.

    Note

    When releasing the recalled logs using REST API, note the recall time range from console or CLI, and format the time as follows:

    • Recall start time: Round down (floor) the value. If the recall start time is From Mon, Mar 7, 2022, 05:45:33 UTC, then round down the time and specify it as from_time=2022-03-07T5:45:32.000Z.
    • Recall end time: Round up (ceil) the value. If the recall end time is To Wed, Mar 15, 2023, 17:26:53 UTC, then round up the time and specify it as to_time=2023-03-15T17:26:54.000Z.

Purge Log Data

Oracle Logging Analytics lets you purge log events that were loaded by agent or by an on-demand upload, to reduce the index size of the log data.

Purging enables you to bring down your usage to reduce overage charges. Oracle Logging Analytics can purge log data automatically per a set schedule or manually based on your need. Before you purge log data, create IAM policies to set up permissions for the task. See Allow Users to Purge Log Data.

There are multiple ways to purge log data.

  • By purging on-demand: All log data from the specified compartment created prior to the selected time range gets purged.
  • By creating a purge policy: The old log data can be purged by specifying a schedule for purging and the query to filter the data to purge. If you want to automate the purge activity, then you can create a purge policy by specifying the purge schedule, selecting the log data to purge, and enabling the policy.
  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

    The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  2. In the Storage page, you can purge log data in one of the following methods:
    • Perform on-demand Purge:

      Click Purge Logs. The Purge Logs dialog box is displayed.

      • Select the compartment in which the logs must be purged.

      • Specify if the subcompartments also must be included in the Purge.

      • Select the date and time prior to which the log data that must be purged was collected.

        Purge action is performed on the log data from all the buckets in the selected compartment which were collected prior to the specified time period. For example, if you specify the date and time as November 2, 2020 12:00:00 and compartment Analyze, then the log data with the time stamp older than November 2, 2020 12:00:00 stored in the compartment Analyze is deleted.

      • In the Query field, enter the query to select a specific set of log data. For example, to select the logs from the entities of the type Linux Host, specify the query 'Entity Type'='Host (Linux)'.

      • Click Estimate Reclaimed Storage to determine the size of the storage that can be reclaimed based on the selection you made in the previous fields.

      • Click Purge.

    • Create a purge policy to purge logs based on a query or age:

      Under Purge Policies, click Create. The Create Purge Policy dialog box opens.

      • Enter a name for the new purge policy.

      • Select the log group compartment to query for the logs. You can optionally specify if the subcompartments must also be queried for the specified logs.

      • Under Purge Logs Older than, select the time period from when the log data must be purged.

      • Under Schedule Interval, select the periodicity, and time of the purge action.

      • In the Query field, enter the query to select a specific set of log data. For example, to select the logs from the source Apache HTTP Server Access Logs, specify the query 'Log Source'='Apache HTTP Server Access Logs'.

      • Click Estimate Reclaimed Storage to determine the size of the storage that would be reclaimed if the selections you made in the previous fields were to be applied now.

      • Optionally, click Show Advanced Options and add tags to your purge policy.

      • Click Create.

      The purge policy is created and will be run periodically as set in the previous steps.

      Note

      If a purge policy is paused due to lack of permissions, after modifying the policy statements as required, manually Resume the purge task.

    To delete a policy, click Actions icon Actions icon next to the policy name, and click Delete.

    To view the purge activities performed, in the Storage page, under Resources, click Activity Report. The Activity Report page is displayed which summarizes all the storage activities. Use the Status and Time filters to view the preferred purge activities.

Allow Users to Purge Log Data

To purge log data, first set up right permissions by creating the following IAM policies:

  1. Create a dynamic group to allow purges for the compartments you want to allow purges in:

    ALL {resource.type='loganalyticsscheduledtask', resource.compartment.id='<compartment ocid>'}

    Alternatively, to allow purges on all compartments:

    ALL {resource.type='loganalyticsscheduledtask'}
  2. Create policies to allow the dynamic group to perform purge operation:

    allow dynamic-group <group_name> to read compartments in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_PURGE} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERY_VIEW} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ} in tenancy
    Note

    • For the proper functioning of the purge policy, the permissions read compartments, LOG_ANALYTICS_STORAGE_PURGE, and LOG_ANALYTICS_QUERY_VIEW must be created at tenancy level. To restrict the purge action permission to specific compartments, the permissions LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE, LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS, and LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ can be set at the required compartment level.

    • In the above policy statements involving dynamic group, if the dynamic group is in a domain other than Default, then the policy statement must be of the format:

      allow dynamic-group '<domain>'/'<group_name>' to ...

      Enclose the domain name and dynamic group name in single quotes.

  3. Additionally, ensure that the user has MANAGE permission on loganalytics-features-family and loganalytics-resources-family. If the user creating the on-demand or scheduled purge has Administrator privileges, then the required permissions are already available:

    allow group <group_name> to MANAGE loganalytics-features-family in tenancy
    allow group <group_name> to MANAGE loganalytics-resources-family in tenancy

Some of the above policy statements are included in the readily available Oracle-defined policy templates. You may want to consider using the template for your use case. See Oracle-defined Policy Templates for Common Use Cases.

For information about dynamic groups and IAM policies, see OCI Documentation: Managing Dynamic Groups and OCI Documentation: Managing Policies.

View Storage Activity Report

You can view the summary of your archive, recall, release, and purge activities to maintain close control of your storage use and also to track the status of your key logs that have been part of the activities.

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  3. In the left panel under Resources, click the Activity Report.

    The page displays the summary of the storage activities initiated such as purge policy, purge on demand, archiving, archiving recall request and recall release.

  4. Use the Activity Type, Status, and Time filters on the left panel to narrow down your search for the storage activities.

  5. Expand the storage activity row to view more details about it.