Oracle Cloud Infrastructure Documentation

Ingesting Data into an OpenSearch Cluster with Fluent Bit

Ingest log data into an OpenSearch cluster with Fluent Bit.

Fluent Bit is a lightweight logging and metrics processor and forwarder.

Install and Configure Fluent Bit

  1. Download Fluent Bit and install on the VM instance. .
    To install the latest version on Linux-based instances, see Linux Packages. For other operating systems, see Supported Platforms.
  2. Run the applicable command for the VM instance's operating system to install the Fluent Bit agent.
    For example, run the following command to install the Fluent Bit agent on a Linux instance:
    sudo rpm -i <fluent-bit-rpm file>
  3. Update the OUTPUT section of the Fluent Bit configuration file, /fluent-bit/fluent-bit.conf to direct the output plugin to the OpenSearch cluster.
    For more information about this section, see Configuration File - Output.

    The following is sample code for this section of the configuration file. Replace the applicable text with the details for your cluster. Specify the OpenSearch cluster's OCID for host, see Getting an OpenSearch Cluster's Details. For http_User and http_Passwd, specify a user that has sufficient permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch. 

    name - es
host - <cluster_ID>
port - 9200
tls - on
tls.verify - on
index - <index_name>
http_User - <cluster_user>
http_Passwd - <cluster_password>
Suppress_Type_Name - On
Replace_Dots - On
Retry_Limit - 6
Buffer_Size - 400MB
  4. Update the INPUT section of the Fluent Bit configuration file to specify the source for the log data and how it should be ingested.
    For more information about this section, see Configuration File - Input.

    For example:

    Name  tail
Tag   file_log
Path  /etc/fluent-bit/project/file.log
Buffer_Max_Size  600m
Multiline  On
Parser_Firstline  multiline
Buffer_Chunk_Size  2m
Skip_Long_Lines  Off
Skip_Empty_Lines  Off
Mem_Buf_Limit  700m
DB  /etc/fluent-bit/db/file.db
DB.Sync  normal
Refresh_Interval  30
Rotate_Wait  20
Exit_On_Eof  Off
Ignore_Older  5m
storage.type  filesystem

Test Data Ingestion with Fluent Bit

This section walks through the steps to test log data ingestion with Fluent Bit based on the configuration you specified in Install and Configure Fluent Bit. Perform these steps while you're connected to the VM instance. The sample commands included are for Linux-based operating systems, for other operating systems, use the applicable commands for that operating system.
  1. Run the following commands to install Java and Python: 
    sudo yum install python3
    sudo yum install java
  2. Run the following command to create the directory for the log data: 
    mkdir /etc/fluent-bit/project
  3. Run the following command to start Fluent Bit: 
    sudo fluent-bit -c /etc/fluent-bit/fluent-bit.confb
  4. Create the python script file generate_log.py in /etc/fluent-bit/project and copy the following code into the file: 
    Import logging
 
# Configure logging to write to a file
 
logging.basicConfig(filename='file.log', level=logging.DEBUG)
 
# Generate some log messages
 
logging.debug('This is a debug message.')
 
logging.info('This is an informational message.')
 
logging.warning('This is a warning message.')
 
logging.error('This is an error message.')
 
logging.critical('This is a critical message.')
 
# Example of logging additional details using formatting
 
name = 'John'
 
age = 30
 
logging.info('User %s, age %d, logged in.', name, age)
 
# Example of logging an exception stack trace
 
try:
 
    result = 10 / 0
 
except Exception as e:
 
 logging.exception('An error occurred: %s', str(e))
  5. Use the following command to run the script created in the previous step: 
    python3 /etc/fluent-bit/project/generate_log.py

    Running this script creates a file file.log and adds the specified logging messages to the file.

After you run the python script, you can verify that the log data was ingested into the OpenSearch cluster by connecting to the cluster's OpenSearch Dashboard to check for the index you specified in the OUTPUT section of the Fluent Bit configuration file. See Task 6: Connect to OpenSearch Dashboards and Quickstart guide for OpenSearch Dashboards.