Ingestion Controls
The Ingestion Controls section is where users can tailor their log tracking infrastructure. Easily toggle your Middleware Agent logs, metrics, and trace ingestion on and off. The Middleware Host Agent will search for changes in your account configuration and automatically update the configuration YAML file and reload itself.
Fine-tune your log collection process by selecting your preferred hosts for tracking infrastructure file-based logs, Middleware Agent Logs, and specify custom directory paths for monitoring your log collection.
Infrastructure File-Based Logs
Quickly enable OR disable the Middleware Agent from monitoring file-based logs for collection. Infrastructure filed-based logs are collected by default.
To disable the collection of file-based logs, navigate to Settings, select Ingestion Controls, and press the Infrastructure File-based Logs radio button.
Middleware Agent Logs
Quickly enable or disable Middleware Agent log collection. Middleware Agent pod-logs are generated by the Middleware Agent.
Middleware Agent Log Paths
You can dynamically select the log files you want to monitor. Add directory paths in the Middleware Agent Log Paths section.
Below is the accepted regex pattern:
'^(\\/(?:[\\w_\\-^* ]+\\/)*(?:[\\w_\\-*]+(?:\\**\\/)?)?[\\w_\\-.]+[^.]$)'
The follwing are examples of available paths:
/home/ubuntu/abc.log
/home/ubuntu/*.log
/home/ubuntu/**/*.log
Custom Attribute Controls
Middleware supports standard regex rules to allow extraction of attributes and their values from log messages. The attributes scraped will be available in filters, search bars, and expanded log attributes.
Log Extraction Example
Grouping table. Using the below log message as an example, we are going to extract four key attributes: date, process_id, level, and content.
[2024-02-26 09:15:41 -0500] [671097] [ERROR] Shutting down: Master
We can extract the desired attributes with the below regex:
\[(?P<date>.+)\] \[(?P<process_id>.+)\] \[(?P<level>.*)\] (?P<content>.+)
Given the original log message and above regex, we receive the following key and value pairs:
date = 2024-02-26 09:15:41 -0500
process_id = 671097
level = ERROR
content =Shutting down: Master
The following image shows how this looks from within the Middleware Application:
level, which accepts ERROR, WARN, INFO, DEBUG as valid values. Based on the level attribute, logs are available under specific severity levels. Sensitive Data Masking for Logs
Middleware provides users the ability to mask data at time of ingestion. This means that all data will be masked before it leaves your systems, allowing full control over security regulations and company policies.
Similar to the above attribute extraction, Data Masking is also based on standard regex rules.
The setting will expect 2 inputs from users as shown in the image below:
- A Regex Pattern for the keywords that users want to hide
- A Replacement Value, e.g.
*****,HIDDEN, etc.
If your log content looks like: Transaction ID: 675-98-3434
\\d{3}-?\\d{2}-?\\d{4} modifies the log before sending: Transaction ID: *****
Metrics Ingestion
By default, MW Agent metrics are automatically ingested. To disable this feature, press the radio button and click Save. Disabling metrics will limit your overall understanding of your infrastructure vitals and performance.
Trace Ingestion
By default, MW Agent traces are automatically ingested. To disable this feature, press the radio button and click Save. Disabling traces will limit your ability to isolate issues within your infrastructure.
Was this page helpful?