Agent Pipeline
The Agent Pipeline consists of one or more processors that apply specific rules to data that the host agent is sending. After these processing rules are applied to the data, the processed data will be sent to the Middleware endpoint by the host Agent.
We have currently agent pipeline for logs here.
Prerequisites
Middleware Kubernetes Agent must be installed on kubernetes cluster. To install the Middleware Kubernetes Agent, see our Installation Guide.
Create Log Pipeline
Let's go through how can one create the agent pipeline for logs.
Navigate to Settings > Agent Pipeline
This is showing existing pipeline and one can enable/disable pipeline using toggle, edit details and delete pipeline for respective pipeline.
Here, showing sources are the kubernetes clusters connected to the respective pipelines.
For create log Pipeline, user needs to click on Add New Pipeline.
Here, user can add details related to pipeline as shown in above image.
After adding required details, click on Create Pipeline.
Here, user can see graph view of pipelines processor and list of pipeline processors for log pipeline.
Now, Lets's talk about logs processors.
1. Attributes Processor
- Attributes Processor has capibility to replace or remove specified attributes from logs.
- It has one fields to remove specified attributes from logs.
- It has rename attribute fields where user can specify current attribute names to be renamed to new attribute names.
2. Compact Processor
- Compact Processor has the capability to remove attributes based on specified matching values.
- If an
attribute's valuein the logs matches thespecified matching values, that attribute is removed from the logs.
3. Filter Processor
- Filter Processor has the capability to filter attributes based on specified conditions.
- Users have to specify the
attribute key,condition, andvaluesto filter the data. - It has additional operators
AND&ORfor combining conditions to filter the data more comprehensively.
4. Redaction Processor
- Redaction Processor has the capability to prevent sensitive information from leaking into logs.
- It has an
Allowed Attributesfield that only allowsspecified attributesto be kept in logs. Other attributes are deleted from logs. - It has a
Regexdropdown field which has default popular regexes users can select to mask sensitive data. - It has a
Custom Regexfield that accepts custom regex name and its value to specify user-specific custom regexes to mask sensitive data.
Here, we are providing default regexes patterns so user can match their data accordingly for masking.
| Regex Name | Value |
|---|---|
SSN Number | (?:00[1-9]|0[1-9]\d|[1-5]\d{2}|6[0-5]\d|6[7-9]\d|7[0-6]\d|77[0-2]|8\d{2})-(0[1-9]|[1-9]\d)-(000[1-9]|00[1-9]\d|0[1-9]\d{2}|[1-9]\d{3})$ |
Credit Card Number | ^(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|6(?:011|5[0-9]{2})[0-9]{12}|(?:2131|1800|35d{3})d{11})$ |
Phone Number | ^(?:(?:\+|00)[1-9]\d{0,2}[- ]?)?(?:(?:\(\d{1,4}\)[- ]?)|(?:\d{1,4}[- ]?))?\d{1,4}[- ]?\d{1,4}(?:[- ]?\d{1,4})?$ |
Email Address | ^[^s@]+@[^s@]+\.[^s@]+$ |
IPv4 Address | ^(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$ |
IPv6 Address | ^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$ |
MAC Address | ^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})|([0-9a-fA-F]{4}\.[0-9a-fA-F]{4}\.[0-9a-fA-F]{4})$ |
UUID | ^[0-9a-f]{8}-[0-9a-f]{4}-[0-5][0-9a-f]{3}-[089ab][0-9a-f]{3}-[0-9a-f]{12}$ |
GUID | ^(?:\{{0,1}(?:[0-9a-fA-F]){8}-(?:[0-9a-fA-F]){4}-(?:[0-9a-fA-F]){4}-(?:[0-9a-fA-F]){4}-(?:[0-9a-fA-F]){12}\}{0,1})$ |
After adding processor, The graph looks like given below.
There is options to Add Source(Cluster), Add/Edit Log Processor, Update Pipeline if processor order changed and also a Delete Selected Node to delete selected processor.
For creating pipeline, user needs to add one or more than one processor. User can also add same processor multiple times.