Agent Pipeline

The Agent Pipeline consists of one or more processors that apply specific rules to data that the host agent is sending. After these processing rules are applied to the data, the processed data will be sent to the Middleware endpoint by the host Agent.

We have currently agent pipeline for logs here.

Prerequisites

Middleware Kubernetes Agent must be installed on kubernetes cluster. To install the Middleware Kubernetes Agent, see our Installation Guide.

Create Log Pipeline

Let's go through how can one create the agent pipeline for logs.

Navigate to Settings > Agent Pipeline

Middleware Agent Pipeline configuration interface

This is showing existing pipeline and one can enable/disable pipeline using toggle, edit details and delete pipeline for respective pipeline.

Here, showing sources are the kubernetes clusters connected to the respective pipelines.

For create log Pipeline, user needs to click on Add New Pipeline.

Existing Agent pipelines and connected Kubernetes sources

Here, user can add details related to pipeline as shown in above image.

After adding required details, click on Create Pipeline.

Form to add details for creating a new Middleware Agent pipeline

Here, user can see graph view of pipelines processor and list of pipeline processors for log pipeline.

Now, Lets's talk about logs processors.

1. Attributes Processor

  • Attributes Processor has capibility to replace or remove specified attributes from logs.
  • It has one fields to remove specified attributes from logs.
  • It has rename attribute fields where user can specify current attribute names to be renamed to new attribute names.
Middleware Agent pipeline attributes processor configuration for log data

2. Compact Processor

  • Compact Processor has the capability to remove attributes based on specified matching values.
  • If an attribute's value in the logs matches the specified matching values, that attribute is removed from the logs.
Middleware Agent pipeline compact processor configuration for log data

3. Filter Processor

  • Filter Processor has the capability to filter attributes based on specified conditions.
  • Users have to specify the attribute key, condition, and values to filter the data.
  • It has additional operators AND & OR for combining conditions to filter the data more comprehensively.
Middleware Agent pipeline filter processor configuration for log data

4. Redaction Processor

  • Redaction Processor has the capability to prevent sensitive information from leaking into logs.
  • It has an Allowed Attributes field that only allows specified attributes to be kept in logs. Other attributes are deleted from logs.
  • It has a Regex dropdown field which has default popular regexes users can select to mask sensitive data.
  • It has a Custom Regex field that accepts custom regex name and its value to specify user-specific custom regexes to mask sensitive data.
Middleware Agent pipeline redaction processor configuration for log data

Here, we are providing default regexes patterns so user can match their data accordingly for masking.

Regex NameValue
SSN Number(?:00[1-9]|0[1-9]\d|[1-5]\d{2}|6[0-5]\d|6[7-9]\d|7[0-6]\d|77[0-2]|8\d{2})-(0[1-9]|[1-9]\d)-(000[1-9]|00[1-9]\d|0[1-9]\d{2}|[1-9]\d{3})$
Credit Card Number^(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|6(?:011|5[0-9]{2})[0-9]{12}|(?:2131|1800|35d{3})d{11})$
Phone Number^(?:(?:\+|00)[1-9]\d{0,2}[- ]?)?(?:(?:\(\d{1,4}\)[- ]?)|(?:\d{1,4}[- ]?))?\d{1,4}[- ]?\d{1,4}(?:[- ]?\d{1,4})?$
Email Address^[^s@]+@[^s@]+\.[^s@]+$
IPv4 Address^(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$
IPv6 Address^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))$
MAC Address^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})|([0-9a-fA-F]{4}\.[0-9a-fA-F]{4}\.[0-9a-fA-F]{4})$
UUID^[0-9a-f]{8}-[0-9a-f]{4}-[0-5][0-9a-f]{3}-[089ab][0-9a-f]{3}-[0-9a-f]{12}$
GUID^(?:\{{0,1}(?:[0-9a-fA-F]){8}-(?:[0-9a-fA-F]){4}-(?:[0-9a-fA-F]){4}-(?:[0-9a-fA-F]){4}-(?:[0-9a-fA-F]){12}\}{0,1})$

After adding processor, The graph looks like given below.

Middleware Agent pipeline flowchart diagram

There is options to Add Source(Cluster), Add/Edit Log Processor, Update Pipeline if processor order changed and also a Delete Selected Node to delete selected processor.

For creating pipeline, user needs to add one or more than one processor. User can also add same processor multiple times.