Data Ingestions API
Data Ingestion API - Setup & Installation Docs | Middleware
This guide provides an overview and setup instructions for Middleware’s OpenTelemetry Data Ingestion API. The OpenTelemetry Ingestion API has two endpoints: Metrics and Logs.
Resource Types
For both endpoints, the resource type attribute groups the ingested data under the specified label on Middleware dashboards and reports. The resource type field requires one of the following options.
| Group | Resource Type |
|---|---|
host | host.id |
k8s.node | k8s.node.uid |
k8s.pod | k8s.pod.uid |
k8s.deployment | k8s.deployment.uid |
k8s.daemonset | k8s.daemonset.uid |
k8s.replicaset | k8s.replicaset.uid |
k8s.statefulset | k8s.statefulset.uid |
k8s.namespace | k8s.namespace.uid |
service | service.name |
os | os.type |
custom | custom |
Metrics Endpoint
The Metrics endpoint lets you send custom metrics to the Middleware backend.
To send custom metrics to Middleware, POST to the following endpoint.
POST https://<UID>.middleware.io/v1/metrics
View the following example of a curl request sending a custom metric, swap-usage, to Middleware.
curl -X POST "https://demo.middleware.io/v1/metrics" \
-H "Accept: application.json" \
-H "Content-type: application.json" \
-d @ << EOF
{
"resource_metrics": [
{
"resource": {
"attributes": [
{
"key": "mw.account_key",
"value": {
"string_value": "xxxxxxxxxx"
}
},
{
"key": "mw.resource_type",
"value": {
"string_value": "custom"
}
}
]
},
"scope_metrics": [
{
"metrics": [
{
"name": "uploads.count",
"unit": "1",
"description": "Count of user uploads",
"sum": {
"aggregationTemporality": 2,
"isMonotonic": true,
"dataPoints": [
{
"asInt": 5,
"startTimeUnixNano": 1711559739191393000,
"timeUnixNano": 1711559739191393000,
"attributes": [
{
"key": "uploads.count.user_email",
"value": {
"stringValue": "jane@domain.com"
}
}
]
}
]
}
},
{
"name": "connections.active",
"unit": "1",
"description": "Number of active connections",
"gauge": {
"dataPoints": [
{
"asInt": 215,
"timeUnixNano": 1711559739191393000,
"attributes": [
{
"key": "connections.active.hostname",
"value": {
"stringValue": "vm-prod-2"
}
}
]
}
]
}
},
{
"name": "downloads.speeds",
"unit": "By",
"description": "Download speeds of users in a particular region",
"histogram": {
"aggregationTemporality": 2,
"dataPoints": [
{
"startTimeUnixNano": 1711559739191393000,
"timeUnixNano": 1711559739191393000,
"count": 6,
"sum": 258,
"bucketCounts": [
1,
1,
1,
1,
0,
0,
0,
0,
1,
1
],
"explicitBounds": [
0,
10,
20,
30,
40,
50,
60,
70,
80,
90,
100
],
"min": 0,
"max": 98,
"attributes": [
{
"key": "region",
"value": {
"stringValue": "us-west"
}
}
]
}
]
}
}
]
}
]
}
]
}
EOF
Resource Attributes
| Field | Required | Description |
|---|---|---|
mw.account_key | ✅ | Authenticates into your Middleware Account |
mw.resource_type | ✅ | Groups and labels the metric, see Resource Types for allowed values |
Metrics
There are two components for Metrics: metadata and datapoint
Metadata
The metadata fields are the request body attributes that define the metric and determine how it will appear in Middleware.
| Metadata Field | Required | Description |
|---|---|---|
name | ✅ | Defines the metric name |
description | ✖️ | Additional details about the metric |
unit | ✖️ | Identifies the measurement type |
Datapoint
The datapoint fields are defined within the data attribute. The datapoint fields are consistent across all data attribute types, and explained below. The data attribute is named one of the following types:
- Gauge: A scalar metric that always exports the current value for every data point. It should be used for unknown aggregation.
- Sum: A scalar metric that is calculated as a sum of all reported measurements over a time interval.
- Histogram: A metric that is calculated by aggregating as a histogram of all reported measurements over a time interval.
- Exponential Histogram: A metric that is calculated by aggregating as an exponential histogram of all reported double measurements over a time interval.
- Summary: Metric data used to convey quantile summaries.
| Datapoint Field | Required | Description |
|---|---|---|
attributes | ✅ | Key-value pairs associated with the data point |
time_unix_nano | ✅ | Set to the end time of the aggregation |
start_time_unix_nano | ✖️ | Indicates the start time, should be included whenever possible |
Log Endpoint
The Log Endpoint lets you send custom logs to the Middleware backend.
To send custom logs to Middleware, POST to the following endpoint.
POST https://<UID>.middleware.io:443/v1/logs
View the following example of a curl request sending custom logs to Middleware.
curl -X POST "https://demo.middleware.io:443/v1/logs" \
-H "Accept: application/json" \
-H "Content-type: application/json" \
-d @- << EOF
{
"resource_logs": [
{
"resource": {
"attributes": [
{
"key": "mw.account_key",
"value": {
"string_value": "xxxxxxxxxx"
}
},
{
"key": "mw.resource_type",
"value": {
"string_value": "custom"
}
},
{
"key": "service_name",
"value": {
"string_value": "nginx-123"
}
}
]
},
"scope_logs": [
{
"log_records": [
{
"severity_text": "WARN",
"body": {
"string_value": "upstream server not accepting request"
},
"severity_number": "11",
"attributes": [
{
"key": "server",
"value": {
"string_value": "nginx"
}
}
],
"time_unix_nano": "1694030143000"
}
]
}
]
}
]
}
EOF
Resource Attributes
| Field | Required | Description |
|---|---|---|
mw.account_key | ✅ | Authenticates into your Middleware Account |
mw.resource_type | ✅ | Groups and labels the log source, see Resource Types for allowed values |
service_name | ✖️ | Additional parameter to identify and filter logs |
Log Records
| Field | Required | Description |
|---|---|---|
severity_text | ✖️ | Labels log severity (string), requires one of FATAL, ERROR, WARN, INFO, DEBUG, TRACE |
severity_number | ✖️ | Labels log severity (integer), number must be between 1 and 24 |
body | ✅ | The actual log body to ingest |
time_unix_nano | ✅ | Sets a timestamp for the log |
severity_number
| Severity | Message |
1-4 | TRACE |
5-8 | DEBUG |
9-12 | INFO |
13-16 | WARN |
17-20 | ERROR |
21-24 | FATAL |
Was this page helpful?