Oracle WebLogic Server
Monitor Oracle WebLogic Server in Middleware using standard OpenTelemetry components. Metrics are collected remotely, so nothing is installed on the WebLogic host for metrics collection.
How it works#
| Signal | Collected by | Runs on |
|---|---|---|
| Server, thread pool, JDBC, session, JMS and JVM metrics | OpenTelemetry JMX Scraper, over the JMX Remote API | A collector host that can reach the WebLogic JMX port |
| Metrics published by WebLogic's SNMP agent | OpenTelemetry Collector SNMP receiver | A collector host that can reach the WebLogic SNMP port |
| Server, domain and access logs | OpenTelemetry Collector filelog receiver | The WebLogic host |
| Traces | Middleware Java APM agent | Inside the WebLogic JVM (Java 8 or later) |
Both metric methods send data to Middleware over OTLP, either directly or through a local Middleware Agent.
Choose a metrics method#
| Use | When |
|---|---|
| JMX (recommended) | The WebLogic JVM supports the JMX Remote API (JSR 160). This gives the most detail: per-server, per-data-source and per-application metrics, plus JVM metrics. |
| SNMP | The WebLogic release predates the JMX Remote API (WebLogic Server 8.1 and earlier), or SNMP is the management protocol approved in your environment. |
Prerequisites#
- Collector host: A Linux or Windows host with network access to WebLogic and outbound HTTPS (port 443) to Middleware.
- For JMX: Java 8 or later on the collector host:
java -version. The WebLogic JVM itself can run an older Java release; only the collector host runs the scraper. - For SNMP: OpenTelemetry Collector Contrib installed on the collector host.
- Read-only credentials: A WebLogic user in the
Monitorsgroup for JMX over T3, a read-only role for the JVM JMX connector, or a read-only SNMP community or SNMPv3 user.
Collect metrics over JMX#
1 Expose JMX from WebLogic#
Choose one connector. The JVM connector is the most widely compatible with OpenTelemetry tooling. The T3 connector uses WebLogic's own protocol and needs no JVM changes.
Add the standard JMX options to the server's start arguments, for example in $DOMAIN_HOME/bin/setUserOverrides.sh (or setDomainEnv.sh on releases without it):
JAVA_OPTIONS="${JAVA_OPTIONS} \
-Dcom.sun.management.jmxremote.port=9999 \
-Dcom.sun.management.jmxremote.rmi.port=9999 \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.ssl=true \
-Dcom.sun.management.jmxremote.password.file=/path/to/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/path/to/jmxremote.access"
export JAVA_OPTIONSHere,
jmxremote.accessgrants the monitoring user read-only access, for examplemonitor readonly.jmxremote.rmi.portpins the RMI port so only one firewall port is needed. Older JVMs ignore it and pick a random second port; see Troubleshooting.- Give each managed server its own port.
By default the JVM connector exposes the JVM's platform MBean server. For WebLogic's own metrics to appear there, make WebLogic use the platform MBean server. With WLST:
connect('<admin-user>', '<admin-password>', 't3://<admin-host>:7001')
edit()
startEdit()
cd('/JMX/<domain-name>')
cmo.setPlatformMBeanServerUsed(true)
activate()Restart the servers after both changes.
WebLogic serves its Runtime MBean Server over T3 with no JVM changes. The service URL is:
service:jmx:t3://<server-host>:<listen-port>/jndi/weblogic.management.mbeanservers.runtimeCopy WebLogic's T3 client JAR, for example $WL_HOME/server/lib/wlthint3client.jar, to the collector host. Step 4 adds it to the scraper's classpath. Use a WebLogic user in the Monitors group.
The T3 connector isn't covered by the OpenTelemetry JMX Scraper documentation. Use the scraper's connection test in step 4 to confirm it works with your WebLogic release.
2 Install the JMX Scraper on the collector host#
Download the latest release of the OpenTelemetry JMX Scraper:
sudo mkdir -p /opt/otel-jmx
sudo curl -L -o /opt/otel-jmx/opentelemetry-jmx-scraper.jar \
https://github.com/open-telemetry/opentelemetry-java-contrib/releases/latest/download/opentelemetry-jmx-scraper.jar3 Define the WebLogic metrics#
The scraper has no built-in WebLogic definitions, so create /opt/otel-jmx/weblogic.yaml with the rules below. The rules use the OpenTelemetry JMX metrics YAML format.
---
rules:
- bean: com.bea:Type=ServerRuntime,*
prefix: weblogic.server.
metricAttribute:
weblogic.server.name: param(Name)
mapping:
State:
metric: state
type: state
metricAttribute:
weblogic.server.state:
running: RUNNING
starting: [STARTING, RESUMING, STANDBY, ADMIN]
stopping: [SUSPENDING, FORCE_SUSPENDING, SHUTTING_DOWN, FORCE_SHUTTING_DOWN]
failed: '*'
OpenSocketsCurrentCount:
metric: open_sockets
type: updowncounter
unit: "{socket}"
desc: Current number of open sockets
- bean: com.bea:Type=ThreadPoolRuntime,*
prefix: weblogic.threadpool.
metricAttribute:
weblogic.server.name: param(ServerRuntime)
mapping:
ExecuteThreadTotalCount:
metric: threads
type: updowncounter
unit: "{thread}"
desc: Total threads in the self-tuning thread pool
ExecuteThreadIdleCount:
metric: threads.idle
type: updowncounter
unit: "{thread}"
desc: Idle threads in the pool
HoggingThreadCount:
metric: threads.hogging
type: updowncounter
unit: "{thread}"
desc: Threads held by a request for an unusually long time
StuckThreadCount:
metric: threads.stuck
type: updowncounter
unit: "{thread}"
desc: Threads WebLogic has marked as stuck
PendingUserRequestCount:
metric: requests.pending
type: updowncounter
unit: "{request}"
desc: User requests waiting for a thread
QueueLength:
metric: queue.length
type: updowncounter
unit: "{request}"
desc: Requests in the priority queue
CompletedRequestCount:
metric: requests.completed
type: counter
unit: "{request}"
desc: Requests completed by the pool
- bean: com.bea:Type=JDBCDataSourceRuntime,*
prefix: weblogic.jdbc.
metricAttribute:
weblogic.server.name: param(ServerRuntime)
weblogic.datasource.name: param(Name)
mapping:
ActiveConnectionsCurrentCount:
metric: connections.active
type: updowncounter
unit: "{connection}"
desc: Connections currently in use
CurrCapacity:
metric: connections.capacity
type: updowncounter
unit: "{connection}"
desc: Current size of the connection pool
WaitingForConnectionCurrentCount:
metric: connections.waiting
type: updowncounter
unit: "{request}"
desc: Requests waiting for a connection
FailedReserveRequestCount:
metric: reserve.failed
type: counter
unit: "{request}"
desc: Connection requests that failed
LeakedConnectionCount:
metric: connections.leaked
type: counter
unit: "{connection}"
desc: Connections reserved but not returned to the pool
- bean: com.bea:Type=WebAppComponentRuntime,*
prefix: weblogic.webapp.
metricAttribute:
weblogic.server.name: param(ServerRuntime)
weblogic.webapp.name: param(Name)
mapping:
OpenSessionsCurrentCount:
metric: sessions.open
type: updowncounter
unit: "{session}"
desc: Currently open HTTP sessions
SessionsOpenedTotalCount:
metric: sessions.opened
type: counter
unit: "{session}"
desc: HTTP sessions opened since start
- bean: com.bea:Type=JMSDestinationRuntime,*
prefix: weblogic.jms.destination.
metricAttribute:
weblogic.server.name: param(ServerRuntime)
weblogic.jms.destination.name: param(Name)
mapping:
MessagesCurrentCount:
metric: messages
type: updowncounter
unit: "{message}"
desc: Messages stored on the destination
MessagesPendingCount:
metric: messages.pending
type: updowncounter
unit: "{message}"
desc: Messages pending delivery or acknowledgement
ConsumersCurrentCount:
metric: consumers
type: updowncounter
unit: "{consumer}"
desc: Current consumersHere,
- Each rule matches WebLogic runtime MBeans by
Type, and turns the MBean'sNameandServerRuntimekeys into metric attributes. - Attribute names vary between WebLogic releases. If a rule returns nothing, check the attribute in the WebLogic MBean reference for your release and remove or rename it.
- JVM metrics (heap, garbage collection, threads, classes) come from the scraper's built-in
jvmtarget, set in the next step.
4 Configure and run the scraper#
Create /opt/otel-jmx/weblogic.properties. Choose where to send the data:
otel.jmx.service.url=service:jmx:rmi:///jndi/rmi://<server-host>:9999/jmxrmi
otel.jmx.target.system=jvm
otel.jmx.config=/opt/otel-jmx/weblogic.yaml
otel.jmx.username=<monitoring-user>
otel.jmx.password=<monitoring-password>
otel.metric.export.interval=60s
otel.metrics.exporter=otlp
otel.exporter.otlp.protocol=http/protobuf
otel.exporter.otlp.endpoint=https://<MW_UID>.middleware.io:443
otel.exporter.otlp.headers=authorization=<MW_API_KEY>
otel.resource.attributes=service.name=weblogic,host.name=<server-host>Send to a Middleware Agent on the collector host, which adds its own buffering and host metadata:
otel.jmx.service.url=service:jmx:rmi:///jndi/rmi://<server-host>:9999/jmxrmi
otel.jmx.target.system=jvm
otel.jmx.config=/opt/otel-jmx/weblogic.yaml
otel.jmx.username=<monitoring-user>
otel.jmx.password=<monitoring-password>
otel.metric.export.interval=60s
otel.metrics.exporter=otlp
otel.exporter.otlp.protocol=http/protobuf
otel.exporter.otlp.endpoint=http://localhost:9320
otel.resource.attributes=service.name=weblogic,host.name=<server-host>Here,
otel.jmx.service.url: for the T3 connector, use theservice:jmx:t3://...URL from step 1.otel.resource.attributes: sethost.nameto the WebLogic host, not the collector host, so metrics attach to the right server. Run one scraper per WebLogic server.- Restrict the file to the account that runs the scraper (
chmod 600), as it holds credentials.
Test the connection, then start the scraper:
java -jar /opt/otel-jmx/opentelemetry-jmx-scraper.jar -config /opt/otel-jmx/weblogic.properties -test
java -jar /opt/otel-jmx/opentelemetry-jmx-scraper.jar -config /opt/otel-jmx/weblogic.propertiesFor the T3 connector, put the WebLogic client JAR on the classpath and name the main class:
java -Djmx.remote.protocol.provider.pkgs=weblogic.management.remote \
-cp /opt/otel-jmx/opentelemetry-jmx-scraper.jar:/opt/otel-jmx/wlthint3client.jar \
io.opentelemetry.contrib.jmxscraper.JmxScraper -config /opt/otel-jmx/weblogic.propertiesTo run it as a service on Linux, create /etc/systemd/system/otel-jmx-weblogic.service:
[Unit]
Description=OpenTelemetry JMX Scraper for WebLogic
After=network-online.target
[Service]
User=otel-jmx
ExecStart=/usr/bin/java -jar /opt/otel-jmx/opentelemetry-jmx-scraper.jar -config /opt/otel-jmx/weblogic.properties
Restart=on-failure
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload && sudo systemctl enable --now otel-jmx-weblogicCollect metrics over SNMP#
WebLogic Server includes an SNMP agent that publishes server attributes through the BEA-WEBLOGIC-MIB (OID .1.3.6.1.4.1.140.625). See Oracle's SNMP guide for your release, for example WebLogic Server 8.1.
1 Enable the WebLogic SNMP agent#
In the WebLogic Administration Console, enable the domain's SNMP agent and set:
- A read-only community, or an SNMPv3 user on releases that support it.
- The UDP port the agent listens on.
- Allowed managers limited to the collector host, where your release supports it.
The SNMP agent runs on the domain's Administration Server.
2 Choose the OIDs to collect#
List what the agent publishes from the collector host:
snmpwalk -v 2c -c <community> <admin-host>:<snmp-port> .1.3.6.1.4.1.140.625Pick numeric OIDs (counters, gauges and integers) from the MIB file shipped with your WebLogic release. String values, such as server state names, can't be collected as metrics.
3 Configure the OpenTelemetry Collector#
Add the SNMP receiver to your OpenTelemetry Collector Contrib configuration. Replace the example OIDs with the ones you chose in step 2:
receivers:
snmp/weblogic:
collection_interval: 60s
endpoint: udp://<admin-host>:<snmp-port>
version: v2c
community: <community>
resource_attributes:
service.name:
scalar_oid: <scalar OID ending in .0, e.g. a server name>
attributes:
weblogic.row:
indexed_value_prefix: row-
metrics:
weblogic.<metric_name>:
unit: "{thread}"
gauge:
value_type: int
column_oids:
- oid: <column OID from BEA-WEBLOGIC-MIB>
attributes:
- name: weblogic.row
exporters:
otlphttp/middleware:
endpoint: https://<MW_UID>.middleware.io:443
headers:
authorization: <MW_API_KEY>
service:
pipelines:
metrics:
receivers: [snmp/weblogic]
exporters: [otlphttp/middleware]Here,
column_oidscollects one value per table row. Each row gets the attribute named underattributes, either from the row index (indexed_value_prefix) or from another column at the same index (oid).scalar_oidscollects a single value; the OID must end in.0.- For SNMPv3, set
version: v3,user,security_level: auth_priv,auth_type,auth_password,privacy_typeandprivacy_passwordinstead ofcommunity.
SNMPv1 and v2c send the community string in plain text. Use a read-only community, restrict the agent to the collector host, and prefer SNMPv3 where your release supports it.
Collect logs#
WebLogic writes these logs on each host:
| Log | Default location |
|---|---|
| Server log | $DOMAIN_HOME/servers/<server-name>/logs/<server-name>.log |
| Domain log (Administration Server) | $DOMAIN_HOME/servers/<admin-server>/logs/<domain-name>.log |
| HTTP access log | $DOMAIN_HOME/servers/<server-name>/logs/access.log |
Collect them with the filelog receiver in OpenTelemetry Collector Contrib on the WebLogic host. Server and domain log entries start with ####< and can span several lines, so join them into one record:
receivers:
filelog/weblogic:
include:
- /path/to/domain/servers/*/logs/*.log
exclude:
- /path/to/domain/servers/*/logs/access.log
multiline:
line_start_pattern: '^####<'
resource:
service.name: weblogic
exporters:
otlphttp/middleware:
endpoint: https://<MW_UID>.middleware.io:443
headers:
authorization: <MW_API_KEY>
service:
pipelines:
logs:
receivers: [filelog/weblogic]
exporters: [otlphttp/middleware]Collect traces#
For WebLogic servers running Java 8 or later, add the Middleware Java APM agent to the server's JAVA_OPTIONS. It traces incoming HTTP requests, JDBC calls and outgoing calls without code changes.
WebLogic isn't part of OpenTelemetry's application server test suite, so verify traces in a non-production domain first.
Network ports#
| From | To | Port | Protocol |
|---|---|---|---|
| Collector host | WebLogic server (JVM connector) | The jmxremote port, for example 9999 | TCP (RMI) |
| Collector host | WebLogic server (T3 connector) | The server listen port, default 7001 | TCP (T3) |
| Collector host | WebLogic Administration Server | The SNMP agent port | UDP |
| Collector host, WebLogic host | Middleware | 443 | HTTPS (OTLP) |
View your data#
Search for weblogic. and jvm. metrics in the Metrics Explorer, and build WebLogic dashboards with the Dashboard Builder. Useful widgets include stuck and hogging threads, pending requests, JDBC active and waiting connections, open sessions and JVM heap.
Metrics collected#
These metrics come from the JMX rules in this guide:
| Metric | Description |
|---|---|
weblogic.server.state | Server lifecycle state (running, starting, stopping, failed) |
weblogic.server.open_sockets | Current number of open sockets |
weblogic.threadpool.threads | Total threads in the self-tuning thread pool |
weblogic.threadpool.threads.idle | Idle threads |
weblogic.threadpool.threads.hogging | Threads held by a request for an unusually long time |
weblogic.threadpool.threads.stuck | Threads marked as stuck |
weblogic.threadpool.requests.pending | User requests waiting for a thread |
weblogic.threadpool.queue.length | Requests in the priority queue |
weblogic.threadpool.requests.completed | Requests completed |
weblogic.jdbc.connections.active | Connections in use, per data source |
weblogic.jdbc.connections.capacity | Current pool size, per data source |
weblogic.jdbc.connections.waiting | Requests waiting for a connection |
weblogic.jdbc.reserve.failed | Failed connection requests |
weblogic.jdbc.connections.leaked | Connections not returned to the pool |
weblogic.webapp.sessions.open | Open HTTP sessions, per web application |
weblogic.webapp.sessions.opened | HTTP sessions opened since start |
weblogic.jms.destination.messages | Messages stored, per JMS destination |
weblogic.jms.destination.messages.pending | Messages pending delivery or acknowledgement |
weblogic.jms.destination.consumers | Current consumers |
jvm.* | Heap and non-heap memory, garbage collection, threads and classes, from the jvm target |
Troubleshooting#
- Check the port from the collector host:
nc -vz <server-host> 9999. - Older JVMs ignore
com.sun.management.jmxremote.rmi.portand open a second, random RMI port. Allow it through the firewall, or connect over the T3 connector instead. - If the JVM reports its hostname as an address the collector can't reach, add
-Djava.rmi.server.hostname=<reachable-address>to the WebLogic start arguments.
With the JVM connector, WebLogic's MBeans are only visible if WebLogic uses the platform MBean server. Set PlatformMBeanServerUsed to true (step 1) and restart. Otherwise, use the T3 connector.
The scraper can't find WebLogic's T3 client. Run it with -cp including the client JAR and -Djmx.remote.protocol.provider.pkgs=weblogic.management.remote, as shown in step 4. The java -jar form ignores extra classpath entries.
MBean attributes differ between WebLogic releases. Check the attribute names for your release in the WebLogic MBean reference, and update weblogic.yaml. A rule that matches no MBeans, for example the JMS rule on a server without JMS destinations, reports nothing.
- Check that the SNMP agent is enabled on the Administration Server and listening on the configured UDP port.
- Test from the collector host with
snmpwalk. Check the community string or SNMPv3 credentials and any allowed-manager restrictions. - Make sure the firewall allows UDP from the collector host.
- Check the endpoint and API key. For direct export, the endpoint is your Middleware URL, and OTLP/HTTP adds
/v1/metricsautomatically. - Add
loggingtootel.metrics.exporter(for exampleotel.metrics.exporter=otlp,logging) to print each export in the scraper's output. - Check that the collector host allows outbound HTTPS to Middleware, or set a proxy.
Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.
What did you think of this content?