Oracle WebLogic Server

Monitor Oracle WebLogic Server in Middleware using standard OpenTelemetry components. Metrics are collected remotely, so nothing is installed on the WebLogic host for metrics collection.

How it works#

SignalCollected byRuns on
Server, thread pool, JDBC, session, JMS and JVM metricsOpenTelemetry JMX Scraper, over the JMX Remote APIA collector host that can reach the WebLogic JMX port
Metrics published by WebLogic's SNMP agentOpenTelemetry Collector SNMP receiverA collector host that can reach the WebLogic SNMP port
Server, domain and access logsOpenTelemetry Collector filelog receiverThe WebLogic host
TracesMiddleware Java APM agentInside the WebLogic JVM (Java 8 or later)

Both metric methods send data to Middleware over OTLP, either directly or through a local Middleware Agent.

Choose a metrics method#

UseWhen
JMX (recommended)The WebLogic JVM supports the JMX Remote API (JSR 160). This gives the most detail: per-server, per-data-source and per-application metrics, plus JVM metrics.
SNMPThe WebLogic release predates the JMX Remote API (WebLogic Server 8.1 and earlier), or SNMP is the management protocol approved in your environment.

Prerequisites#

  1. Collector host: A Linux or Windows host with network access to WebLogic and outbound HTTPS (port 443) to Middleware.
  2. For JMX: Java 8 or later on the collector host: java -version. The WebLogic JVM itself can run an older Java release; only the collector host runs the scraper.
  3. For SNMP: OpenTelemetry Collector Contrib installed on the collector host.
  4. Read-only credentials: A WebLogic user in the Monitors group for JMX over T3, a read-only role for the JVM JMX connector, or a read-only SNMP community or SNMPv3 user.

Collect metrics over JMX#

1 Expose JMX from WebLogic#

Choose one connector. The JVM connector is the most widely compatible with OpenTelemetry tooling. The T3 connector uses WebLogic's own protocol and needs no JVM changes.

Add the standard JMX options to the server's start arguments, for example in $DOMAIN_HOME/bin/setUserOverrides.sh (or setDomainEnv.sh on releases without it):

bash
JAVA_OPTIONS="${JAVA_OPTIONS} \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.rmi.port=9999 \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=true \
  -Dcom.sun.management.jmxremote.password.file=/path/to/jmxremote.password \
  -Dcom.sun.management.jmxremote.access.file=/path/to/jmxremote.access"
export JAVA_OPTIONS

Here,

  • jmxremote.access grants the monitoring user read-only access, for example monitor readonly.
  • jmxremote.rmi.port pins the RMI port so only one firewall port is needed. Older JVMs ignore it and pick a random second port; see Troubleshooting.
  • Give each managed server its own port.

By default the JVM connector exposes the JVM's platform MBean server. For WebLogic's own metrics to appear there, make WebLogic use the platform MBean server. With WLST:

python
connect('<admin-user>', '<admin-password>', 't3://<admin-host>:7001')
edit()
startEdit()
cd('/JMX/<domain-name>')
cmo.setPlatformMBeanServerUsed(true)
activate()

Restart the servers after both changes.

WebLogic serves its Runtime MBean Server over T3 with no JVM changes. The service URL is:

bash
service:jmx:t3://<server-host>:<listen-port>/jndi/weblogic.management.mbeanservers.runtime

Copy WebLogic's T3 client JAR, for example $WL_HOME/server/lib/wlthint3client.jar, to the collector host. Step 4 adds it to the scraper's classpath. Use a WebLogic user in the Monitors group.

The T3 connector isn't covered by the OpenTelemetry JMX Scraper documentation. Use the scraper's connection test in step 4 to confirm it works with your WebLogic release.

2 Install the JMX Scraper on the collector host#

Download the latest release of the OpenTelemetry JMX Scraper:

bash
sudo mkdir -p /opt/otel-jmx
sudo curl -L -o /opt/otel-jmx/opentelemetry-jmx-scraper.jar \
  https://github.com/open-telemetry/opentelemetry-java-contrib/releases/latest/download/opentelemetry-jmx-scraper.jar

3 Define the WebLogic metrics#

The scraper has no built-in WebLogic definitions, so create /opt/otel-jmx/weblogic.yaml with the rules below. The rules use the OpenTelemetry JMX metrics YAML format.

yaml
---
rules:
  - bean: com.bea:Type=ServerRuntime,*
    prefix: weblogic.server.
    metricAttribute:
      weblogic.server.name: param(Name)
    mapping:
      State:
        metric: state
        type: state
        metricAttribute:
          weblogic.server.state:
            running: RUNNING
            starting: [STARTING, RESUMING, STANDBY, ADMIN]
            stopping: [SUSPENDING, FORCE_SUSPENDING, SHUTTING_DOWN, FORCE_SHUTTING_DOWN]
            failed: '*'
      OpenSocketsCurrentCount:
        metric: open_sockets
        type: updowncounter
        unit: "{socket}"
        desc: Current number of open sockets

  - bean: com.bea:Type=ThreadPoolRuntime,*
    prefix: weblogic.threadpool.
    metricAttribute:
      weblogic.server.name: param(ServerRuntime)
    mapping:
      ExecuteThreadTotalCount:
        metric: threads
        type: updowncounter
        unit: "{thread}"
        desc: Total threads in the self-tuning thread pool
      ExecuteThreadIdleCount:
        metric: threads.idle
        type: updowncounter
        unit: "{thread}"
        desc: Idle threads in the pool
      HoggingThreadCount:
        metric: threads.hogging
        type: updowncounter
        unit: "{thread}"
        desc: Threads held by a request for an unusually long time
      StuckThreadCount:
        metric: threads.stuck
        type: updowncounter
        unit: "{thread}"
        desc: Threads WebLogic has marked as stuck
      PendingUserRequestCount:
        metric: requests.pending
        type: updowncounter
        unit: "{request}"
        desc: User requests waiting for a thread
      QueueLength:
        metric: queue.length
        type: updowncounter
        unit: "{request}"
        desc: Requests in the priority queue
      CompletedRequestCount:
        metric: requests.completed
        type: counter
        unit: "{request}"
        desc: Requests completed by the pool

  - bean: com.bea:Type=JDBCDataSourceRuntime,*
    prefix: weblogic.jdbc.
    metricAttribute:
      weblogic.server.name: param(ServerRuntime)
      weblogic.datasource.name: param(Name)
    mapping:
      ActiveConnectionsCurrentCount:
        metric: connections.active
        type: updowncounter
        unit: "{connection}"
        desc: Connections currently in use
      CurrCapacity:
        metric: connections.capacity
        type: updowncounter
        unit: "{connection}"
        desc: Current size of the connection pool
      WaitingForConnectionCurrentCount:
        metric: connections.waiting
        type: updowncounter
        unit: "{request}"
        desc: Requests waiting for a connection
      FailedReserveRequestCount:
        metric: reserve.failed
        type: counter
        unit: "{request}"
        desc: Connection requests that failed
      LeakedConnectionCount:
        metric: connections.leaked
        type: counter
        unit: "{connection}"
        desc: Connections reserved but not returned to the pool

  - bean: com.bea:Type=WebAppComponentRuntime,*
    prefix: weblogic.webapp.
    metricAttribute:
      weblogic.server.name: param(ServerRuntime)
      weblogic.webapp.name: param(Name)
    mapping:
      OpenSessionsCurrentCount:
        metric: sessions.open
        type: updowncounter
        unit: "{session}"
        desc: Currently open HTTP sessions
      SessionsOpenedTotalCount:
        metric: sessions.opened
        type: counter
        unit: "{session}"
        desc: HTTP sessions opened since start

  - bean: com.bea:Type=JMSDestinationRuntime,*
    prefix: weblogic.jms.destination.
    metricAttribute:
      weblogic.server.name: param(ServerRuntime)
      weblogic.jms.destination.name: param(Name)
    mapping:
      MessagesCurrentCount:
        metric: messages
        type: updowncounter
        unit: "{message}"
        desc: Messages stored on the destination
      MessagesPendingCount:
        metric: messages.pending
        type: updowncounter
        unit: "{message}"
        desc: Messages pending delivery or acknowledgement
      ConsumersCurrentCount:
        metric: consumers
        type: updowncounter
        unit: "{consumer}"
        desc: Current consumers

Here,

  • Each rule matches WebLogic runtime MBeans by Type, and turns the MBean's Name and ServerRuntime keys into metric attributes.
  • Attribute names vary between WebLogic releases. If a rule returns nothing, check the attribute in the WebLogic MBean reference for your release and remove or rename it.
  • JVM metrics (heap, garbage collection, threads, classes) come from the scraper's built-in jvm target, set in the next step.

4 Configure and run the scraper#

Create /opt/otel-jmx/weblogic.properties. Choose where to send the data:

properties
otel.jmx.service.url=service:jmx:rmi:///jndi/rmi://<server-host>:9999/jmxrmi
otel.jmx.target.system=jvm
otel.jmx.config=/opt/otel-jmx/weblogic.yaml
otel.jmx.username=<monitoring-user>
otel.jmx.password=<monitoring-password>
otel.metric.export.interval=60s
otel.metrics.exporter=otlp
otel.exporter.otlp.protocol=http/protobuf
otel.exporter.otlp.endpoint=https://<MW_UID>.middleware.io:443
otel.exporter.otlp.headers=authorization=<MW_API_KEY>
otel.resource.attributes=service.name=weblogic,host.name=<server-host>

Send to a Middleware Agent on the collector host, which adds its own buffering and host metadata:

properties
otel.jmx.service.url=service:jmx:rmi:///jndi/rmi://<server-host>:9999/jmxrmi
otel.jmx.target.system=jvm
otel.jmx.config=/opt/otel-jmx/weblogic.yaml
otel.jmx.username=<monitoring-user>
otel.jmx.password=<monitoring-password>
otel.metric.export.interval=60s
otel.metrics.exporter=otlp
otel.exporter.otlp.protocol=http/protobuf
otel.exporter.otlp.endpoint=http://localhost:9320
otel.resource.attributes=service.name=weblogic,host.name=<server-host>

Here,

  • otel.jmx.service.url: for the T3 connector, use the service:jmx:t3://... URL from step 1.
  • otel.resource.attributes: set host.name to the WebLogic host, not the collector host, so metrics attach to the right server. Run one scraper per WebLogic server.
  • Restrict the file to the account that runs the scraper (chmod 600), as it holds credentials.

Test the connection, then start the scraper:

bash
java -jar /opt/otel-jmx/opentelemetry-jmx-scraper.jar -config /opt/otel-jmx/weblogic.properties -test
java -jar /opt/otel-jmx/opentelemetry-jmx-scraper.jar -config /opt/otel-jmx/weblogic.properties

For the T3 connector, put the WebLogic client JAR on the classpath and name the main class:

bash
java -Djmx.remote.protocol.provider.pkgs=weblogic.management.remote \
  -cp /opt/otel-jmx/opentelemetry-jmx-scraper.jar:/opt/otel-jmx/wlthint3client.jar \
  io.opentelemetry.contrib.jmxscraper.JmxScraper -config /opt/otel-jmx/weblogic.properties

To run it as a service on Linux, create /etc/systemd/system/otel-jmx-weblogic.service:

ini
[Unit]
Description=OpenTelemetry JMX Scraper for WebLogic
After=network-online.target

[Service]
User=otel-jmx
ExecStart=/usr/bin/java -jar /opt/otel-jmx/opentelemetry-jmx-scraper.jar -config /opt/otel-jmx/weblogic.properties
Restart=on-failure

[Install]
WantedBy=multi-user.target
bash
sudo systemctl daemon-reload && sudo systemctl enable --now otel-jmx-weblogic

Collect metrics over SNMP#

WebLogic Server includes an SNMP agent that publishes server attributes through the BEA-WEBLOGIC-MIB (OID .1.3.6.1.4.1.140.625). See Oracle's SNMP guide for your release, for example WebLogic Server 8.1.

1 Enable the WebLogic SNMP agent#

In the WebLogic Administration Console, enable the domain's SNMP agent and set:

  • A read-only community, or an SNMPv3 user on releases that support it.
  • The UDP port the agent listens on.
  • Allowed managers limited to the collector host, where your release supports it.

The SNMP agent runs on the domain's Administration Server.

2 Choose the OIDs to collect#

List what the agent publishes from the collector host:

bash
snmpwalk -v 2c -c <community> <admin-host>:<snmp-port> .1.3.6.1.4.1.140.625

Pick numeric OIDs (counters, gauges and integers) from the MIB file shipped with your WebLogic release. String values, such as server state names, can't be collected as metrics.

3 Configure the OpenTelemetry Collector#

Add the SNMP receiver to your OpenTelemetry Collector Contrib configuration. Replace the example OIDs with the ones you chose in step 2:

yaml
receivers:
  snmp/weblogic:
    collection_interval: 60s
    endpoint: udp://<admin-host>:<snmp-port>
    version: v2c
    community: <community>
    resource_attributes:
      service.name:
        scalar_oid: <scalar OID ending in .0, e.g. a server name>
    attributes:
      weblogic.row:
        indexed_value_prefix: row-
    metrics:
      weblogic.<metric_name>:
        unit: "{thread}"
        gauge:
          value_type: int
        column_oids:
          - oid: <column OID from BEA-WEBLOGIC-MIB>
            attributes:
              - name: weblogic.row

exporters:
  otlphttp/middleware:
    endpoint: https://<MW_UID>.middleware.io:443
    headers:
      authorization: <MW_API_KEY>

service:
  pipelines:
    metrics:
      receivers: [snmp/weblogic]
      exporters: [otlphttp/middleware]

Here,

  • column_oids collects one value per table row. Each row gets the attribute named under attributes, either from the row index (indexed_value_prefix) or from another column at the same index (oid).
  • scalar_oids collects a single value; the OID must end in .0.
  • For SNMPv3, set version: v3, user, security_level: auth_priv, auth_type, auth_password, privacy_type and privacy_password instead of community.

SNMPv1 and v2c send the community string in plain text. Use a read-only community, restrict the agent to the collector host, and prefer SNMPv3 where your release supports it.

Collect logs#

WebLogic writes these logs on each host:

LogDefault location
Server log$DOMAIN_HOME/servers/<server-name>/logs/<server-name>.log
Domain log (Administration Server)$DOMAIN_HOME/servers/<admin-server>/logs/<domain-name>.log
HTTP access log$DOMAIN_HOME/servers/<server-name>/logs/access.log

Collect them with the filelog receiver in OpenTelemetry Collector Contrib on the WebLogic host. Server and domain log entries start with ####< and can span several lines, so join them into one record:

yaml
receivers:
  filelog/weblogic:
    include:
      - /path/to/domain/servers/*/logs/*.log
    exclude:
      - /path/to/domain/servers/*/logs/access.log
    multiline:
      line_start_pattern: '^####<'
    resource:
      service.name: weblogic

exporters:
  otlphttp/middleware:
    endpoint: https://<MW_UID>.middleware.io:443
    headers:
      authorization: <MW_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/weblogic]
      exporters: [otlphttp/middleware]

Collect traces#

For WebLogic servers running Java 8 or later, add the Middleware Java APM agent to the server's JAVA_OPTIONS. It traces incoming HTTP requests, JDBC calls and outgoing calls without code changes.

WebLogic isn't part of OpenTelemetry's application server test suite, so verify traces in a non-production domain first.

Network ports#

FromToPortProtocol
Collector hostWebLogic server (JVM connector)The jmxremote port, for example 9999TCP (RMI)
Collector hostWebLogic server (T3 connector)The server listen port, default 7001TCP (T3)
Collector hostWebLogic Administration ServerThe SNMP agent portUDP
Collector host, WebLogic hostMiddleware443HTTPS (OTLP)

View your data#

Search for weblogic. and jvm. metrics in the Metrics Explorer, and build WebLogic dashboards with the Dashboard Builder. Useful widgets include stuck and hogging threads, pending requests, JDBC active and waiting connections, open sessions and JVM heap.

Metrics collected#

These metrics come from the JMX rules in this guide:

MetricDescription
weblogic.server.stateServer lifecycle state (running, starting, stopping, failed)
weblogic.server.open_socketsCurrent number of open sockets
weblogic.threadpool.threadsTotal threads in the self-tuning thread pool
weblogic.threadpool.threads.idleIdle threads
weblogic.threadpool.threads.hoggingThreads held by a request for an unusually long time
weblogic.threadpool.threads.stuckThreads marked as stuck
weblogic.threadpool.requests.pendingUser requests waiting for a thread
weblogic.threadpool.queue.lengthRequests in the priority queue
weblogic.threadpool.requests.completedRequests completed
weblogic.jdbc.connections.activeConnections in use, per data source
weblogic.jdbc.connections.capacityCurrent pool size, per data source
weblogic.jdbc.connections.waitingRequests waiting for a connection
weblogic.jdbc.reserve.failedFailed connection requests
weblogic.jdbc.connections.leakedConnections not returned to the pool
weblogic.webapp.sessions.openOpen HTTP sessions, per web application
weblogic.webapp.sessions.openedHTTP sessions opened since start
weblogic.jms.destination.messagesMessages stored, per JMS destination
weblogic.jms.destination.messages.pendingMessages pending delivery or acknowledgement
weblogic.jms.destination.consumersCurrent consumers
jvm.*Heap and non-heap memory, garbage collection, threads and classes, from the jvm target

Troubleshooting#

  • Check the port from the collector host: nc -vz <server-host> 9999.
  • Older JVMs ignore com.sun.management.jmxremote.rmi.port and open a second, random RMI port. Allow it through the firewall, or connect over the T3 connector instead.
  • If the JVM reports its hostname as an address the collector can't reach, add -Djava.rmi.server.hostname=<reachable-address> to the WebLogic start arguments.

With the JVM connector, WebLogic's MBeans are only visible if WebLogic uses the platform MBean server. Set PlatformMBeanServerUsed to true (step 1) and restart. Otherwise, use the T3 connector.

The scraper can't find WebLogic's T3 client. Run it with -cp including the client JAR and -Djmx.remote.protocol.provider.pkgs=weblogic.management.remote, as shown in step 4. The java -jar form ignores extra classpath entries.

MBean attributes differ between WebLogic releases. Check the attribute names for your release in the WebLogic MBean reference, and update weblogic.yaml. A rule that matches no MBeans, for example the JMS rule on a server without JMS destinations, reports nothing.

  • Check that the SNMP agent is enabled on the Administration Server and listening on the configured UDP port.
  • Test from the collector host with snmpwalk. Check the community string or SNMPv3 credentials and any allowed-manager restrictions.
  • Make sure the firewall allows UDP from the collector host.
  • Check the endpoint and API key. For direct export, the endpoint is your Middleware URL, and OTLP/HTTP adds /v1/metrics automatically.
  • Add logging to otel.metrics.exporter (for example otel.metrics.exporter=otlp,logging) to print each export in the scraper's output.
  • Check that the collector host allows outbound HTTPS to Middleware, or set a proxy.

Need assistance or want to learn more about Middleware? Get in touch with us via our Contact Us or join our Slack channel.

What did you think of this content?