Continuous Profiling

Profiling, in the context of software development, refers to the process of analyzing and measuring the performance characteristics of a program or system. It involves using application profiling tools to gather metrics like CPU usage, memory allocation, and execution time.

Continuous profiling takes things a step further by enabling real-time monitoring and detection of performance regressions. It allows developers to proactively address any emerging issues before they impact users.

Flame Graph

Flame Graphs displays the total amount of memory (in RAM) that each operation accrues in your distributed system. Each operation is an individual node and is presented in a hierarchical format. The longer the node, the more memory that operation has consumed. This view is useful for identifying common resource requests, quickly assessing performance hotspots, and filtering through specific services.

Continuous Profiling Table

The Continuous Profiling Table organizes the same data presented in the Flame Graph in a column-oriented representation. Each row in the table corresponds to an individual operation, with the Self column representing the amount of memory (in RAM) used when that operation fires once, and the Total column representing all of the memory used by that parent operation. This is useful for comparing the difference between the volume of memory being used for a single service compared to the total memory usage of the parent span.

Sandwich Graph

The Sandwich Graph allows you to break down your Flame Graph by parent and child functions. Parent functions are named Callers, which are the original functions in question. Child functions are named Callees, the functions that the original function in question has called. This is useful for mitigating redundancies across different function calls and reducing the total amount of memory used across your entire system.

Continuous Profiling Support Matrix

The following table shows what Continuous Profiling features are available per each APM:

Profiling A Flask Web Application

Prerequisites

Create Your Flask App

import math

def calc1(n: int) -> int:
    value = 0
    for i in range(0, n):
        for j in range(0, n):
            value += math.sqrt(j)
            value += math.sqrt(i)
    return value

import calculations
from flask import Flask
import os

# Required middleware import 
from middleware import MwTracker
tracker = MwTracker()

app = Flask(__name__)
app.secret_key = os.environ.get("FLASK_SECRET_KEY", default="Da!8j97ZZq^6Fe3y!D@U")

from app import app

# To bind to gunicorn
# gunicorn wsgi:app
if __name__ == "__main__":
    app.run()

Finding a Bottleneck

After creating the application and deploying it to the platform of your choice, follow the steps from Python APM, login to middleware.io, and head to APM > Continuous Profiling.

To identify individual functions/methods that consume a significant amount of CPU time, click on the Table and Flamegraph section.

Within the table, sort the functions/methods by the amount of time they consume in descending order by clicking on the “Self” header.

In this example, it’s clear that the calc3 function is causing a notable bottleneck in our application.

By clicking on the function name calc3 in the table, the function will also be highlighted in the flame graph. This will show the call stack with caller functions at the top and callee functions going toward the bottom.

Next Steps

• Intro to APM
• Databases
• Services
• APM Span List