profiling.md

Profiling Adaguc-server

This page describes how to use linux perf to profile adagucserver. perf records events that can be used to generate a flamegraph or callgraph.

These instructions are tested from within a docker container using the docker compose setup. These instructions are not tested when running Adaguc in a cloud environment.

Prerequisites

Install flamegraph tools:

git clone https://github.com/brendangregg/FlameGraph

Make sure host is allowed to capture perf events:

docker run --rm --privileged --net=host --pid=host alpine \
  sysctl -w kernel.perf_event_paranoid=-1 \
  kernel.kptr_restrict=0

Edit ./Docker/docker-compose.yml to create profiling build and allow capturing perf events in the container:

  adaguc-server:
    build:
      context: ..
      args:
        BUILD_TYPE: --profile
      extra_hosts:
      - "host.docker.internal:host-gateway"
    cap_add:
      - SYS_ADMIN
      - PERFMON
      - SYS_PTRACE
    security_opt:
      - seccomp=unconfined

Then restart the containers:

cd ./Docker/
docker compose down
docker compose up -d --build

Inside docker container

exec into the container as root.

docker exec -u0 --privileged -it my-adaguc-server /bin/bash

The next steps will be performed as root from within the adaguc-server container.

Install linux-perf and debug symbols of adaguc-server's dependencies.

cat << EOF > /etc/apt/sources.list.d/debug.list
deb http://deb.debian.org/debian-debug bookworm-debug main contrib non-free
deb http://deb.debian.org/debian-debug bookworm-backports-debug main contrib non-free
deb http://deb.debian.org/debian-debug bookworm-proposed-updates-debug main contrib non-free
deb http://deb.debian.org/debian-debug bookworm-backports-sloppy-debug main contrib non-free
deb http://security.debian.org bookworm-security main
EOF

apt update
apt install procps linux-perf libpng16-16 libpng16-16-dbgsym libproj25-dbgsym libsqlite3-0-dbgsym libc6-dbg

Profile

Setup environment variables for adaguc:

export ADAGUC_TMP=/tmp
export ADAGUC_CONFIG=/adaguc/adaguc-server-config.xml

The goal is to profile the C++ application which resides in ./bin/adagucserver. The binary adagucserver normally gets spawned by the python webserver, and the binary exits as fast as possible.

To be able to profile adagucserver, we can spawn it through perf instead of through the python webserver. This requires us to set the QUERY_STRING environment variable, so we can profile the request that we want (e.g. WMS GetMap).

You can use adaguc-viewer, open your browser's developer tools network tab, perform the request you want to profile, and copy the URL parameters of your request. Set the QUERY_STRING to these URL parameters. For example:

export QUERY_STRING="DATASET=mydataset&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&LAYERS=air-temperature-hagl&WIDTH=1008&HEIGHT=995&CRS=EPSG%3a3857&BBOX=-7470893.149487414,703507.8780614401,-5525657.400627586,2623656.26111256&STYLES=temperature%2fgeneric&FORMAT=image/png&TRANSPARENT=TRUE&time=2023-01-26T00%3a00%3a00Z&DIM_forecast_reference_time=2023-01-24T00%3a00%3a00Z&DIM_temp_at_hagl=2"

If you execute adagucserver manually, it will write the (binary) response to the terminal. You can use this to verify if this is the request you want.

./bin/adagucserver

Profile

With perf record, we create a perf.out file by sampling ./bin/adagucserver. We then post process the perf.data file. This creates a perf.filt file which we will use for visualization.

There are different recording modes. If --call-graph fp does not work well, try --call-graph dwarf.

perf record -o perf.data -F max -g --call-graph fp ./bin/adagucserver
perf script -i perf.data | c++filt > perf.filt

Profiling is done!

Visualize

The next section is about visualizing the recorded data. This is done on your host system (non-root). Make sure to copy your generated profile data by running the following command from the host OS:

docker cp my-adaguc-server:/adaguc/adaguc-server-master/perf.filt ./;

The "filtered" data needs to be "collapsed" by a tool provided in the flamegraph repo. It can then be visualized using a website like speedscope:

cat perf.filt | ~/src/FlameGraph/stackcollapse-perf.pl > perf.collapsed
firefox https://www.speedscope.app

You can also turn the collapsed data into an interactive .svg file:

cat perf.filt | ~/src/FlameGraph/stackcollapse-perf.pl | ~/src/FlameGraph/flamegraph.pl > perf.svg
firefox perf.svg

Callgraph

It is possible to visualize the perf data as a callgraph.

It seems the results are not always as useful (broken/unclear graphs). For completeness, this is how you can generate callgraphs.

Install graphviz and gprof2dot dependencies. On Ubuntu/Debian-likes:

apt install graphviz
pip install gprof2dot

Generate callgraph and view in your browser:

cat perf.data | gprof2dot.py -f perf --skew=0.1 -e0 -n0 | dot -Tsvg -o callgraph.svg
firefox callgraph.svg

This generates a massive graph. Zoomed out it looks a little bit like this (this is about 1/4th of the diagram):