k8s-observability
VictoriaMetrics · VictoriaLogs · VictoriaTraces + Pyroscope · MicroK8s on Proxmox · built by GitLab CI

Logs, metrics, traces and profiles from apps that contain no telemetry code

Three Spring Boot and three .NET 10 services call each other. Agents injected by init containers send every signal to one collector, Alloy, which routes each one to its own backend. Grafana reads all four and links them: a trace opens its logs and the CPU profile of the exact request. A test page instrumented with Grafana Faro starts each trace in the browser, and a PostgreSQL database puts SQL statements into the traces.

The Victoria twin of k8s-observability (Grafana LGTM): same services, same collector, same dashboards, different backends. Try it: vm-demo.x.sunar.cc (test page) · grafana-vm-demo.x.sunar.cc (Grafana).

OTLP gRPC :4317 profiles :4040 Faro /collect :12347 span metrics (Alloy) vmalert → alerts query PostgreSQL 17 /db, /relay · SQL spans Java services Spring Boot 3.3 · JRE 21 service-a → b → c init: OTel Java agent 2.33.1 + Pyroscope extension 2.1.4 no telemetry code in app .NET services ASP.NET Core · .NET 10 dotnet-a → b → c init: OTel .NET auto 1.19.1 + Pyroscope 1.6.1 + plugin no telemetry code in app Alloy 1.21.3 · one collector OTLP receiver memory_limiter → batch drops /actuator spans pyroscope.receive pod logs + self-scrape namespace observability faro.receiver :12347 Browser · demo-ui Grafana Faro 2.13 · :31080 calls carry traceparent ↑ stack pods stdout logs · /metrics VictoriaTraces0.13.0 · traces VictoriaMetrics1.156.0 · metrics VictoriaLogs1.54.0 · logs (OTLP) Pyroscope2.4.3 · profiles Alertmanager 0.35viewed in Grafana Grafana 13.3 · :31300 trace → logs span metrics span → profile service map 5 dashboards folder "Demo" PVCslocal disk, 7 daysno object storage

Select a signal above to follow it alone; select it again to show all. The page's calls go through its nginx to the services with a traceparent from Faro, so a trace starts in the browser; Faro's errors, logs and web vitals go to VictoriaLogs. Both runtimes query PostgreSQL; the agents trace each statement, and the service graph shows postgres as a node. Alloy's spanmetrics and servicegraph connectors turn spans into span metrics and service-graph metrics in VictoriaMetrics, with Tempo's metric names. vmalert evaluates vmalert-rules.yaml; Alertmanager has no receivers, so alerts are read in Grafana.

.gitlab-ci.yml · started by hand

One pipeline builds the whole environment

Build › Pipelines › Run pipeline on main creates a VM on Proxmox, installs MicroK8s and the stack, deploys PostgreSQL, the six services from their own repos and the test page, and tests the page with Playwright. Every job is idempotent, so a rerun updates what exists. destroy is a separate manual job.

1 · vm
vm13 s

Clones the Ubuntu cloud-init template on pve4 as microk8s-victoria (10.1.11.211) and waits for SSH. Skipped if the VM exists.

2 · k8s
microk8s13 s

Installs MicroK8s 1.36 with dns and hostpath-storage and lets containerd pull from the GitLab registry over HTTP.

3 · stack
observability29 s

current/install.sh: VictoriaMetrics, VictoriaLogs, VictoriaTraces, Pyroscope, vmalert, Alertmanager, then Alloy and Grafana with dashboards.

4 · app
deploy-postgres25 s
deploy-appsDEPLOY_APPS_S
deploy-demo-ui13 s

deploy-apps clones k8s-java-test and k8s-dotnet-test at JAVA_TEST_REF / DOTNET_TEST_REF, checks that their pipelines built the images of that commit, applies their manifests and waits for the rollouts. The test page is plain nginx with its files in ConfigMaps.

5 · test
test-ui282 s

Playwright opens the page on its NodePort and checks Faro, both runtimes, traceparent, a mixed chain, /db, /relay and the OOM confirmation. JUnit report in GitLab.

6 · destroy
destroymanual

Stops and deletes the VM with its disks. Run it from the pipeline page when the demo is no longer needed.

Durations from pipeline 5161 (warm cache); deploy-demo-ui and test-ui from 5162, deploy-postgres from 5164, where 4 of test-ui's minutes went to the first pull of the Playwright image (the tests take 8 s). Secrets are project CI/CD variables; VM size and network come from proxmox/vm.env.
initContainers · emptyDir · env

Agents arrive in an init container, not in the app image

The application images hold only the app. Each pod starts an otel-agent init container from a small agent image; it copies the agents into a shared volume, and the env vars in the manifest make the runtime load them. Upgrading an agent means a new agent image tag, not a rebuild of every service.

  1. Pull the agent imagekubelet pulls agent-java:otel2.33.1-pyro2.1.4-1, about 36 MB on busybox.
  2. Copy the agentscp -r /otel/. /agents/ fills the otel emptyDir, then the init container exits.
  3. Start the app with /otel read-onlyThe app container mounts the same volume at /otel.
  4. The runtime loads the agentJAVA_TOOL_OPTIONS=-javaagent:/otel/otel-javaagent.jar; the Pyroscope extension comes in through OTEL_JAVAAGENT_EXTENSIONS.
  5. Signals flow to AlloyOTLP to alloy.observability.svc:4317, profiles to :4040.
pod service-a-deployment-…

init: otel-agent

agent-java:otel2.33.1-pyro2.1.4-1
Completed

emptyDir otel

/agents → /otel

app: service-a

testservice:<commit sha>
/otel read-onlyagent loaded
→ Alloytracesmetricslogsprofiles

        
Select a step to jump to it.
kaniko · GitLab virtual registries · app repos' pipelines

The second build downloads almost nothing

The images are built by the pipelines of own/k8s-java-test and own/k8s-dotnet-test (measured when the builds still lived here). Kaniko keeps its layers in the project registry. Dependencies are restored in their own layer before the source is copied, so a code-only change reuses them. Base images and Maven packages come through the GitLab virtual registries of the own group, which fetch from the internet once and serve from GitLab afterwards.

Job duration, same commit, two runs

pipeline 5160, empty cachepipeline 5161, warm cache
build-image
447 s
56 s
build-agent-dotnet
294 s
9 s · skipped
build-image-dotnet
119 s
77 s
build-agent-java
93 s
7 s · skipped
0100200300400500 s
The Java cold run also filled the Maven cache: Gradle fetched every dependency through the proxy once.

Where each build input comes from

image layers

--cache=true, stored in each app project's kaniko-cache on the LAN registry 10.1.11.11:5050.

base images

Container virtual registries 1 (Docker Hub), 2 (mcr.microsoft.com) and 3 (ghcr.io) on 10.1.11.11:8082. The Dockerfiles take DOCKER_HUB, MCR and GHCR build args that default to the public hosts. Pulled with the job token.

Maven / Gradle

Maven virtual registry 1 with Maven Central and the Gradle Plugin Portal as upstreams. docker.init.gradle points Gradle at it only when MAVEN_PROXY_URL is set, using the read-only deploy token VIRTUAL_REGISTRY_TOKEN.

NuGet

GitLab has no NuGet virtual registry; dotnet restore runs in its own cached layer instead.

agent images

Built only when the tag in the app repo's k8s/agent-image.txt is missing from the registry. Bump the tag after changing agents/<language>.

pyroscope.profile.id

From a slow span to the code that burned the CPU

Pyroscope's span processor tags each root span with pyroscope.profile.id, and the profiler labels its CPU samples with the same id. In Grafana, "Profiles for this span" on a trace opens the flame graph of only that request. On Java the processor comes from the Pyroscope extension jar; on .NET from the auto-instrumentation plugin in agents/dotnet/extension of k8s-dotnet-test.

Trace: GET service-a/forward/dotnet-b/forward/service-c/cpu

service-a GET /forward/** · root dotnet-b GET /forward/{**target} service-c GET /cpu · burnCpu CPU samples every sample carries pyroscope.profile.id of the root span

Profiles for this span

total
java.lang.Thread.run
org/apache/tomcat … DispatcherServlet.doDispatch
LoadController.cpujackson
LoadController.burnCpu.
MessageDigest.digest (SHA-256).
Open a trace in Explore › VictoriaTraces.
On a root span, select "Profiles for this span".
Grafana asks Pyroscope for the samples with that profile id and draws them.

The profiler samples every 10 ms, so a request that takes a few milliseconds may have no samples. /cpu?ms=500 always has enough.

same endpoints in Java and .NET

Endpoints that make problems visible

Both runtimes expose the same paths, so one call chain can mix them, for example http://service-a/forward/dotnet-b/forward/service-c/model.

EndpointWhat it doesStatusWhere it shows
/model, /forward/**Returns a model, or calls the next service and passes my-custom-trace-id on.200trace across servicesservice graph
/set?time=msAdds a delay to every /model call.200p95 latency
/error?message=x, /warn?message=xLogs an error and throws, or logs a warning.500 / 200log levelerror rate
/exception/nullNull dereference.500exception eventexceptions by typeexception profile (.NET)
/exception/argumentValidation error, caught and returned.400error log
/exception/nestedException with a timeout as its cause.500stack trace with cause
/exception/handledCaught and logged; the request succeeds.200error log.NET exceptions/s
/cpu?ms=500Chains SHA-256 hashes for up to 10 s.200burnCpu in CPU flame graphCPU panels
/memory?mb=50Allocates and drops up to 256 MB per call on the heap. The heap stays below the pod limit (Java 50%, .NET 75%), so heavy load gets some 500s instead of a killed pod.200allocateChunks in alloc profileGC panels
/memory?mb=128&retain=true, /memory/releaseKeeps the memory until released, in native memory outside the heap (Java Unsafe.allocateMemory, .NET Marshal.AllocHGlobal), so no heap limit stops it before the pod limit; release frees it at once.200, then OOMKilledpod memorypod restart
/db?sleepMs=700INSERTs a visit, runs SELECT pg_sleep(…), SELECTs the visits per service.200SQL spanspostgres in service graph
/relay?next=dotnet-b:10000,dotnet-c:5000:dbEach hop waits its time, optionally runs the /db queries, then calls the next hop. Up to 30 s per hop, 60 s in total.200 / 400gaps in server spansSQL spans at the end
current/observability/dashboards · folder "Demo"

Five dashboards, provisioned from git

install.sh packs the JSON files into the grafana-dashboards ConfigMap and the Grafana sidecar loads them. Edits made in the UI are not saved; change the JSON instead.

Services

demo-services · per service
  • requests, error rate, p95
  • latency p50/p95/p99, HTTP status
  • outgoing calls, service map
  • recent traces, logs
  • CPU flame graph

JVM

demo-jvm · per job
  • heap and non-heap by pool
  • GC pauses and time
  • threads by state
  • CPU, loaded classes

.NET runtime

demo-dotnet · per job
  • working set, GC heap by generation
  • collections, allocation rate
  • thread pool, CPU by mode
  • exceptions by type, Kestrel

Observability stack

demo-stack
  • component up/down
  • ingest rate per backend
  • Alloy exports and failures
  • firing alerts, warn/error logs

Frontend (Faro)

demo-frontend
  • page loads, sessions
  • web vitals p75 (LCP, FCP, TTFB, INP)
  • JS exceptions, top errors
  • browser call latency, recent traces
git.sunar.cc/own/k8s-observability-victoriametrics

Where things live

Every folder below has a README for people and a CLAUDE.md with the rules an AI assistant must follow there.

.gitlab-ci.ymlthe pipeline: vm, k8s, stack, app, test, destroy
proxmox/create, provision and destroy the MicroK8s VM; vm.env defaults
current/install.shinstalls the stack into the namespace observability
current/observability/Helm values and manifests of every component, Alloy pipeline, alert rules
  dashboards/Grafana dashboards as JSON
current/apps/deploy.sh: deploys the services from own/k8s-java-test and own/k8s-dotnet-test
own/k8s-java-test(own repo) Spring Boot service, agents/java, k8s/service-a/b/c
own/k8s-dotnet-test(own repo) .NET 10 service, agents/dotnet (plugin in extension/), k8s/dotnet-a/b/c
current/postgres/PostgreSQL for /db and /relay, its schema and install script
current/demo-ui/test page (nginx + Faro), Playwright tests, Pangolin script
docs/this page as a single HTML file
things that are easy to get wrong

Operating notes

Changing the stack

  1. Agent change: edit agents/<language> in the app repo and bump its tag in k8s/agent-image.txt, or the old image keeps being used. Run the app pipeline, then deploy-apps here.
  2. The .NET plugin's package versions must match the auto-instrumentation release (OpenTelemetry.AutoInstrumentation.PluginApi 1.19.1, OpenTelemetry.Instrumentation.AspNetCore 1.20.2).
  3. CORECLR_NOTIFICATION_PROFILERS needs its trailing ; on .NET 9 and later.
  4. In the app repos, edit k8s/service-a.yaml / k8s/dotnet-a.yaml and generate b and c from them; they differ only by name.
  5. Spring's error page lives at /server-error, because the app maps /error itself.
  6. Pin every chart and image version; versions are listed in the root README.
  7. vm-demo.x.sunar.cc and grafana-vm-demo.x.sunar.cc come from current/demo-ui/pangolin.py; SSO is off for now. Set SSO = True and rerun it to require a Pangolin login.
  8. The pod network is 10.244.0.0/16 (POD_CIDR in proxmox/vm.env). MicroK8s' default 10.1.0.0/16 contains the LAN, so pods could not reach the LAN DNS and Grafana could not download its Drilldown apps. It is set when MicroK8s is installed, so changing it needs a new VM.
  9. The demo-ui nginx resolves services at request time and waits up to 90 s for an answer, above the 60 s a /relay may take.

Moving to AWS

  1. Give the Victoria StatefulSets and Pyroscope EBS-backed PVCs sized for the retention.
  2. Optionally back up VictoriaMetrics to S3 with vmbackup, using an IAM role (IRSA or Pod Identity).

Open items

  1. Backups and Pangolin URLs are not set up.
  2. Alert receivers (for example Telegram) are not configured.