Logs, metrics, traces and profiles from apps that contain no telemetry code
Three Spring Boot and three .NET 10 services call each other. Agents injected by init containers send every signal to one collector, Alloy, which routes each one to its own backend. Grafana reads all four and links them: a trace opens its logs and the CPU profile of the exact request. A test page instrumented with Grafana Faro starts each trace in the browser, and a PostgreSQL database puts SQL statements into the traces.
The Victoria twin of k8s-observability (Grafana LGTM): same services, same collector, same dashboards, different backends. Try it: vm-demo.x.sunar.cc (test page) · grafana-vm-demo.x.sunar.cc (Grafana).
Select a signal above to follow it alone; select it again to show all. The page's calls go through its nginx to the services with a traceparent from Faro, so a trace starts in the browser; Faro's errors, logs and web vitals go to VictoriaLogs. Both runtimes query PostgreSQL; the agents trace each statement, and the service graph shows postgres as a node. Alloy's spanmetrics and servicegraph connectors turn spans into span metrics and service-graph metrics in VictoriaMetrics, with Tempo's metric names. vmalert evaluates vmalert-rules.yaml; Alertmanager has no receivers, so alerts are read in Grafana.
One pipeline builds the whole environment
Build › Pipelines › Run pipeline on main creates a VM on Proxmox, installs MicroK8s and the stack, deploys PostgreSQL, the six services from their own repos and the test page, and tests the page with Playwright. Every job is idempotent, so a rerun updates what exists. destroy is a separate manual job.
Clones the Ubuntu cloud-init template on pve4 as microk8s-victoria (10.1.11.211) and waits for SSH. Skipped if the VM exists.
Installs MicroK8s 1.36 with dns and hostpath-storage and lets containerd pull from the GitLab registry over HTTP.
current/install.sh: VictoriaMetrics, VictoriaLogs, VictoriaTraces, Pyroscope, vmalert, Alertmanager, then Alloy and Grafana with dashboards.
deploy-apps clones k8s-java-test and k8s-dotnet-test at JAVA_TEST_REF / DOTNET_TEST_REF, checks that their pipelines built the images of that commit, applies their manifests and waits for the rollouts. The test page is plain nginx with its files in ConfigMaps.
Playwright opens the page on its NodePort and checks Faro, both runtimes, traceparent, a mixed chain, /db, /relay and the OOM confirmation. JUnit report in GitLab.
Stops and deletes the VM with its disks. Run it from the pipeline page when the demo is no longer needed.
proxmox/vm.env.
Agents arrive in an init container, not in the app image
The application images hold only the app. Each pod starts an otel-agent init container from a small agent image; it copies the agents into a shared volume, and the env vars in the manifest make the runtime load them. Upgrading an agent means a new agent image tag, not a rebuild of every service.
- Pull the agent imagekubelet pulls
agent-java:otel2.33.1-pyro2.1.4-1, about 36 MB on busybox. - Copy the agents
cp -r /otel/. /agents/fills theotelemptyDir, then the init container exits. - Start the app with /otel read-onlyThe app container mounts the same volume at
/otel. - The runtime loads the agent
JAVA_TOOL_OPTIONS=-javaagent:/otel/otel-javaagent.jar; the Pyroscope extension comes in throughOTEL_JAVAAGENT_EXTENSIONS. - Signals flow to AlloyOTLP to
alloy.observability.svc:4317, profiles to:4040.
init: otel-agent
emptyDir otel
app: service-a
The second build downloads almost nothing
The images are built by the pipelines of own/k8s-java-test and own/k8s-dotnet-test (measured when the builds still lived here). Kaniko keeps its layers in the project registry. Dependencies are restored in their own layer before the source is copied, so a code-only change reuses them. Base images and Maven packages come through the GitLab virtual registries of the own group, which fetch from the internet once and serve from GitLab afterwards.
Job duration, same commit, two runs
Where each build input comes from
--cache=true, stored in each app project's kaniko-cache on the LAN registry 10.1.11.11:5050.
Container virtual registries 1 (Docker Hub), 2 (mcr.microsoft.com) and 3 (ghcr.io) on 10.1.11.11:8082. The Dockerfiles take DOCKER_HUB, MCR and GHCR build args that default to the public hosts. Pulled with the job token.
Maven virtual registry 1 with Maven Central and the Gradle Plugin Portal as upstreams. docker.init.gradle points Gradle at it only when MAVEN_PROXY_URL is set, using the read-only deploy token VIRTUAL_REGISTRY_TOKEN.
GitLab has no NuGet virtual registry; dotnet restore runs in its own cached layer instead.
Built only when the tag in the app repo's k8s/agent-image.txt is missing from the registry. Bump the tag after changing agents/<language>.
From a slow span to the code that burned the CPU
Pyroscope's span processor tags each root span with pyroscope.profile.id, and the profiler labels its CPU samples with the same id. In Grafana, "Profiles for this span" on a trace opens the flame graph of only that request. On Java the processor comes from the Pyroscope extension jar; on .NET from the auto-instrumentation plugin in agents/dotnet/extension of k8s-dotnet-test.
Trace: GET service-a/forward/dotnet-b/forward/service-c/cpu
Profiles for this span
The profiler samples every 10 ms, so a request that takes a few milliseconds may have no samples. /cpu?ms=500 always has enough.
Endpoints that make problems visible
Both runtimes expose the same paths, so one call chain can mix them, for example http://service-a/forward/dotnet-b/forward/service-c/model.
| Endpoint | What it does | Status | Where it shows |
|---|---|---|---|
/model, /forward/** | Returns a model, or calls the next service and passes my-custom-trace-id on. | 200 | trace across servicesservice graph |
/set?time=ms | Adds a delay to every /model call. | 200 | p95 latency |
/error?message=x, /warn?message=x | Logs an error and throws, or logs a warning. | 500 / 200 | log levelerror rate |
/exception/null | Null dereference. | 500 | exception eventexceptions by typeexception profile (.NET) |
/exception/argument | Validation error, caught and returned. | 400 | error log |
/exception/nested | Exception with a timeout as its cause. | 500 | stack trace with cause |
/exception/handled | Caught and logged; the request succeeds. | 200 | error log.NET exceptions/s |
/cpu?ms=500 | Chains SHA-256 hashes for up to 10 s. | 200 | burnCpu in CPU flame graphCPU panels |
/memory?mb=50 | Allocates and drops up to 256 MB per call on the heap. The heap stays below the pod limit (Java 50%, .NET 75%), so heavy load gets some 500s instead of a killed pod. | 200 | allocateChunks in alloc profileGC panels |
/memory?mb=128&retain=true, /memory/release | Keeps the memory until released, in native memory outside the heap (Java Unsafe.allocateMemory, .NET Marshal.AllocHGlobal), so no heap limit stops it before the pod limit; release frees it at once. | 200, then OOMKilled | pod memorypod restart |
/db?sleepMs=700 | INSERTs a visit, runs SELECT pg_sleep(…), SELECTs the visits per service. | 200 | SQL spanspostgres in service graph |
/relay?next=dotnet-b:10000,dotnet-c:5000:db | Each hop waits its time, optionally runs the /db queries, then calls the next hop. Up to 30 s per hop, 60 s in total. | 200 / 400 | gaps in server spansSQL spans at the end |
Five dashboards, provisioned from git
install.sh packs the JSON files into the grafana-dashboards ConfigMap and the Grafana sidecar loads them. Edits made in the UI are not saved; change the JSON instead.
Services
- requests, error rate, p95
- latency p50/p95/p99, HTTP status
- outgoing calls, service map
- recent traces, logs
- CPU flame graph
JVM
- heap and non-heap by pool
- GC pauses and time
- threads by state
- CPU, loaded classes
.NET runtime
- working set, GC heap by generation
- collections, allocation rate
- thread pool, CPU by mode
- exceptions by type, Kestrel
Observability stack
- component up/down
- ingest rate per backend
- Alloy exports and failures
- firing alerts, warn/error logs
Frontend (Faro)
- page loads, sessions
- web vitals p75 (LCP, FCP, TTFB, INP)
- JS exceptions, top errors
- browser call latency, recent traces
Where things live
Every folder below has a README for people and a CLAUDE.md with the rules an AI assistant must follow there.
Operating notes
Changing the stack
- Agent change: edit
agents/<language>in the app repo and bump its tag ink8s/agent-image.txt, or the old image keeps being used. Run the app pipeline, thendeploy-appshere. - The .NET plugin's package versions must match the auto-instrumentation release (
OpenTelemetry.AutoInstrumentation.PluginApi1.19.1,OpenTelemetry.Instrumentation.AspNetCore1.20.2). CORECLR_NOTIFICATION_PROFILERSneeds its trailing;on .NET 9 and later.- In the app repos, edit
k8s/service-a.yaml/k8s/dotnet-a.yamland generate b and c from them; they differ only by name. - Spring's error page lives at
/server-error, because the app maps/erroritself. - Pin every chart and image version; versions are listed in the root README.
- vm-demo.x.sunar.cc and grafana-vm-demo.x.sunar.cc come from
current/demo-ui/pangolin.py; SSO is off for now. SetSSO = Trueand rerun it to require a Pangolin login. - The pod network is
10.244.0.0/16(POD_CIDRinproxmox/vm.env). MicroK8s' default10.1.0.0/16contains the LAN, so pods could not reach the LAN DNS and Grafana could not download its Drilldown apps. It is set when MicroK8s is installed, so changing it needs a new VM. - The demo-ui nginx resolves services at request time and waits up to 90 s for an answer, above the 60 s a
/relaymay take.
Moving to AWS
- Give the Victoria StatefulSets and Pyroscope EBS-backed PVCs sized for the retention.
- Optionally back up VictoriaMetrics to S3 with
vmbackup, using an IAM role (IRSA or Pod Identity).
Open items
- Backups and Pangolin URLs are not set up.
- Alert receivers (for example Telegram) are not configured.