Go Profiling and Debugging — A Deep, Practitioner-Level Guide

Written from the perspective of a Go principal engineer. This document covers the full toolchain and mental models needed to diagnose CPU, memory, concurrency, and latency problems in production and development Go systems.

---

Table of Contents

1. Philosophy: Measure, Don’t Guess
2. The pprof Ecosystem
3. CPU Profiling
4. Memory Profiling
5. Goroutine Profiling & Leak Detection
6. Block and Mutex Profiling (Contention)
7. The Execution Tracer (go tool trace)
8. Benchmarking with testing.B and benchstat
9. Escape Analysis and Compiler Diagnostics
10. Garbage Collector Tuning
11. The runtime/metrics Package
12. Debugging with Delve
13. Race Detection
14. Deadlock and Goroutine Diagnostics
15. Continuous Profiling in Production
16. Flame Graphs and Visualization
17. Common Pitfalls Catalog
18. Case Studies
19. Checklist: Investigating a Production Incident
20. Reference Commands Cheat Sheet

---

1. Philosophy: Measure, Don’t Guess

Go was designed with observability as a first-class citizen — the runtime ships with pprof, trace, and runtime/metrics built directly into the standard library. This is not an accident: Go’s authors (Pike, Griesemer, Thompson, and later the concurrency-heavy runtime team) anticipated that goroutines, channels, and the GC would create failure modes that traditional debuggers (gdb-style, step-through) are poorly suited for.

The core principle: never optimize or “fix” a performance problem based on intuition. Go gives you sampling profilers, execution tracers, and statistical benchmarking tools specifically so that you replace guesses with data. A senior Go engineer’s instinct should be: “what does the profile say?” before “I think the bottleneck is X.”

Three categories of problems dominate Go production issues:

• CPU-bound hot paths — inefficient algorithms, excessive allocation causing GC pressure, reflection-heavy serialization.
• Memory issues — leaks (usually goroutine-retained references, not classic C-style leaks), unbounded caches, slice-retention bugs.
• Concurrency issues — goroutine leaks, lock contention, channel deadlocks, scheduler latency (GOMAXPROCS misconfiguration).

Each has a dedicated tool. Knowing which tool answers which question is 80% of the battle.

---

2. The pprof Ecosystem

pprof is Go’s profiling format and toolchain, derived from Google’s internal gperftools/pprof C++ tool, adapted for Go’s runtime. There are two entry points:

2.1 runtime/pprof

Low-level API for embedding profile collection directly into a program — useful for CLI tools, batch jobs, or short-lived processes where you want a profile written to disk on demand.

import (
    "os"
    "runtime/pprof"
)

func main() {
    f, _ := os.Create("cpu.prof")
    defer f.Close()

    pprof.StartCPUProfile(f)
    defer pprof.StopCPUProfile()

    doWork()
}

For heap:

f, _ := os.Create("heap.prof")
defer f.Close()
runtime.GC() // get up-to-date statistics
pprof.WriteHeapProfile(f)

2.2 net/http/pprof

The far more common entry point for long-running servers. Importing it for its side effect registers HTTP handlers under /debug/pprof/:

import _ "net/http/pprof"

func main() {
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    // ... rest of the application
}

Security note: Never expose /debug/pprof/ on a public-facing port. It leaks internal source paths, memory layout, and can be a DoS vector (CPU profile requests hold a CPU core for 30s+ by default). Bind it to a localhost-only or internal-mesh port, or protect it behind auth middleware.

Available endpoints:

Endpoint	Description
/debug/pprof/profile?seconds=30	CPU profile (blocking, samples for N seconds)
/debug/pprof/heap	Heap allocations snapshot
/debug/pprof/goroutine	Stack traces of all current goroutines
/debug/pprof/block	Goroutine blocking on synchronization primitives
/debug/pprof/mutex	Mutex contention
/debug/pprof/threadcreate	OS thread creation stacks
/debug/pprof/allocs	All allocations since program start (not just live)
/debug/pprof/trace?seconds=5	Execution trace (different tool: go tool trace)

2.3 Fetching and Analyzing

go tool pprof http://localhost:6060/debug/pprof/heap
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

This drops you into an interactive REPL:

(pprof) top10
(pprof) list functionName
(pprof) web        # opens SVG call graph in browser (requires graphviz)
(pprof) png > out.png
(pprof) traces
(pprof) peek regexPattern

Key pprof commands every Go engineer should memorize:

• top — sorted list of functions by flat/cumulative resource usage.
• list <func> — annotated source code showing per-line cost.
• web / svg — call-graph visualization (requires Graphviz: apt install graphviz).
• traces — raw sample stacks.
• peek <func> — callers/callees of a specific function.
• -base flag — diff two profiles (go tool pprof -base old.prof new.prof), essential for A/B comparison.

---

3. CPU Profiling

CPU profiles are statistical samples, not exact traces. The Go runtime interrupts execution ~100 times per second (runtime.SetCPUProfileRate, default 100Hz) via SIGPROF and records the current stack. This means:

• Very short-lived functions may be under-sampled or missed entirely.
• Profiles need enough wall-clock duration (typically 30s+) to be statistically meaningful.
• Inlined functions may not appear as separate frames unless you build with -gcflags="-l" to disable inlining (only for diagnostic builds, never production).

3.1 Flat vs. Cumulative

• Flat: time spent in that function’s own code, excluding callees.
• Cumulative: time spent in that function plus everything it calls.

A function with high cumulative but low flat time is a “router” — the real cost is downstream. A function with high flat time is where actual CPU cycles burn — look here first for algorithmic optimization.

3.2 Practical Workflow

go test -bench=. -cpuprofile=cpu.prof -benchtime=5s
go tool pprof cpu.prof
(pprof) top20 -cum
(pprof) list HotFunction

Look for:

• Unexpected allocations inside hot loops (runtime.mallocgc appearing high in the profile means allocation pressure, not just “CPU work”).
• runtime.memmove, runtime.typedmemmove — often from unnecessary copies of large structs; pass pointers instead.
• Reflection usage (reflect.Value.*) — common in JSON/encoding-heavy code paths; consider codegen (e.g., easyjson, ffjson) or manual marshaling for hot paths.
• runtime.mapaccess*/runtime.mapassign dominating — map-heavy algorithms; consider arrays/slices with known indices, or sync.Map only if the access pattern truly fits its documented use case (mostly-read, disjoint keys per goroutine).

3.3 GOMAXPROCS Interplay

CPU profile percentages are relative to GOMAXPROCS × wall-clock time. On a machine with more cores than the container’s cgroup limit allows, Go historically over-detected GOMAXPROCS from runtime.NumCPU(), leading to more OS threads than usable cores and increased scheduling overhead. Since Go 1.5 you can set it explicitly; since Go 1.21+ there’s improved (though still imperfect until go.uber.org/automaxprocs-style solutions or the built-in cgroup-aware GOMAXPROCS in Go 1.25+) handling. Always verify GOMAXPROCS matches your container’s CPU quota — a very common, very silent source of latency spikes in Kubernetes deployments.

---

4. Memory Profiling

Go’s heap profiler is also sampling-based by default (1 sample per 512KB allocated, controlled by runtime.MemProfileRate). Two critical profile “views” exist:

4.1 inuse_space vs alloc_space

• inuse_space (default view): memory currently live/reachable — use this to find leaks or unexpectedly large live heaps.
• alloc_space: total cumulative bytes allocated since process start, regardless of whether they’ve been freed — use this to find allocation-heavy hot paths that pressure the GC, even if the memory doesn’t leak.

go tool pprof -inuse_space http://localhost:6060/debug/pprof/heap
go tool pprof -alloc_space http://localhost:6060/debug/pprof/heap

Analogously, -inuse_objects and -alloc_objects give you object counts rather than bytes — useful for finding pathological small-object churn even when total bytes look small.

4.2 Diagnosing a Memory Leak

Since Go is garbage collected, “leaks” almost always mean unintentionally retained references, not malloc without free. Common causes:

1. Goroutine leaks — a goroutine blocked forever on a channel send/receive holds its entire stack and any captured variables alive forever. This is the #1 cause of Go memory leaks in production.
2. Slice sub-slicing retaining the backing array — smallSlice := bigSlice[:10] keeps the entire backing array of bigSlice alive as long as smallSlice is referenced. Fix: copy the needed data with append([]T(nil), bigSlice[:10]...) if you need to release the rest.
3. Global caches/maps without eviction — map[string]*Session that’s never cleaned; use TTL eviction, container/list + map for LRU, or a library like groupcache/ristretto.
4. time.Timer/time.Ticker not stopped — retained by the runtime timer heap.
5. Closures capturing large structs by reference unintentionally, e.g., inside long-lived callback registries.

4.3 Diffing Heap Profiles Over Time

The single most reliable leak-hunting technique:

curl -o heap1.prof http://localhost:6060/debug/pprof/heap
# ... wait 10 minutes under load ...
curl -o heap2.prof http://localhost:6060/debug/pprof/heap
go tool pprof -base heap1.prof heap2.prof
(pprof) top20

This shows exactly what grew between the two snapshots, filtering out steady-state noise — dramatically more useful than a single snapshot.

4.4 debug.FreeOSMemory() and GODEBUG=madvdontneed=1

Go doesn’t always return freed memory to the OS immediately (visible as high RSS vs low live heap). debug.FreeOSMemory() forces a GC + release cycle — useful for diagnosis, not a production fix. On Linux, GODEBUG=madvdontneed=1 changes memory-release syscall behavior (older kernels); modern Go (1.16+) defaults appropriately per-OS.

---

5. Goroutine Profiling & Leak Detection

go tool pprof http://localhost:6060/debug/pprof/goroutine

Or for a raw, human-readable dump (extremely useful during an incident, no tooling required):

curl http://localhost:6060/debug/pprof/goroutine?debug=2

debug=2 gives full stack traces per goroutine, grouped — this is often the fastest way to spot “1,400 goroutines stuck at the same chan receive line” during an incident.

5.1 Reading the Output

Goroutines are grouped by identical stack trace, with a count prefix:

goroutine profile: total 1432
1400 @ 0x43a1b2 0x43f...
#   0x... internal/poll.runtime_pollWait+0x...
#   0x... net/http.(*persistConn).readLoop+0x...

A count of 1400 goroutines stuck in the same location is a near-certain leak signature — normal operation rarely has that much stack-trace duplication at rest.

5.2 Common Leak Patterns

• Unbuffered channel send with no receiver — a goroutine that writes to a channel nobody reads from anymore (e.g., the reader errored out and returned early) blocks forever.
• context.Context never canceled — child goroutines waiting on <-ctx.Done() that never fires because the parent forgot to call cancel(). Always defer cancel() immediately after context.WithCancel/WithTimeout/WithDeadline.
• select without a default and no timeout path in a fire-and-forget goroutine.
• HTTP response bodies not closed/drained — leaks the underlying connection’s goroutine machinery and prevents connection reuse.

5.3 Programmatic Goroutine Count Monitoring

runtime.NumGoroutine()

Exporting this as a Prometheus gauge and alerting on sustained upward trends (not just absolute value, since baseline varies) is one of the highest-value, lowest-cost production safeguards you can add to a Go service.

---

6. Block and Mutex Profiling (Contention)

These are off by default because they add overhead; enable explicitly:

import "runtime"

func init() {
    runtime.SetBlockProfileRate(1)       // sample every blocking event
    runtime.SetMutexProfileFraction(1)   // sample every mutex contention event
}

• Block profile (/debug/pprof/block): time goroutines spend blocked on channel ops, sync.Mutex.Lock, sync.WaitGroup.Wait, network I/O wait, etc. Great for finding serialization points in supposedly-concurrent code.
• Mutex profile (/debug/pprof/mutex): specifically contended mutexes — where multiple goroutines fight over the same lock. A high count here often means: lock granularity is too coarse (one giant mutex protecting an entire struct instead of per-field or sharded locks), or a hot path holds a lock longer than necessary (e.g., doing I/O while holding a lock).

Production tip: SetMutexProfileFraction(1) samples every event and can be expensive under very high contention; start with a fraction like 100 (every 100th event) in production, drop to 1 only during a targeted diagnostic window.

6.1 Fixing Contention

• Reduce critical section size — do expensive work outside the lock, only mutate shared state inside.
• Shard locks (e.g., 16 or 32 buckets each with their own mutex, hashed by key) instead of one global lock — the pattern used internally by sync.Map and many high-throughput caches.
• Consider sync/atomic for simple counters instead of a mutex-protected int.
• Consider sync.RWMutex if reads vastly outnumber writes — but beware: RWMutex has higher overhead per-lock than Mutex when contention is low, so it’s not a free win.

---

7. The Execution Tracer (go tool trace)

While pprof answers “where is CPU/memory spent,” the execution tracer answers “what happened over time, across every goroutine, the scheduler, and the GC, on a microsecond timeline.” It is the single best tool for diagnosing latency problems (as opposed to throughput/CPU problems).

import (
    "os"
    "runtime/trace"
)

func main() {
    f, _ := os.Create("trace.out")
    defer f.Close()
    trace.Start(f)
    defer trace.Stop()

    doWork()
}

Or via HTTP: curl http://localhost:6060/debug/pprof/trace?seconds=5 > trace.out

Then:

go tool trace trace.out

This opens a browser-based visualization with several critical views:

• View trace — a per-P (processor) timeline showing goroutine execution, GC pauses, syscalls, and network waits, with microsecond resolution. You can visually spot GC STW (stop-the-world) pauses, goroutine scheduling delays, and syscall blocking.
• Goroutine analysis — per-goroutine breakdown of execution/wait/block/syscall time.
• Network/Sync/Syscall blocking profiles — same data as pprof’s block profile but with a temporal view.
• Minimum mutator utilization (MMU) — shows what fraction of CPU time was available to your actual program code vs. GC, over sliding time windows. Critical for latency-sensitive services (trading systems, real-time APIs).

7.1 What to Look For

• Long gaps between goroutine “runnable” and “running” — indicates scheduler contention, usually from GOMAXPROCS too low relative to concurrent work, or too many goroutines fighting for limited Ps.
• Frequent, long GC STW pauses — visible as synchronized gaps across all Ps. Modern Go GC (since 1.5’s concurrent collector) keeps STW phases in the microsecond range under normal conditions; if you see multi-millisecond STW consistently, investigate heap size, allocation rate, or GOGC settings.
• Syscall-heavy goroutines blocking Ps — a goroutine in a blocking syscall detaches its P (handed to another M/thread) but excessive syscalls (e.g., unbuffered file/network I/O) can still show up as scheduling pressure.

go tool trace has a steeper learning curve than pprof but is indispensable for tail-latency (p99/p999) investigations that CPU profiling alone cannot explain.

---

8. Benchmarking with testing.B and benchstat

Profiling tells you where time goes; benchmarking tells you whether a change actually helped, with statistical rigor.

func BenchmarkParseJSON(b *testing.B) {
    data := loadTestData()
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _, _ = Parse(data)
    }
}

Run with:

go test -bench=BenchmarkParseJSON -benchmem -count=10 -cpuprofile=cpu.prof -memprofile=mem.prof > bench.txt

• -benchmem reports allocs/op and B/op — often more important than ns/op since allocation rate drives GC pressure.
• -count=10 runs the benchmark 10 times so you can apply statistics rather than trust a single noisy run.
• -cpuprofile/-memprofile attach profiling directly to the benchmark run, letting you go tool pprof the exact hot path under controlled conditions (far cleaner signal than profiling a live, noisy production server).

8.1 benchstat

go install golang.org/x/perf/cmd/benchstat@latest
go test -bench=. -count=10 ./... > old.txt
# make your change
go test -bench=. -count=10 ./... > new.txt
benchstat old.txt new.txt

benchstat computes statistical significance (via a non-parametric test) between two sets of benchmark runs, telling you whether an observed improvement is real or just noise — critical discipline before claiming “this optimization helped.”

8.2 Sub-benchmarks and Table-Driven Perf Tests

func BenchmarkEncode(b *testing.B) {
    sizes := []int{10, 100, 1000, 10000}
    for _, n := range sizes {
        b.Run(fmt.Sprintf("n=%d", n), func(b *testing.B) {
            data := makeData(n)
            b.ResetTimer()
            for i := 0; i < b.N; i++ {
                Encode(data)
            }
        })
    }
}

This reveals algorithmic complexity empirically (does time scale linearly, quadratically?) rather than assuming Big-O from reading code.

---

9. Escape Analysis and Compiler Diagnostics

Understanding why something allocates on the heap (and therefore pressures the GC) instead of the stack is essential for allocation-sensitive hot paths.

go build -gcflags="-m -m" ./...

Output like:

./main.go:12:6: moved to heap: x
./main.go:20:9: &y escapes to heap

Common causes of unwanted heap escapes:

• Returning a pointer to a local variable (necessary and fine in most cases, but be aware it forces heap allocation).
• Passing a value to an interface{}/any parameter — this typically boxes the value onto the heap unless the compiler can prove otherwise.
• Storing a pointer into a struct/slice/map whose lifetime outlives the current stack frame.
• Closures capturing variables by reference when the closure itself escapes (e.g., stored in a struct or passed to go func(){}).
• Variable size not known at compile time (e.g., slice length from a variable) preventing stack allocation.

For extremely hot paths (serialization libraries, low-latency trading, high-QPS routers), engineers deliberately restructure code to keep small, short-lived values on the stack — but this should always be validated with -gcflags="-m" and benchmarks, not assumed.

---

10. Garbage Collector Tuning

Go’s GC is a concurrent, tri-color mark-and-sweep collector. Two primary knobs:

10.1 GOGC (heap growth ratio)

Default 100: the GC triggers when the heap has grown 100% since the last collection’s live heap size. Lower values (GOGC=50) trigger GC more often, reducing peak memory but increasing CPU overhead from more frequent collections. Higher values (GOGC=200 or more) trade memory for less GC CPU overhead — good for CPU-bound batch jobs with memory headroom.

debug.SetGCPercent(50)

10.2 GOMEMLIMIT (Go 1.19+)

A soft memory limit (in bytes) that the runtime uses to trigger more aggressive GC as usage approaches the limit — critical in containerized environments to avoid OOM-kill from exceeding the cgroup memory limit. This is the modern, recommended complement (or even replacement) to tuning GOGC blindly.

debug.SetMemoryLimit(500 << 20) // 500 MiB soft limit

Or via environment: GOMEMLIMIT=500MiB.

Best practice for containers: set GOMEMLIMIT to roughly 80-90% of the container’s memory limit, and leave GOGC at a sane default (or slightly relaxed like 150-200) — this lets the runtime use available memory efficiently in the common case while still providing a hard backstop against OOM kills under memory pressure spikes.

10.3 GC Pacing and Diagnostics

GODEBUG=gctrace=1 ./myapp

Prints one line per GC cycle:

gc 15 @6.001s 2%: 0.02+1.2+0.01 ms clock, 0.1+0.5/1.1/0+0.08 ms cpu, 4->5->3 MB, 6 MB goal, 8 P

Key fields: cumulative GC CPU percentage (should typically stay well under 25% for healthy services — a documented, informal Go team rule of thumb), heap-before→heap-after→live-heap, and heap goal. Sustained GC CPU% above ~30% strongly suggests either excessive allocation rate (fix at the allocation source, not via GOGC alone) or a GOGC/GOMEMLIMIT misconfiguration.

---

11. The runtime/metrics Package

Since Go 1.16, runtime/metrics provides a stable, structured API to pull dozens of internal runtime metrics (superseding the older, less structured runtime.MemStats for new code) — ideal for wiring into Prometheus/OpenTelemetry exporters.

import "runtime/metrics"

samples := []metrics.Sample{
    {Name: "/gc/heap/allocs:bytes"},
    {Name: "/sched/goroutines:goroutines"},
    {Name: "/gc/pauses:seconds"},
}
metrics.Read(samples)
for _, s := range samples {
    fmt.Println(s.Name, s.Value)
}

Run metrics.All() to discover every metric name available in your Go version — the set has grown significantly release over release (scheduler latency histograms, GC pause distributions, memory-class breakdowns).

Wiring these into your existing observability stack (rather than relying solely on /debug/pprof/ for ad-hoc investigation) turns profiling data into continuously monitored, alertable signal — the difference between finding out about a leak from a profile during an incident versus catching the trend a week earlier from a dashboard.

---

12. Debugging with Delve

Delve (dlv) is the de facto standard Go debugger — purpose-built for Go’s runtime (goroutines, channels, defer/panic/recover), unlike generic gdb, which understands very little of Go’s runtime internals.

12.1 Basic Usage

dlv debug ./cmd/myapp -- --flag=value

Inside the debugger:

(dlv) break main.processRequest
(dlv) continue
(dlv) print req
(dlv) locals
(dlv) goroutines
(dlv) goroutine 12 stack
(dlv) next
(dlv) step
(dlv) stepout

12.2 Attaching to a Running Process

dlv attach <pid>

Extremely useful for diagnosing a stuck/hung production process without restarting it — you get a live goroutine dump and can inspect variable state, not just a static stack trace.

12.3 Conditional Breakpoints and Tracepoints

(dlv) break main.go:45 if userID == 12345
(dlv) trace main.HandleRequest

trace sets a tracepoint that logs entry/exit without stopping execution — useful for understanding call frequency/arguments in a running system without the overhead of full stepping.

12.4 Debugging Core Dumps

GOTRACEBACK=crash ./myapp   # generates a core dump on crash
dlv core ./myapp core_file

Combined with GOTRACEBACK=crash and ulimit -c unlimited, this lets you do full post-mortem analysis of a crashed production binary, inspecting exact goroutine state at the moment of the fatal error.

12.5 Remote Debugging (headless mode)

dlv debug --headless --listen=:2345 --api-version=2 ./cmd/myapp

Then connect from your IDE (VS Code, GoLand) via the Delve DAP/JSON-RPC protocol — standard for debugging inside containers/Kubernetes pods via kubectl port-forward.

---

13. Race Detection

Go’s race detector (built on ThreadSanitizer) instruments memory accesses to catch data races — concurrent unsynchronized access to the same memory location where at least one access is a write.

go test -race ./...
go build -race -o myapp-race .
./myapp-race

Critical facts:

• The race detector only catches races that actually occur during execution — it is not static analysis. A race in a rarely-hit code path won’t be caught unless that path executes under the instrumented binary.
• It adds significant CPU (~2-20x) and memory (~5-10x) overhead — never run -race binaries in production for real traffic; use them in CI, load testing, and staging with representative traffic instead.
• Always run integration/E2E test suites with -race in CI — races found here are far cheaper than races found in production (which often manifest as nondeterministic, hard-to-reproduce corruption or crashes).

13.1 Common Race Patterns

• Writing to a shared map from multiple goroutines without a mutex (Go maps are explicitly not safe for concurrent read+write; a naked concurrent write can even corrupt the runtime and cause a fatal, unrecoverable crash — fatal error: concurrent map writes — which, note, is not the same as -race detection; that fatal error triggers even without -race).
• Capturing a loop variable by reference in a goroutine (fixed in Go 1.22+, where loop variables are now per-iteration by default — but be aware of this if working in older Go versions or reading legacy code):

// Pre-1.22 bug pattern:
for _, item := range items {
    go func() {
        process(item) // captures the shared loop variable
    }()
}
// Fix (pre-1.22): pass item explicitly
for _, item := range items {
    go func(item Item) {
        process(item)
    }(item)
}

• Reading/writing a struct field from multiple goroutines without synchronization, even if it “looks read-only” (e.g., lazy-initialization patterns without sync.Once).

---

14. Deadlock and Goroutine Diagnostics

14.1 Runtime Deadlock Detection

Go’s runtime detects all-goroutines-asleep deadlocks automatically and crashes with:

fatal error: all goroutines are asleep - deadlock!

This only catches global deadlocks (every single goroutine blocked) — it does not detect partial deadlocks where some goroutines remain runnable while a subset is stuck. Those require manual investigation via SIGQUIT dumps or pprof’s goroutine profile.

14.2 Triggering a Full Stack Dump on a Live Process

kill -QUIT <pid>

By default this prints all goroutine stacks to stderr and terminates the process (unless you’ve set debug.SetTraceback or have a custom signal handler). For a non-terminating live dump, hit the pprof endpoint instead:

curl http://localhost:6060/debug/pprof/goroutine?debug=2

14.3 GOTRACEBACK Environment Variable

Controls verbosity of crash output:

• GOTRACEBACK=none — no traceback (never use this in anything but the most locked-down security contexts).
• GOTRACEBACK=single (default) — traceback of the crashing goroutine only.
• GOTRACEBACK=all — all user goroutines.
• GOTRACEBACK=system — all goroutines, including runtime-internal ones.
• GOTRACEBACK=crash — like system, plus triggers a core dump.

Set GOTRACEBACK=all (or system for deep runtime debugging) in production for services where post-mortem crash diagnostics matter.

---

15. Continuous Profiling in Production

Point-in-time profiling (SSH in, hit /debug/pprof/, download, analyze) doesn’t scale for intermittent issues or fleet-wide analysis. Continuous profiling tools sample production processes at low overhead continuously and store historical profile data, queryable like logs/metrics.

Common approaches:

• Google Cloud Profiler, Datadog Continuous Profiler, Grafana Pyroscope (open source, CNCF), Parca (open source, eBPF-based, can profile without code changes for many languages).
• Overhead is designed to be low (typically <2-5% CPU) via reduced sampling rates and efficient aggregation, safe for always-on production use — unlike manually cranking SetMutexProfileFraction(1).
• The key value: correlating a profile snapshot with a specific incident timestamp after the fact, without needing to have manually captured a profile during the incident window.

For any team running Go at meaningful production scale, wiring in continuous profiling is one of the highest-leverage observability investments — it converts “we’d need to reproduce this to profile it” into “let’s just look at what the profiler already recorded during the incident.”

---

16. Flame Graphs and Visualization

go tool pprof -http=:8080 cpu.prof launches a full local web UI (successor to the old -web/Graphviz-only flow) with:

• Flame graph view — width represents relative time/resource cost, depth represents call stack depth. Wide plateaus at the top of the stack are your optimization targets.
• Graph view — classic call-graph with weighted edges.
• Peek/Source/Disassembly views — line-level and even assembly-level cost attribution.
• Top/Flat/Cumulative sortable tables.

go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

This is generally the fastest, most ergonomic way to explore a profile interactively — prefer it over the plain REPL for exploratory work, and reserve the REPL for scripted/CI-driven checks (e.g., asserting top function stays below X% in a perf-regression test).

---

17. Common Pitfalls Catalog

Symptom	Likely Cause	Tool to Confirm
Rising RSS, stable request rate	Goroutine leak or unbounded cache	goroutine profile diff, inuse_space heap diff
High p99 latency, normal average	GC STW pauses, scheduler contention	go tool trace
High CPU, low throughput	Excessive allocation → GC pressure	alloc_space profile, -benchmem
Sudden crash: “all goroutines asleep”	Global deadlock	Crash stack trace itself
Intermittent slow requests	Lock contention	mutex/block profile
OOM-killed in Kubernetes	No GOMEMLIMIT, GOGC too high, or true leak	GOMEMLIMIT + heap diff
fatal error: concurrent map writes	Unsynchronized map access	Code review + -race on relevant test path
High GC CPU % in gctrace=1	Allocation-heavy hot path	alloc_objects profile, escape analysis
Goroutine count grows linearly with requests, never drops	Missing cancel(), unclosed response bodies	goroutine?debug=2 diff over time

---

18. Case Studies

18.1 The Silent Goroutine Leak

Symptom: A service’s memory grew ~50MB/hour under constant load, eventually OOM-killed every ~18 hours.

Investigation: inuse_space heap diff between two 30-minute-apart snapshots showed growth concentrated in bufio.Reader buffers and HTTP transport internals, not application structs. goroutine?debug=2 revealed thousands of goroutines stuck in net/http.(*persistConn).readLoop.

Root cause: A downstream HTTP client wasn’t closing/draining resp.Body on early-return error paths, leaving the connection (and its associated read-loop goroutine) alive indefinitely — the connection was never returned to the pool nor cleaned up.

Fix: Always defer resp.Body.Close() immediately after checking the error from the HTTP call, and drain the body with io.Copy(io.Discard, resp.Body) before closing when reusing keep-alive connections is desired.

18.2 The p99 Latency Cliff

Symptom: p50 latency was fine (5ms), but p99 spiked to 800ms periodically, roughly every 2 minutes.

Investigation: go tool trace during a captured window showed a clear, synchronized gap across all Ps lasting ~600ms, correlated with a GC cycle. gctrace=1 confirmed large GC cycles with heap growing from 400MB→900MB before each collection.

Root cause: GOGC was left at default 100 with a naturally bursty allocation pattern (large batch processing every 2 minutes), causing large mark-assist stalls when the GC fell behind the mutator’s allocation rate.

Fix: Set GOMEMLIMIT appropriately and lowered GOGC to 50, trading slightly higher average GC CPU for much smaller, more frequent (and thus shorter) individual pauses — eliminating the periodic cliff.

---

19. Checklist: Investigating a Production Incident

1. Reproduce or capture live: If ongoing, immediately snapshot /debug/pprof/goroutine?debug=2, /debug/pprof/heap, and a 30s CPU profile before the state changes.
2. Check runtime.NumGoroutine() trend — is it climbing unboundedly?
3. Diff heap snapshots across the incident window if memory-related.
4. Check GODEBUG=gctrace=1 output (or continuous profiler equivalent) for GC CPU% and pause duration trends.
5. Correlate with go tool trace if latency (not throughput) is the symptom.
6. Check GOMAXPROCS vs actual container CPU quota.
7. Check GOMEMLIMIT vs actual container memory quota.
8. Review recent deploys for new allocation-heavy code paths, new locks, or new goroutine-spawning logic without corresponding cleanup.
9. Confirm with a benchmark + -race + profile in staging before deploying a fix — don’t ship a “probably fixed it” patch without verifying against the actual captured profile.

---

20. Reference Commands Cheat Sheet

# CPU profile from running server
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

# Heap in-use snapshot
go tool pprof -http=:8080 -inuse_space http://localhost:6060/debug/pprof/heap

# Heap allocation (cumulative) snapshot
go tool pprof -http=:8080 -alloc_space http://localhost:6060/debug/pprof/heap

# Goroutine dump, human readable
curl http://localhost:6060/debug/pprof/goroutine?debug=2

# Diff two heap snapshots
go tool pprof -base heap_before.prof heap_after.prof

# Execution trace
curl -o trace.out http://localhost:6060/debug/pprof/trace?seconds=5
go tool trace trace.out

# Benchmark with allocation stats, 10 runs
go test -bench=. -benchmem -count=10 ./...

# Compare benchmark runs statistically
benchstat old.txt new.txt

# Escape analysis
go build -gcflags="-m -m" ./...

# Race detector
go test -race ./...

# GC trace
GODEBUG=gctrace=1 ./myapp

# Live process debugging
dlv attach <pid>

# Core dump analysis
GOTRACEBACK=crash ./myapp
dlv core ./myapp core_file

# Full goroutine dump + terminate (careful in prod!)
kill -QUIT <pid>

---

Closing Notes

Go’s diagnostic tooling is unusually complete for a language runtime — the combination of pprof, trace, runtime/metrics, race detection, and Delve covers essentially every class of production issue: CPU, memory, concurrency, and latency. The discipline that separates a good Go engineer from a great one isn’t knowing every tool exists — it’s the habit of reaching for the right tool based on the symptom’s shape (throughput problem → CPU/alloc profile; latency problem → trace; growth-over-time problem → heap/goroutine diffing; crash → core dump + Delve), and treating every optimization claim as a hypothesis to be validated with benchstat, not asserted from intuition.