Linux CLI Essentials for Debugging Agent Processes
Agent failures often show up as a stuck worker, a silent tool subprocess, a full disk of traces, or a bad env var - not as a nice stack trace in your IDE.
This cheatsheet is the short CLI set that makes those failures legible on Linux hosts and containers (macOS is similar for many commands; prefer the same tools against Linux prod or WSL2).
How to Use This Cheatsheet
- Walk A → F when a page fires: process, logs, resources, network, env, containers.
- Prefer read-only commands first; restart and kill only with a runbook.
- Capture command output into the incident ticket (redact secrets).
- Pair with pack/pin checks from git history when behavior changed but the host looks healthy.
- Verify distro-specific unit names and paths at build; examples use common patterns.
A - Process Discovery
| Goal | Command | Notes |
|---|---|---|
| Find by name | pgrep -af agent | Shows PID + full command line |
| Broader Python | pgrep -af python | Noisy on shared hosts; filter carefully |
| Tree view | ps auxf | head -n 40 | Parent/child tool processes |
| One PID detail | ps -p PID -o pid,ppid,user,%cpu,%mem,etime,cmd | Age and resource snapshot |
| Threads (busy loops) | ps -p PID -L | Many threads can mean hung SDK calls |
| Open files | ls -l /proc/PID/fd | Leaked sockets or stuck pipes |
| Working directory | readlink -f /proc/PID/cwd | Wrong cwd breaks relative pack paths |
| Executable | readlink -f /proc/PID/exe | Confirms which binary is live |
# Typical first 30 seconds of an "agent is down" page
pgrep -af 'uvicorn|gunicorn|agent|langgraph' || true
ps aux --sort=-%cpu | head -n 15B - Logs and Journal
| Goal | Command | Notes |
|---|---|---|
| Unit logs | journalctl -u agent-worker.service -n 200 --no-pager | Replace unit name |
| Follow | journalctl -u agent-worker.service -f | Live tail |
| Since time | journalctl -u agent-worker.service --since "10 min ago" | Incident window |
| Priority | journalctl -p err -u agent-worker.service -n 100 | Errors only |
| File tail | tail -n 200 /var/log/agent/worker.log | App file logging |
| Follow file | tail -f /var/log/agent/worker.log | Local/dev |
| Search | `rg -n "tool_error | Traceback |
| Last boot | journalctl -b -u agent-worker.service | After host reboot |
Look for: run ids, prompt_version, tool_schema_version, model errors, repeated tool timeouts, OOM killer messages (Killed process).
C - Resources (CPU, Memory, Disk)
| Goal | Command | Notes |
|---|---|---|
| Live overview | top or htop | CPU and memory hogs |
| Memory summary | free -h | Host pressure |
| Disk usage | df -h | Full disk breaks logs and downloads |
| Big dirs | du -sh /var/log/* /tmp/* 2>/dev/null | sort -h | Trace/log blowups |
| OOM hints | dmesg -T | rg -i 'oom|killed process' | Kernel OOM |
| File limits | ulimit -n | Too few FDs under tool fan-out |
| Load | uptime | Sustained high load |
Agent-specific: embedding caches, trace exporters, and browser tool profiles can fill disks quietly.
D - Network and Ports
| Goal | Command | Notes |
|---|---|---|
| Listening ports | ss -lntp | Who owns 8000/8080/etc. |
| Established | ss -tp | Busy model or tool connections |
| DNS smoke | getent hosts api.openrouter.ai | Example host; use your providers |
| HTTP smoke | curl -sS -o /dev/null -w '%{http_code}\n' https://example.com | Egress sanity |
| Local health | curl -sS http://127.0.0.1:8000/health | If you expose one |
| Trace route (optional) | traceroute -n HOST | When paths break oddly |
ss -lntp | rg '8000|8080|3000' || true
curl -sS -m 5 http://127.0.0.1:8000/health || echo "health failed"Do not probe production metadata IPs or internal ranges from a compromised debug session without policy approval.
E - Environment, Config, and Signals
| Goal | Command | Notes |
|---|---|---|
| Env of PID | tr '\0' '\n' < /proc/PID/environ | rg '^[A-Z0-9_]+=' | cut -d= -f1 | Names only; avoid dumping secrets |
| User / groups | id | Permission mismatches |
| Capability (containers) | capsh --print 2>/dev/null | When relevant |
| Graceful stop | kill -TERM PID | Prefer SIGTERM over SIGKILL |
| Force stop | kill -KILL PID | Last resort; can interrupt tool side effects |
| Reload (if supported) | systemctl reload agent-worker | Only if unit supports it |
| Restart | systemctl restart agent-worker | Runbook-owned action |
# Confirm pin path style vars exist without printing values
PID=$(pgrep -n -f agent-worker || true)
if [ -n "${PID}" ]; then
tr '\0' '\n' < /proc/$PID/environ | cut -d= -f1 | rg 'PIN|PROMPT|MODEL|OPENROUTER|LANG' || true
fiF - Containers and Common Agent Runtimes
| Goal | Command | Notes |
|---|---|---|
| List | docker ps / podman ps | Running agent sidecars |
| Logs | docker logs --tail 200 CONTAINER | App stdout |
| Follow | docker logs -f CONTAINER | Live |
| Exec shell | docker exec -it CONTAINER sh | Inspect inside |
| Stats | docker stats --no-stream | CPU/mem per container |
| K8s pods | kubectl get pods -l app=agent | Cluster deploys |
| K8s logs | kubectl logs deploy/agent-worker --tail=200 | |
| K8s describe | kubectl describe pod POD | Events, OOMKilled |
| K8s exec | kubectl exec -it POD -- sh |
Sandbox note: code-exec tools may run sibling containers. Debug both the orchestrator worker and the sandbox job.
G - Quick Incident Order of Operations
- Scope: which env, which service, which start time.
- Process up? Section A.
- Errors in logs? Section B (search for run_id if the user reported one).
- Resource death? Section C (disk and OOM first).
- Egress / port? Section D.
- Bad pin or missing env? Section E + git/pin history.
- Mitigate: restart only if runbook allows; flip pin/flag for bad behavior without host crash.
- Capture: versions (
prompt_version,code_sha), command outputs, timeline.
FAQs
macOS vs Linux - what breaks?
/proc is Linux-specific. On macOS use ps, lsof -i, and log show / app logs; prefer debugging Linux containers that mirror prod.
What if I do not have root?
Use user-scoped systemctl --user, app log paths you own, and platform UIs for restarts. Still learn the read-only discovery commands.
How do I avoid leaking secrets in tickets?
Log names of env vars, not values. Redact Authorization headers and tokens from curl -v output.
Why is the process up but the agent "dead"?
Stuck event loop, exhausted thread pool, upstream model outage, or bad pin that fails every run. Logs and open connections matter more than PID existence.
What is a good health endpoint for agents?
Liveness (process up) plus optional readiness (can reach model + load pin). Do not run a full LLM call on every probe.
How do tool subprocesses show up?
As children in ps trees or as short-lived containers. Capture parent PID from the worker log line that started the tool.
journalctl is empty - now what?
App may log only to files or stdout collected by the orchestrator. Check Docker/K8s logs and configured file paths in the unit file (systemctl cat UNIT).
Can I strace production workers?
Only with explicit approval; it is invasive and can stall processes. Prefer metrics, logs, and staging repro first.
How does this pair with observability products?
Langfuse, LangSmith, Phoenix, and friends help on traces when the pipeline works. CLI still saves you when the exporter is down or the host is sick. See What Agent Observability Actually Needs to Capture.
What should every on-call runbook paste?
Exact unit/deploy names, log commands, health URL, pin rollback action, and who owns model provider status pages.
Is `kill -9` ever OK for agents?
Only after TERM fails and you accept possible partial tool side effects (half-sent email, half-applied refund). Prefer tool-level kill switches when available.
How do I confirm which prompt version is live?
From run traces or worker logs that print the pin. If missing, that is a product bug - add pin fields to every run. Do not guess from mtime of files on one box.
Related
- Why Agent Teams Still Need Solid Git and Linux Fundamentals - why CLI literacy remains required
- Git & Linux Basics for Agent Developers - first shell habits
- Onboarding a New Developer to an Agent Codebase - include this sheet in ramp
- Git, Linux CLI & Team Onboarding Best Practices - operational habits
- What Agent Observability Actually Needs to Capture - versioned traces
- Timeout Strategies for Long-Running Agent Steps - stuck steps
- Kill Switches: Stopping a Runaway Agent Mid-Execution - safer than kill -9
- Rollback Strategies When a Deployed Agent Regresses - bad pin vs bad host
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.