Linux CLI Essentials for Debugging Agent Processes

Q: macOS vs Linux - what breaks?

/proc is Linux-specific. On macOS use ps, lsof -i, and log show / app logs; prefer debugging Linux containers that mirror prod.

Q: What if I do not have root?

Use user-scoped systemctl --user, app log paths you own, and platform UIs for restarts. Still learn the read-only discovery commands.

Q: How do I avoid leaking secrets in tickets?

Log names of env vars, not values. Redact Authorization headers and tokens from curl -v output.

Q: How do tool subprocesses show up?

As children in ps trees or as short-lived containers. Capture parent PID from the worker log line that started the tool.

Agent failures often show up as a stuck worker, a silent tool subprocess, a full disk of traces, or a bad env var - not as a nice stack trace in your IDE.

This cheatsheet is the short CLI set that makes those failures legible on Linux hosts and containers (macOS is similar for many commands; prefer the same tools against Linux prod or WSL2).

How to Use This Cheatsheet

Walk A → F when a page fires: process, logs, resources, network, env, containers.
Prefer read-only commands first; restart and kill only with a runbook.
Capture command output into the incident ticket (redact secrets).
Pair with pack/pin checks from git history when behavior changed but the host looks healthy.
Verify distro-specific unit names and paths at build; examples use common patterns.

A - Process Discovery

Goal	Command	Notes
Find by name	`pgrep -af agent`	Shows PID + full command line
Broader Python	`pgrep -af python`	Noisy on shared hosts; filter carefully
Tree view	`ps auxf \| head -n 40`	Parent/child tool processes
One PID detail	`ps -p PID -o pid,ppid,user,%cpu,%mem,etime,cmd`	Age and resource snapshot
Threads (busy loops)	`ps -p PID -L`	Many threads can mean hung SDK calls
Open files	`ls -l /proc/PID/fd`	Leaked sockets or stuck pipes
Working directory	`readlink -f /proc/PID/cwd`	Wrong cwd breaks relative pack paths
Executable	`readlink -f /proc/PID/exe`	Confirms which binary is live

# Typical first 30 seconds of an "agent is down" page
pgrep -af 'uvicorn|gunicorn|agent|langgraph' || true
ps aux --sort=-%cpu | head -n 15

B - Logs and Journal

Goal	Command	Notes
Unit logs	`journalctl -u agent-worker.service -n 200 --no-pager`	Replace unit name
Follow	`journalctl -u agent-worker.service -f`	Live tail
Since time	`journalctl -u agent-worker.service --since "10 min ago"`	Incident window
Priority	`journalctl -p err -u agent-worker.service -n 100`	Errors only
File tail	`tail -n 200 /var/log/agent/worker.log`	App file logging
Follow file	`tail -f /var/log/agent/worker.log`	Local/dev
Search	`rg -n "tool_error	Traceback
Last boot	`journalctl -b -u agent-worker.service`	After host reboot

Look for: run ids, prompt_version, tool_schema_version, model errors, repeated tool timeouts, OOM killer messages (Killed process).

C - Resources (CPU, Memory, Disk)

Goal	Command	Notes
Live overview	`top` or `htop`	CPU and memory hogs
Memory summary	`free -h`	Host pressure
Disk usage	`df -h`	Full disk breaks logs and downloads
Big dirs	`du -sh /var/log/* /tmp/* 2>/dev/null \| sort -h`	Trace/log blowups
OOM hints	`dmesg -T \| rg -i 'oom\|killed process'`	Kernel OOM
File limits	`ulimit -n`	Too few FDs under tool fan-out
Load	`uptime`	Sustained high load

Agent-specific: embedding caches, trace exporters, and browser tool profiles can fill disks quietly.

D - Network and Ports

Goal	Command	Notes
Listening ports	`ss -lntp`	Who owns 8000/8080/etc.
Established	`ss -tp`	Busy model or tool connections
DNS smoke	`getent hosts api.openrouter.ai`	Example host; use your providers
HTTP smoke	`curl -sS -o /dev/null -w '%{http_code}\n' https://example.com`	Egress sanity
Local health	`curl -sS http://127.0.0.1:8000/health`	If you expose one
Trace route (optional)	`traceroute -n HOST`	When paths break oddly

ss -lntp | rg '8000|8080|3000' || true
curl -sS -m 5 http://127.0.0.1:8000/health || echo "health failed"

Do not probe production metadata IPs or internal ranges from a compromised debug session without policy approval.

E - Environment, Config, and Signals

Goal	Command	Notes
Env of PID	`tr '\0' '\n' < /proc/PID/environ \| rg '^[A-Z0-9_]+=' \| cut -d= -f1`	Names only; avoid dumping secrets
User / groups	`id`	Permission mismatches
Capability (containers)	`capsh --print` 2>/dev/null	When relevant
Graceful stop	`kill -TERM PID`	Prefer SIGTERM over SIGKILL
Force stop	`kill -KILL PID`	Last resort; can interrupt tool side effects
Reload (if supported)	`systemctl reload agent-worker`	Only if unit supports it
Restart	`systemctl restart agent-worker`	Runbook-owned action

# Confirm pin path style vars exist without printing values
PID=$(pgrep -n -f agent-worker || true)
if [ -n "${PID}" ]; then
  tr '\0' '\n' < /proc/$PID/environ | cut -d= -f1 | rg 'PIN|PROMPT|MODEL|OPENROUTER|LANG' || true
fi

F - Containers and Common Agent Runtimes

Goal	Command	Notes
List	`docker ps` / `podman ps`	Running agent sidecars
Logs	`docker logs --tail 200 CONTAINER`	App stdout
Follow	`docker logs -f CONTAINER`	Live
Exec shell	`docker exec -it CONTAINER sh`	Inspect inside
Stats	`docker stats --no-stream`	CPU/mem per container
K8s pods	`kubectl get pods -l app=agent`	Cluster deploys
K8s logs	`kubectl logs deploy/agent-worker --tail=200`
K8s describe	`kubectl describe pod POD`	Events, OOMKilled
K8s exec	`kubectl exec -it POD -- sh`

Sandbox note: code-exec tools may run sibling containers. Debug both the orchestrator worker and the sandbox job.

G - Quick Incident Order of Operations

Scope: which env, which service, which start time.
Process up? Section A.
Errors in logs? Section B (search for run_id if the user reported one).
Resource death? Section C (disk and OOM first).
Egress / port? Section D.
Bad pin or missing env? Section E + git/pin history.
Mitigate: restart only if runbook allows; flip pin/flag for bad behavior without host crash.
Capture: versions (prompt_version, code_sha), command outputs, timeline.

FAQs

macOS vs Linux - what breaks?

/proc is Linux-specific. On macOS use ps, lsof -i, and log show / app logs; prefer debugging Linux containers that mirror prod.

What if I do not have root?

Use user-scoped systemctl --user, app log paths you own, and platform UIs for restarts. Still learn the read-only discovery commands.

How do I avoid leaking secrets in tickets?

Log names of env vars, not values. Redact Authorization headers and tokens from curl -v output.

Why is the process up but the agent "dead"?

Stuck event loop, exhausted thread pool, upstream model outage, or bad pin that fails every run. Logs and open connections matter more than PID existence.

What is a good health endpoint for agents?

Liveness (process up) plus optional readiness (can reach model + load pin). Do not run a full LLM call on every probe.

How do tool subprocesses show up?

As children in ps trees or as short-lived containers. Capture parent PID from the worker log line that started the tool.

journalctl is empty - now what?

App may log only to files or stdout collected by the orchestrator. Check Docker/K8s logs and configured file paths in the unit file (systemctl cat UNIT).

Can I strace production workers?

Only with explicit approval; it is invasive and can stall processes. Prefer metrics, logs, and staging repro first.

How does this pair with observability products?

Langfuse, LangSmith, Phoenix, and friends help on traces when the pipeline works. CLI still saves you when the exporter is down or the host is sick. See What Agent Observability Actually Needs to Capture.

What should every on-call runbook paste?

Exact unit/deploy names, log commands, health URL, pin rollback action, and who owns model provider status pages.

Is `kill -9` ever OK for agents?

Only after TERM fails and you accept possible partial tool side effects (half-sent email, half-applied refund). Prefer tool-level kill switches when available.

How do I confirm which prompt version is live?

From run traces or worker logs that print the pin. If missing, that is a product bug - add pin fields to every run. Do not guess from mtime of files on one box.

Why Agent Teams Still Need Solid Git and Linux Fundamentals - why CLI literacy remains required
Git & Linux Basics for Agent Developers - first shell habits
Onboarding a New Developer to an Agent Codebase - include this sheet in ramp
Git, Linux CLI & Team Onboarding Best Practices - operational habits
What Agent Observability Actually Needs to Capture - versioned traces
Timeout Strategies for Long-Running Agent Steps - stuck steps
Kill Switches: Stopping a Runaway Agent Mid-Execution - safer than kill -9
Rollback Strategies When a Deployed Agent Regresses - bad pin vs bad host

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

This cheatsheet is the short CLI set that makes those failures legible on Linux hosts and containers (macOS is similar for many commands; prefer the same tools against Linux prod or WSL2).

How to Use This Cheatsheet

Walk A → F when a page fires: process, logs, resources, network, env, containers.
Prefer read-only commands first; restart and kill only with a runbook.
Capture command output into the incident ticket (redact secrets).
Pair with pack/pin checks from git history when behavior changed but the host looks healthy.
Verify distro-specific unit names and paths at build; examples use common patterns.

A - Process Discovery

Goal	Command	Notes
Find by name	`pgrep -af agent`	Shows PID + full command line
Broader Python	`pgrep -af python`	Noisy on shared hosts; filter carefully
Tree view	`ps auxf \| head -n 40`	Parent/child tool processes
One PID detail	`ps -p PID -o pid,ppid,user,%cpu,%mem,etime,cmd`	Age and resource snapshot
Threads (busy loops)	`ps -p PID -L`	Many threads can mean hung SDK calls
Open files	`ls -l /proc/PID/fd`	Leaked sockets or stuck pipes
Working directory	`readlink -f /proc/PID/cwd`	Wrong cwd breaks relative pack paths
Executable	`readlink -f /proc/PID/exe`	Confirms which binary is live

# Typical first 30 seconds of an "agent is down" page
pgrep -af 'uvicorn|gunicorn|agent|langgraph' || true
ps aux --sort=-%cpu | head -n 15

B - Logs and Journal

Goal	Command	Notes
Unit logs	`journalctl -u agent-worker.service -n 200 --no-pager`	Replace unit name
Follow	`journalctl -u agent-worker.service -f`	Live tail
Since time	`journalctl -u agent-worker.service --since "10 min ago"`	Incident window
Priority	`journalctl -p err -u agent-worker.service -n 100`	Errors only
File tail	`tail -n 200 /var/log/agent/worker.log`	App file logging
Follow file	`tail -f /var/log/agent/worker.log`	Local/dev
Search	`rg -n "tool_error	Traceback
Last boot	`journalctl -b -u agent-worker.service`	After host reboot

Look for: run ids, prompt_version, tool_schema_version, model errors, repeated tool timeouts, OOM killer messages (Killed process).

C - Resources (CPU, Memory, Disk)

Goal	Command	Notes
Live overview	`top` or `htop`	CPU and memory hogs
Memory summary	`free -h`	Host pressure
Disk usage	`df -h`	Full disk breaks logs and downloads
Big dirs	`du -sh /var/log/* /tmp/* 2>/dev/null \| sort -h`	Trace/log blowups
OOM hints	`dmesg -T \| rg -i 'oom\|killed process'`	Kernel OOM
File limits	`ulimit -n`	Too few FDs under tool fan-out
Load	`uptime`	Sustained high load

Agent-specific: embedding caches, trace exporters, and browser tool profiles can fill disks quietly.

D - Network and Ports

Goal	Command	Notes
Listening ports	`ss -lntp`	Who owns 8000/8080/etc.
Established	`ss -tp`	Busy model or tool connections
DNS smoke	`getent hosts api.openrouter.ai`	Example host; use your providers
HTTP smoke	`curl -sS -o /dev/null -w '%{http_code}\n' https://example.com`	Egress sanity
Local health	`curl -sS http://127.0.0.1:8000/health`	If you expose one
Trace route (optional)	`traceroute -n HOST`	When paths break oddly

ss -lntp | rg '8000|8080|3000' || true
curl -sS -m 5 http://127.0.0.1:8000/health || echo "health failed"

Do not probe production metadata IPs or internal ranges from a compromised debug session without policy approval.

E - Environment, Config, and Signals

Goal	Command	Notes
Env of PID	`tr '\0' '\n' < /proc/PID/environ \| rg '^[A-Z0-9_]+=' \| cut -d= -f1`	Names only; avoid dumping secrets
User / groups	`id`	Permission mismatches
Capability (containers)	`capsh --print` 2>/dev/null	When relevant
Graceful stop	`kill -TERM PID`	Prefer SIGTERM over SIGKILL
Force stop	`kill -KILL PID`	Last resort; can interrupt tool side effects
Reload (if supported)	`systemctl reload agent-worker`	Only if unit supports it
Restart	`systemctl restart agent-worker`	Runbook-owned action

# Confirm pin path style vars exist without printing values
PID=$(pgrep -n -f agent-worker || true)
if [ -n "${PID}" ]; then
  tr '\0' '\n' < /proc/$PID/environ | cut -d= -f1 | rg 'PIN|PROMPT|MODEL|OPENROUTER|LANG' || true
fi

F - Containers and Common Agent Runtimes

Goal	Command	Notes
List	`docker ps` / `podman ps`	Running agent sidecars
Logs	`docker logs --tail 200 CONTAINER`	App stdout
Follow	`docker logs -f CONTAINER`	Live
Exec shell	`docker exec -it CONTAINER sh`	Inspect inside
Stats	`docker stats --no-stream`	CPU/mem per container
K8s pods	`kubectl get pods -l app=agent`	Cluster deploys
K8s logs	`kubectl logs deploy/agent-worker --tail=200`
K8s describe	`kubectl describe pod POD`	Events, OOMKilled
K8s exec	`kubectl exec -it POD -- sh`

Sandbox note: code-exec tools may run sibling containers. Debug both the orchestrator worker and the sandbox job.

G - Quick Incident Order of Operations

Scope: which env, which service, which start time.
Process up? Section A.
Errors in logs? Section B (search for run_id if the user reported one).
Resource death? Section C (disk and OOM first).
Egress / port? Section D.
Bad pin or missing env? Section E + git/pin history.
Mitigate: restart only if runbook allows; flip pin/flag for bad behavior without host crash.
Capture: versions (prompt_version, code_sha), command outputs, timeline.

FAQs

macOS vs Linux - what breaks?

/proc is Linux-specific. On macOS use ps, lsof -i, and log show / app logs; prefer debugging Linux containers that mirror prod.

What if I do not have root?

Use user-scoped systemctl --user, app log paths you own, and platform UIs for restarts. Still learn the read-only discovery commands.

How do I avoid leaking secrets in tickets?

Log names of env vars, not values. Redact Authorization headers and tokens from curl -v output.

Why is the process up but the agent "dead"?

Stuck event loop, exhausted thread pool, upstream model outage, or bad pin that fails every run. Logs and open connections matter more than PID existence.

What is a good health endpoint for agents?

Liveness (process up) plus optional readiness (can reach model + load pin). Do not run a full LLM call on every probe.

How do tool subprocesses show up?

As children in ps trees or as short-lived containers. Capture parent PID from the worker log line that started the tool.

journalctl is empty - now what?

App may log only to files or stdout collected by the orchestrator. Check Docker/K8s logs and configured file paths in the unit file (systemctl cat UNIT).

Can I strace production workers?

Only with explicit approval; it is invasive and can stall processes. Prefer metrics, logs, and staging repro first.

How does this pair with observability products?

What should every on-call runbook paste?

Exact unit/deploy names, log commands, health URL, pin rollback action, and who owns model provider status pages.

Is `kill -9` ever OK for agents?

Only after TERM fails and you accept possible partial tool side effects (half-sent email, half-applied refund). Prefer tool-level kill switches when available.

How do I confirm which prompt version is live?

From run traces or worker logs that print the pin. If missing, that is a product bug - add pin fields to every run. Do not guess from mtime of files on one box.

Why Agent Teams Still Need Solid Git and Linux Fundamentals - why CLI literacy remains required
Git & Linux Basics for Agent Developers - first shell habits
Onboarding a New Developer to an Agent Codebase - include this sheet in ramp
Git, Linux CLI & Team Onboarding Best Practices - operational habits
What Agent Observability Actually Needs to Capture - versioned traces
Timeout Strategies for Long-Running Agent Steps - stuck steps
Kill Switches: Stopping a Runaway Agent Mid-Execution - safer than kill -9
Rollback Strategies When a Deployed Agent Regresses - bad pin vs bad host

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Linux CLI Essentials for Debugging Agent Processes

Search Documentation

Linux CLI Essentials for Debugging Agent Processes