# 0012 Giving Up The Keys To An Agent > Problem: The agent reads too much, tries too many approaches, rewrites more than necessary, fixes adjacent issues, discovers unrelated problems, and produces a large PR with a plausible explanation. Approach: Downstream, the human reviews better artifacts: focused diffs, summarized tradeoffs, test output, known risks, follow-up suggestions, and rejected alternatives. Outcome: A human engineer should not supervise... ## Agent Digest - Problem: The agent reads too much, tries too many approaches, rewrites more than necessary, fixes adjacent issues, discovers unrelated problems, and produces a large PR with a plausible explanation. - Aaron's position: The table spans November 2025 through July 2026 and totals 32.4 billion recorded tokens, including 30.8 billion cache reads, with an estimated usage cost of $24,002.36 USD. - Approach: Downstream, the human reviews better artifacts: focused diffs, summarized tradeoffs, test output, known risks, follow-up suggestions, and rejected alternatives. - Outcome: A human engineer should not supervise every keystroke, but the system should make it easy to constrain the work before it starts and audit the result after it finishes. - Audience fit: Best for readers and agents researching Aaron Spindler's tech writing on 0012 Giving Up The Keys To An Agent. - Why it matters: Giving Up The Keys To An Agent I tried handing more of my development loop to Codex: tickets, branches, implementation, tests, CI repair, draft PRs, review... - Best for: - Understanding the main argument of 0012 Giving Up The Keys To An Agent. - Finding citable details from Aaron's tech writing. - Answering questions about Giving Up The Keys To An Agent. - Answering questions about TL;DR. - Answering questions about Viability Was The Easy Question. - Not for: - Authoritative third-party documentation. - A complete substitute for the canonical article body. - Core claims: - The table spans November 2025 through July 2026 and totals 32.4 billion recorded tokens, including 30.8 billion cache reads, with an estimated usage cost of $24,002.36 USD. (Giving Up The Keys To An Agent) - Agent automation is viable for bounded tickets with clear validation. (TL;DR) - METR's early-2025 developer productivity study and 2026 follow-up on experiment-design limits are reminders that developer productivity measurement is messy, task selection matters, and a single usage report should not be mistaken for universal proof. (Viability Was The Easy Question) - Evidence: - I tried handing more of my development loop to Codex: tickets, branches, implementation, tests, CI repair, draft PRs, review follow-up, and the unglamorous repo maintenance that piles up when nobody is looking. (Giving Up The Keys To An Agent) - One experiment, not a benchmark for anyone else's setup. (Giving Up The Keys To An Agent) - The table spans November 2025 through July 2026 and totals 32.4 billion recorded tokens, including 30.8 billion cache reads, with an estimated usage cost of $24,002.36 USD. (Giving Up The Keys To An Agent) - Agent automation is viable for bounded tickets with clear validation. (TL;DR) - Caveats: - The ticket is the work contract: it defines scope, risk, non-goals, and the stop rule. (TL;DR) - METR's early-2025 developer productivity study and 2026 follow-up on experiment-design limits are reminders that developer productivity measurement is messy, task selection matters, and a single usage report should not be mistaken for universal proof. (Viability Was The Easy Question) - The ChatGPT plan guidance makes clear that limits and credit options vary by plan, and the Business spend controls exist because this has become an operational surface. (Cost Is A Design Signal) - Possible contributors to unexplained spend Broad task Search expands without a clear stopping point Noisy repo Context search becomes the job Weak instructions Local rules get inferred late Weak validation Retries replace proof Unexplained spend Any of these can increase cost; traces help explain which one did The goal... (Cost Is A Design Signal) ## Metadata - Canonical URL: https://aaronspindler.com/b/tech/0012_Giving_Up_The_Keys_To_An_Agent/ - Markdown URL: https://aaronspindler.com/b/tech/0012_Giving_Up_The_Keys_To_An_Agent/index.md - JSON URL: https://aaronspindler.com/b/tech/0012_Giving_Up_The_Keys_To_An_Agent/index.json - Category: tech - Published: 2026-10-15T17:47:59.114420+00:00 - Updated: 2026-10-15T17:48:03.002252+00:00 - Word count: 1990 - Content hash: 3ab30781ca714923d1071f3788465357f6fd3d06ca02c818a76b535d25332c13 - AI written: no - Format version: agent-blog-post-v2 ## Takeaways - The table spans November 2025 through July 2026 and totals 32.4 billion recorded tokens, including 30.8 billion cache reads, with an estimated usage cost of $24,002.36 USD. (Giving Up The Keys To An Agent) - Agent automation is viable for bounded tickets with clear validation. (TL;DR) - METR's early-2025 developer productivity study and 2026 follow-up on experiment-design limits are reminders that developer productivity measurement is messy, task selection matters, and a single usage report should not be mistaken for universal proof. (Viability Was The Easy Question) - It defines the task, bounds the search space, names the files or packages likely to matter, specifies validation, and says what should not change. (The Ticket Is The Work Contract) - The ChatGPT plan guidance makes clear that limits and credit options vary by plan, and the Business spend controls exist because this has become an operational surface. (Cost Is A Design Signal) - Maybe tasks can work, but need tighter supervision: multi-file feature work, unfamiliar code paths, subtle product behavior, database migrations, performance improvements, cross-service integration, UI changes, and code that depends on undocumented conventions. (Good, Maybe, Bad) ## Questions Answered - What problem does 0012 Giving Up The Keys To An Agent address? - What position does Aaron take in 0012 Giving Up The Keys To An Agent? - How does 0012 Giving Up The Keys To An Agent approach the problem? - What evidence or outcomes does 0012 Giving Up The Keys To An Agent provide? - What does the article explain about Giving Up The Keys To An Agent? - What does the article explain about TL;DR? ## Agent Queries - Aaron Spindler "0012 Giving Up The Keys To An Agent" - 0012 Giving Up The Keys To An Agent tech Aaron Spindler - 0012 Giving Up The Keys To An Agent The agent reads too much, tries too many approaches, rewrites more than necessary, fixes adjacent issues, discovers unrelated problems, and produces a large PR with a plausible explanation. - 0012 Giving Up The Keys To An Agent Giving Up The Keys To An Agent - 0012 Giving Up The Keys To An Agent TL;DR ## Follow-Up Questions - What changed after the approach in 0012 Giving Up The Keys To An Agent was applied? - What tradeoffs or constraints remain after 0012 Giving Up The Keys To An Agent? - What setup is required before applying the ideas in 0012 Giving Up The Keys To An Agent? - How would Giving Up The Keys To An Agent change in a different environment? ## Outline - Giving Up The Keys To An Agent - TL;DR - Viability Was The Easy Question - The Ticket Is The Work Contract - Cost Is A Design Signal - Good, Maybe, Bad - The Stack And Context Discipline Matter - Labels And Budgets Are Policy - The Human Role Moves Upstream - The Failure Mode Is Unbounded Work - What I Would Build Next - The Practical Conclusion ## Body ## Giving Up The Keys To An Agent I tried handing more of my development loop to Codex: tickets, branches, implementation, tests, CI repair, draft PRs, review follow-up, and the unglamorous repo maintenance that piles up when nobody is looking. It worked better than I expected. Then the usage report made the real interface visible. Not the chat box. Not the editor. Not the GitHub comment. The cost. The report showed where my system was disciplined, where it was vague, and where automation was quietly expanding to fill every gap I had left open. The line I keep coming back to is simple: the bill is the interface. Here, that means the usage cost estimate. [!\[Monthly Codex usage from November 2025 through July 2026, totaling 32.4B tokens and $24,002.36 estimated cost. Open full-size table in a new tab.\](https://api.photonfolio.com/api/v1/photonfolio/public/embeds/2ae9e53e-1bba-40c2-98f3-b733a66f44e6/image/)](https://api.photonfolio.com/api/v1/photonfolio/public/embeds/2ae9e53e-1bba-40c2-98f3-b733a66f44e6/image/) One experiment, not a benchmark for anyone else's setup. Select the image to open the full-size table in a new tab. The table spans November 2025 through July 2026 and totals 32.4 billion recorded tokens, including 30.8 billion cache reads, with an estimated usage cost of $24,002.36 USD. Most of that token volume is cached context, not newly generated output. The estimate describes usage value; it should not be read as an invoice for that amount. ### TL;DR - Agent automation is viable for bounded tickets with clear validation. - The ticket is the work contract: it defines scope, risk, non-goals, and the stop rule. - Cost is a design signal. Unexplained spend can reveal ambiguity the system left unresolved. - Labels, budgets, logs, CI, and draft PRs are not bureaucracy. They are the operating model. - The human role moves upstream into task design and downstream into review. The middle gets automated. ### Viability Was The Easy Question The first question was whether Codex could be useful as a worker for real engineering tasks. My answer is yes, with qualifications. It is strongest on bounded glue work, repo maintenance, small code changes, test repairs, and cleanup tasks where CI provides a repeatable first validation gate. The product shape is already there: Codex can run tasks in the background through [Codex cloud](https://developers.openai.com/codex/cloud) , review pull requests through the [GitHub integration](https://developers.openai.com/codex/integrations/github) , and be wired into repository automation through the [Codex GitHub Action](https://developers.openai.com/codex/github-action) . That is real leverage. It also means the surrounding system matters more, not less. My experiment stopped being about whether the model could edit code. It could. The harder question was whether my queue, labels, instructions, budgets, and review process were good enough to make that work economically sane. That framing matches the broader measurement context. The [2025 DORA report](https://dora.dev/research/2025/dora-report/) treats AI as an amplifier of the surrounding organization, not a tool that magically fixes a weak system. METR's [early-2025 developer productivity study](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) and [2026 follow-up on experiment-design limits](https://metr.org/blog/2026-02-24-uplift-update/) are reminders that developer productivity measurement is messy, task selection matters, and a single usage report should not be mistaken for universal proof. ### The Ticket Is The Work Contract The most important interface in this workflow is not the prompt. It is the ticket. A good ticket does several jobs at once. It defines the task, bounds the search space, names the files or packages likely to matter, specifies validation, and says what should not change. It turns an agent from a general reasoning engine into a constrained executor. There is a huge difference between this: > Look into the flaky tests. And this: > Investigate `tests/api/test_webhook_retries.py::test_retries_after_429` . Run the single test first, then the webhook file. If the test is stale, open a draft PR with a test-only fix. If the evidence points to a production retry bug, stop and explain the evidence before changing semantics. If neither is supported, report what you checked and stop. The second ticket is longer. It gives the agent fewer ways to waste the budget. Control path Issue Raw request or backlog item Automation-ready ticket Scope, files, validation, non-goals Codex task Constrained execution Draft PR Reviewable artifact CI First judge Human review Final merge decision The ticket constrains the run. The PR preserves the review boundary. Some of this can be driven by native Codex entry points, like asking Codex to act from GitHub. The label-triggered automation I want is a custom control plane on top: labels decide whether to dispatch work, which prompt to use, how much budget to allow, and whether Codex may edit code or only leave a plan. ### Cost Is A Design Signal It is tempting to treat model spend as accounting. That is the wrong abstraction. Cost is a product signal. Codex billing depends on how you access it. ChatGPT plans provide included usage, with eligible additional usage charged through credits or plan-specific billing. API-key workflows follow API pricing. The [Codex pricing page](https://developers.openai.com/codex/pricing) and [credit rate card](https://help.openai.com/en/articles/11481834-chatgpt-rate-card-business-enterpriseedu-credit-based-pricing) map credit-based usage to input tokens, cached input tokens, and output tokens. The [ChatGPT plan guidance](https://help.openai.com/en/articles/11369540-using-codex-with-your-chatgpt-plan) makes clear that limits and credit options vary by plan, and the Business [spend controls](https://help.openai.com/en/articles/20001155-managing-credits-and-spend-controls-in-chatgpt-business) exist because this has become an operational surface. A high bill is not automatically bad. Expensive work can still be worth doing. But unexplained spend is a smell. Possible contributors to unexplained spend Broad task Search expands without a clear stopping point Noisy repo Context search becomes the job Weak instructions Local rules get inferred late Weak validation Retries replace proof Unexplained spend Any of these can increase cost; traces help explain which one did The goal is not to make every run cheap. The goal is to make spend explainable. When spend spikes, I now ask four questions: - Was the task too broad? - Was the repo context too noisy? - Did the agent lack reusable instructions? - Was the validation loop too weak? ### Good, Maybe, Bad The useful question is not "can Codex do this?" The useful question is "is this worth paying Codex to attempt?" **Good tasks** are bounded, repeatable, and externally checkable: dependency bumps with clear changelogs, mechanical refactors, test repairs, small feature flags, docs updates tied to code changes, grep-able cleanup, and CI failures with narrow logs. **Maybe tasks** can work, but need tighter supervision: multi-file feature work, unfamiliar code paths, subtle product behavior, database migrations, performance improvements, cross-service integration, UI changes, and code that depends on undocumented conventions. **Bad tasks for unattended runs** are vague, high-impact, or hard to validate: broad architecture changes, security-sensitive rewrites, production incident response without tight human control, ambiguous product decisions, billing logic, data deletion flows, and anything where correctness is mostly social or organizational. The categories are not moral judgments. They are dispatch rules. Good work can go straight to a constrained run. Maybe work should be split into inspect, plan, implement, validate, stop. Bad work should stay human-led until the shape is clear enough to delegate a smaller piece. ### The Stack And Context Discipline Matter A naive workflow gives an agent a repo and asks it to figure things out. That works until the repository gets large, the task gets ambiguous, or the cumulative usage cost becomes visible. The better pattern is layered context. At the bottom, the repo needs short durable instructions. An `AGENTS.md` file should not be a novel. It should tell the agent how to work in this codebase: commands, conventions, test strategy, common traps, and review expectations. Above that, the agent needs efficient ways to inspect code. Symbol-level context is better than dumping entire files. Focused search is better than wandering through the tree. Compressed shell output is better than pasting pages of logs. The goal is not to starve the model of context. The goal is to feed it context with intent. For my workflow, the durable shape is boring and useful: - Tickets define the job. - Labels define risk and budget. - Repo instructions define local conventions. - CI defines the first correctness gate. - Draft PRs define the review boundary. - Humans decide what merges. ### Labels And Budgets Are Policy Once automation becomes routine, labels stop being metadata and start becoming policy. A label can answer questions that should not be improvised mid-run: - Can Codex run on this? - Should it edit code or only leave a plan? - What is the maximum budget? - How many test retries are allowed? - Can it touch production code? - Which validation commands are required? - Should failure create a follow-up ticket? That is where the workflow starts to look less like "AI coding" and more like a job system. That is the right direction. A human engineer should not supervise every keystroke, but the system should make it easy to constrain the work before it starts and audit the result after it finishes. Labels only become policy when the automation enforces them. A budget written in a ticket is not a spending cap by itself; the dispatcher needs to track usage and enforce the chosen time, retry, and cost limits. The ticket is the contract. Labels select the policy. The PR is the artifact. CI is the first judge. Review is the final one. ### The Human Role Moves Upstream A common mistake is to evaluate Codex by asking whether it can replace the human inside the loop. That is not the useful version. The human role moves upstream and downstream. Upstream, the human defines better work: clearer tickets, smaller scope, sharper success criteria, explicit non-goals, better repo instructions, and better task classification. Downstream, the human reviews better artifacts: focused diffs, summarized tradeoffs, test output, known risks, follow-up suggestions, and rejected alternatives. The middle gets automated. That middle is not all of software engineering, but it is a lot of the repetitive work around software engineering. ### The Failure Mode Is Unbounded Work Some failures are easy to spot: CI fails, the diff looks wrong, or review catches a misunderstanding. Passing CI can still miss incorrect behavior, security problems, or the wrong product decision. The more dangerous failure mode is unbounded work that looks productive. The agent reads too much, tries too many approaches, rewrites more than necessary, fixes adjacent issues, discovers unrelated problems, and produces a large PR with a plausible explanation. It spends real money doing all of that. This is why "does it work?" is an incomplete question. The better questions are: - Did it stay inside the requested scope? - Did it stop when it should have stopped? - Did it ask for clarification when the task became ambiguous? - Did it preserve reviewability? - Did the cost match the value of the work? - Would I run this class of task again? The useful improvement loop is not "rerun the prompt until it works." It is traces, evals, failure summaries, and human feedback changing the system. OpenAI's [agent improvement loop](https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop) points in that direction: make the evidence reusable, then use it to tighten the next run. ### What I Would Build Next The next version of my setup would make the control plane more explicit. I would not start by giving Codex more freedom. I would start by giving it fewer excuses to wander. 1. **A stricter ticket template.** Every automation-ready ticket should include scope, expected files, validation commands, non-goals, and maximum budget. 2. **Budget-aware labels.** A small docs fix and a multi-package refactor should not run under the same policy. 3. **Better failure summaries.** Failed runs should classify what happened: missing command, unclear instruction, flaky test, bad ticket, or genuine code ambiguity. 4. **Shorter reusable repo instructions.** The best instructions are not comprehensive. They are sharp enough to remove predictable mistakes. 5. **A stronger stop rule.** If the agent cannot find the right path quickly, it should stop and report. Continuing is not always diligence. Sometimes it is just spend. ### The Practical Conclusion Codex automation is viable when it is treated as an engineering system instead of a demo. That means the boring parts matter: tickets, labels, budgets, logs, tests, draft PRs, review rules, and failures. The exciting version of AI coding is a model that can do anything. The useful version is a system that knows what it is allowed to do, how much it is allowed to spend, how to prove that it worked, and when to stop. The goal is not autonomous coding. The goal is bounded delegation. And the bill is how I learned where those bounds were missing. ## Links - [https://api.photonfolio.com/api/v1/photonfolio/public/embeds/2ae9e53e-1bba-40c2-98f3-b733a66f44e6/image/](https://api.photonfolio.com/api/v1/photonfolio/public/embeds/2ae9e53e-1bba-40c2-98f3-b733a66f44e6/image/) (external) - [Codex cloud](https://developers.openai.com/codex/cloud) (external) - [GitHub integration](https://developers.openai.com/codex/integrations/github) (external) - [Codex GitHub Action](https://developers.openai.com/codex/github-action) (external) - [2025 DORA report](https://dora.dev/research/2025/dora-report/) (external) - [early-2025 developer productivity study](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) (external) - [2026 follow-up on experiment-design limits](https://metr.org/blog/2026-02-24-uplift-update/) (external) - [Codex pricing page](https://developers.openai.com/codex/pricing) (external) - [credit rate card](https://help.openai.com/en/articles/11481834-chatgpt-rate-card-business-enterpriseedu-credit-based-pricing) (external) - [ChatGPT plan guidance](https://help.openai.com/en/articles/11369540-using-codex-with-your-chatgpt-plan) (external) - [spend controls](https://help.openai.com/en/articles/20001155-managing-credits-and-spend-controls-in-chatgpt-business) (external) - [agent improvement loop](https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop) (external) ## Related Posts - No related posts linked from this article.