The first version of the workplace assistant wasn’t an agent at all. It was a scheduled message that appeared in the morning with the recorded meetings from the day before, pending calendar invites, and scheduling conflicts.
We tested it within our small team: design, engineering, support, and product. The response was tepid. We started adding capabilities through structured commands. Eventually we realized no one was going to figure out that UX. That’s when I thought this could actually be an agent.
The team was involved in developing the product. I was doing all the coding, working with Claude Code and Codex.
I wrote a simple agent loop based on an Anthropic post about building agents. Then I started writing tools to search for meetings, fetch transcripts, and respond to calendar invites. It was just so much more natural to use. You could talk to it, and it would tell you what it could do.
Keeping the agent working
The agent kept overrunning its context limit and erroring out. I moved some of the work into tools with their own LLM calls so it didn’t all have to happen in the main agent conversation.
Transcript analysis, email search, and OneDrive file analysis can all process tons of tokens, sometimes using a kind of map-reduce. They return a short message to the main agent with the final answer. These tools are kind of crude subagents. We did a lot of human testing, both me and my close collaborators, to check that they were returning what the agent needed.
Getting myself out of manual testing
Testing meant using a Teams simulator that had to be restarted constantly. I was the bottleneck because I was testing changes manually. So I built a CLI that let Claude Code run the agent directly, without a separate compilation step or the simulator.
Once the CLI worked, Claude Code could reproduce and fix bugs, test new features, and optimize prompts. It also needed less of the guidance I’d been giving it while it was coding.
I added evals so Claude Code could verify that its changes had the intended effect. If we were seeing validation errors or incorrect tool calls, it needed a way to check whether a change fixed them.
I still checked how Claude had verified the change: which commands it ran, what the output looked like, which tools the agent called, and how many turns it took.
Transcripts were a recurring example. The agent might get to the answer, but along the way it would search my calendar and try different inputs to the transcript tool over and over. I don’t remember the exact changes, but it involved adjusting the tool parameters and descriptions, then testing repeatedly until we could see that it worked.
Giving Claude enough information to fix bugs
A bug report was usually a vague complaint or a screenshot. I had to figure out how to reproduce it myself. Once I added bug reporting with traces, Claude could replay what happened, adjust the tools, and check whether the changes fixed it. The data Claude could access was heavily redacted, and users explicitly opted in to sharing it.
Every morning I was looking at the same signals: logs, traces, and bug reports. Claude had added routines, and Opus 4.5 had gotten good enough at directing subagents to generate pull requests automatically. I also had good local tooling to validate changes and prevent regressions.
Across the agent and its embedded MCP, I’ve now merged thousands of pull requests. Each PR goes through a design review and two automated reviews in CI. Each release gets end-to-end testing before deployment.
I still review fixes that change behavior or touch sensitive systems, and decide whether the verification is sufficient. There are a lot of subtler judgment calls too.
The routine investigates further on its own and brings me a menu of options when it needs my judgment: feature requests, unsolved bugs, optional investigations, and optimization ideas. I decide which of those to pursue and where to dig into stability, performance, or cost.
I didn’t start with all this in place. I started with a morning message that people weren’t that interested in. The product changed through use, and so did the way I built it.