A day in the life building a Critter Stack release w/ AI

From RollingStone.com

When I start work in the mornings, but after feeding the family dog and well before my wife and our kids get up, I have a little Claude routine to give me a summary of activity across the JasperFx GitHub organization (home of the Critter Stack tools plus some private stuff for JasperFx Software). Today I was greeted with a pretty good backlog of open pull requests and a few issues from the community plus a significant request from a big client to address some memory pressure issues in their extremely high volume system. Long story short, there’s more than enough work to justify a round of Nuget releases today.

I’m also trying to clear the decks of any open issues or pull requests before we start work on new major versions of all the tools in the .NET 11 wave starting when Microsoft releases that on November 10th.

As a kind of fun exercise, I thought I’d try to keep a running diary today of how I went about that to show — clumsy as it might be — how I’m trying to manage AI agents to work through all of that across 6-7 different repositories and maybe about a 100 different Nuget packages today.

Dogfooding Stoat

First off, I wanted to do more dogfooding today of Stoat, the Critter Stack’s work-in-progress agent orchestration and durable agentic memory tool. With a single supervising agent session in Claude Desktop, I talked through a plan for today that included the issues I wanted to address, the pull requests that I would be trying to review and merge, and the steps for publishing and consuming Nugets between the different Critter Stack tools.

Below is a screen shot of the Stoat “plan” at about 8:45 in my morning with a couple agents already working:

There’s a mix of GitHub issues that I mostly trust an agent to “just” deal with, but for today I’m going with “let me review and merge all pull requests.” A big chunk of the plan though is for me to be reviewing existing pull requests separately. The Nuget publishing is part of the Stoat plan, and that’s gated by my manual review work and part of the plan. Stoat is able to launch agents on approved issues though.

There’s a bazillion of these agent orchestration tools out there and I’m not yet convinced that Stoat is hugely worthwhile outside of my own workflow, but we’ll see. It is part of the Critter Stack strategy for AI assisted development with Event Modeling and Spec Driven Development though.

About Pull Requests and CI via GitHub Actions

I’m still using a very GitHub-centric workflow, and I prefer to keep as much information as possible inside of GitHub issues or pull requests with Markdown. Likewise, we use pull requests as primarily a tracking tool for internal work and certainly for external contributions, but now it’s time to talk about compromises.

I’ve very aggressively used AI agents to improve and expand the number of test suites we have running in CI *on every pull request and push to main* like we’ve done for over a decade. At this point Wolverine has 46 separate CI actions and Marten has 30 some. Because of the way every yokel like me has jacked up their CI checks and the rate of commits with AI tools, GitHub Actions now has a pretty long queue wait for runners and I can easily run into CI taking up to 2 hours plus for a single Wolverine pull request at busier times of day.

So here’s the compromise. I’m going to disable GitHub Actions altogether for the Critter Stack for just today while I’m working on the plan, then turn on CI just in time to act as a quality gate prior to making any Nuget release.

I’ve since enabled the CI runners again for everything but Wolverine that hasn’t quite finished yet.

Some thoughts and notes about this:

• It’s required some effort to create “CommitGate / PushGate” scripts that try to optimize automated test suite execution for local execution as opposed to the CI scripts that depend on having a separate GitHub action for basically everything
• There are some new services popping up to be a paid subscription to take the place of GitHub Actions for faster CI, and I’m tempted to get that for JasperFx
• It’s making me think a lot about risk mitigation in pushes and releases. It’s also leading to people getting more disciplined and parsimonious about what test runs apply to what changes rather than just letting CI do a big bang test run on every single change

Stoat got some enhancements today to understand when CI is turned on (in addition to its item by item capability to auto-merge pull requests when CI goes green) and to do more “local gating.” The next thing I need to add to Stoat is to understand when “local gating” is heavyweight so that it limits the active work to only one heavyweight activity at a time. For example, if the box is running all of the Wolverine tests (there’s quite a bit of parallelism in our new local gating), that’s it, that’s all the box can be doing because of the resource utilization on the box.

First couple hiccups with Stoat

The UI is gawdawful. Fundamental things do not change with AI, quality in any software tool still requires feedback and iteration. And also, I’m still a rank amateur at UI/UX. Fear not, other people with better UI/UX skills will soon be helping with Stoat. I’m making a lot of notes today as I go on this based on dogfooding today.

The context switching burns

I’ve talked to several people who say the same thing. I’ll spin up several agents, then duck and weave going back and forth to answer questions to each. It’s even worse when you add in trying to sequentially review several pull requests in a row that are all dealing with different areas of functionality. Trying to make complicated decisions way too fast is definitely not part of our new AI world that I especially enjoy.

10:06 AM

GitHub seems to be having a lot of errors right now. Ugh.

Okay, it got better.

Big time scalability issue with Marten

A JasperFx client has been telling me for a week that they’re seeing memory utilization problems in production during rollouts. They were finally able to give us some pretty concrete reports of where the memory usage was, so that launched an investigation into a couple different ways of alleviating that.

But, addressing that memory usage meant that I needed to do some significant stress testing to try to simulate the problems and measure the before and after impact of our theoretical changes. I had to modify Stoat a little bit so that it could “drain” all ongoing work before doing the heavy stress testing. I had already invested in teaching it how to do local Nuget packing from upstream projects to combine build products with downstream code without having to deal with the friction of formally publishing alpha Nugets to Nuget.org.

Cross repository development always comes at a cost and you should probably always be a bit cautious about that. But that’s where I’m at with the Critter Stack, so here we go.

If you’re curious, that stress testing and memory optimization led to this pull request today.

I think this will come up a couple more times today, but a big issue with agentic development is respecting resource usage on the machine. I’ve occasionally seen the Claude + Docker combination swamp my box’s memory when I’ve had too many agents doing heavy work at one time — and I have 128gb of RAM.

Oh, and it’s only anecdotal, but PostgreSQL runs far more lean in Docker for local development than SQL Server. And Oracle is an absolute beast. Glad I do far more development with PostgreSQL than the others, but SQL Server is a real issue for my local development sometimes.

The AI agents make it a lot easier for you to build deep quality into tools like the Critter Stack because you can just casually say “hey, go build a stress test and check out the memory usage before and after this change” or “hey, go do some exploratory testing against X, Y, and Z using permutations A, B, and C” that have very frequently helped find and fix previously unknown bugs for us. But, you have to actually think to do that and to have relatively specific instructions for your AI agents. AI doesn’t magically seek out and find problems for you without specific prompting.

More context switching

I fielded some questions about idempotency protections with Wolverine from a client — which Wolverine recently added originally for our new recurring cron messaging, then expanded quite a bit for a different client. See Logical Message Deduplication with Wolverine for more information.

I also need to get more serious about our homegrown CRM. One thing at a time.

Stoat telling me what’s going on!

Mostly for CritterWatch development, we had trouble a while back with test suites hanging within AI agent sessions, and it wasn’t clear if the tests were slow or dead or Docker ran out of memory or whatnot. In no small part because of that, I invested some serious “vibe coding” time into the Bobcat Supervisor library that uses the Microsoft Testing Platform to give a lot more visibility into long running tests and to even help parallelize tests, identify flaky tests, and do some selective test retries. It even “knows” in some cases to do Docker resets in the case of certain test failures.

Anyway, part of the Bobcat Supervisor that’s not applied to all Critter Stack projects is that Stoat can show us a little bit of progress tracking in its UI for where it’s at when it’s running these long ass test suites:

11:09 AM, time to review pull requests

I’ve got 16 open pull requests to Wolverine. One is for Wolverine 7.0 I’ll ignore (to switch to the new Dekaf library underneath our Kafka transport that JT did for us). A couple are for pull requests built out from the Stoat plan earlier today. I’m going to focus first on the pull requests from other contributors that I’m trying to get into this release.

I’ll admit that I do use AI to help me review pull requests with a prompt like this one:

When I get pull requests — and especially now that AI agents is lowering the bar for people to just try to do things themselves — I generally triage them into a handful of buckets:

1. It’s all good, just pull it in
2. It’s mostly good, but needs some immediate follow ups or changes, in which case I merge it then immediately do the follow up. I’ll add a comment to the original PR, but I don’t really want to have to wait on someone else to happen to see the comment then get around to doing the follow up. I did this a couple times today for documentation and some minor edge cases. I’ve done it in the past for performance reasons.
3. Build a superseding pull request instead because I think the solution needs to be done differently. When this happens I try to make sure the original author gets a co-author credit on the new pull request and add a reference to the old pull request pointing at the new pull request before closing the original. I do this a time or two a week. I did that today for the Marten scalability thing I’m working up today because I was afraid the approach in the original pull request might lead to potential dead locks under load.
4. Reject it with a note explaining why we’re doing that. I did that once today for something I just admittedly didn’t want to have to support, but that’s very rare. Sometimes I’ll say thank you, but that the work in question is just too risky for that to be coming in from an external contributor. If I ever do that to you, just know that that’s also probably something you don’t want to be responsible for anyway!

I’ll also frequently have an AI agent add more tests before taking in the pull request. The killer problem with community contributions is that folks are quite naturally focused on tactical problems that impact them while you the OSS maintainer really need to be strategic and think about the bigger picture. I’m also hopefully more aware of edge cases and problems that have bit us in the past and generally have a good idea about where I think the technical risks are that might not be obvious from the outside.

I know some people are seeing a lot of AI slop pull requests to their OSS projects, but I’ve been mostly happy with what we get. I don’t know if that’s because the Critter Stack community is just a bunch of discerning technical folks with good judgement, AI tools have gotten much better, we just have a great codebase where the right patterns are very clear, or we’re just not popular enough to attract the slop that more visible OSS projects are getting inundated with.

A second pull request did have a judgement call. A user is requesting to add a new partial index to a Wolverine table in our scheduled messaging support. It’s probably a good idea, but that requires a schema change and we usually say that requires either an “opt in” gating flag or a major release. But, since it’s an additive change that wouldn’t impact functionality if you didn’t apply it, I’m going to let it through. There’s a little bit of a chance that Wolverine’s ability to automatically apply schema changes could bite somebody in production, but this is a partial index that shouldn’t be a problem. I think this is an acceptable risk compared to the potential pay off in improved performance.

Watch me be wrong though 😦

And actually, this is a bit of egg on my face. One of the things I try to do is notice any kind of trends in issues or potential trends in an issue and go do more exploratory testing to try to head off problems. In the case of the pull request above, it was something we’d already dealt with in our SQL Server support for Wolverine, but I failed to apply to other databases. What I will do to follow up later is have an AI agent go explore whether that’s a problem we should address in all the other database engines we support (Oracle, MySql, Sqlite).

I hate OpenAPI

I hate OpenAPI with a passion. Tedious, nit picky stuff. I guess thank you to people who are very detail oriented about OpenAPI and help us patch holes in Wolverine.HTTP’s OpenAPI metadata generation.

No, you can’t just vibe code this stuff

This one: https://github.com/JasperFx/wolverine/pull/5349. The new Dunning Kruger poster children are the folks who write into Reddit or YouTube comments and scoff that you should never use something off the shelf if you can just get AI to write it for you.

There’s so much work we do to Wolverine or Marten with AI that only starts from reported problems in real usage. Often from oddball production mishaps or Kubernetes scale up / scale down incidents. Or network hiccups like temporarily losing connectivity to a database or message broker. If you vibe code something, you’re getting the happy path. That’s going to give you a sugar rush high, but that’s a long ways away from a tool like Wolverine that’s been beaten on by real usage over time. Not to mention Marten’s support for very high volumes that all the newer, upstart event store tools haven’t had to face yet.

Some of the things that have generated Wolverine bugs + improvements lately:

• CancellationToken passing. It’s harder sometimes than you think and easy to do wrong
• Not losing work that’s in flight when the system is being shutdown
• Order of operations issues on system shutdown
• All the weird problems that pop up in extreme loads that you never see in your normal development. Even when you *think* you’ve done load testing because the real world is always more complicated

Another frustrating thing to me is how AI tools are turning up latent bugs in what I would have described as very stable code. It’s apparently important to keep an open mind about long running codebases and still consider improvements to code you think is “done.”

If you haven’t seen this, the PostgreSQL 19 release has been somewhat delayed because AI tools keep finding existing bugs in really old code. I’d call PostgreSQL one of the very best OSS projects on the planet that involves some of the best people around. I’m bringing this up in a way because reading about their struggles made me feel better.

11:30 AM — Lunch, indulging in post-US election Schadenfreude, XBox time, finally working out

I kicked off the stress test on Marten finally too and tried to mostly walk away from the computer.

About the US election this week, I’m genuinely more hopeful than I’ve been about the US and Texas specifically than I’ve been in over a decade. Seriously, the GOP did some very serious harm to public education in Texas, and this Dad is hopeful that the new state government starts to turn that around.

Code behaves differently under extreme data volumes sometimes

Thank you, I’ll be here all week! Here’s some of the results of the stress tests against Marten’s Async Daemon when using the fancy pants per-tenant partitioning for extreme data loads:

AI tools again do make it pretty easy to spin up stress tests and memory testing and what not, but it’s again still up to you to *think* to have it do that.

In this case, I think the new “Budget” approach up there is going to make our client happy. Assuming that I can get this release ready today.

(the Marten release got made a couple hours ago, but I just realized that I missed generating release notes. On it!)

In the words of the late, great Tom Petty…

“The wait is the hardest part…”

Stoat has some functionality to wait for new Nuget versions to appear, then release downstream items for execution later in the plan. Waiting for Nuget indexing used to be a bottleneck for me in these kinds of days, but Nuget got faster then GitHub Actions got overloaded to be the new bottleneck.

And we can continue!

Just pointing out that Stoat can indeed kick off downstream work when it can tell that upstream dependencies are in place. I won’t say that Stoat exactly worked smoothly today, but it got better today!

And now there’s a pull request for me to review:

And I had the clever thought that I’d have Stoat send me Discord messages when it needs my attention. The jury is mixed on whether this was worthwhile.

More pull requests, this time for Marten

These were a lot easier. Marten is pretty stable and battle tested now and we aren’t adding a ton of new functionality. Some things still pop up though.

More waiting, but at least there’s feedback

If you squint really hard at the two cards in the first column, you can see a progress bar telling us where the test suite process is at. If you zoomed into a single card, it gives you more context than just that. Likewise, in the card at the bottom right, it’s monitoring CI jobs for that pull request (I’d enabled CI again at this point) and letting me know how many CI jobs had finished successfully, failed, or yet to finish.

This kind of status tracking has been helpful.

Unsurprisingly, my box is running hot and the fans are blowing hard as it runs a ton of Wolverine tests and Polecat on SQL Server tests at the same time.

Late in the work day

Almost there, stay on target…

And there’s a security advisory in some kind of .NET package from Microsoft that’s making a quality gate fail. No big deal to update, but AI agents are definitely finding a lot more security vulnerabilities that were already there and I’m finding myself having to react to those more often in Critter Stack work. We had never had a security advisory with Marten until this year, but now I think we’ve had 3 or 4 due to oddball SQL injection vulnerabilities in Marten’s full text search or places where we use JSONPath or PostgreSQL containment operators. And that’s after I tried to get Fable to identify a couple places where we were vulnerable.

It’s all to the good, but I’m definitely seeing quite a few bug reports to Wolverine especially for latent bugs that I don’t think anyone (including me) ever would have noticed without AI finding them. Likewise, I mentioned how it’s now cheaper to tell an AI agent to go write chaos monkey tests or stress tests or whatnot. Now we’re getting issues from people doing that themselves in their own application hardening or PoC work and passing that along to us.

I will say that that seems to be diminishing lately? Knock on wood.

Almost there.

There’s probably a couple more hours of stuff to run, but I’m about at the point where I think Stoat and Claude can handle the rest of it without me.

Below is a screenshot of me using the Bobcat UI just to monitor ongoing tests as I had Stoat run a “LocalGate” of Polecat right before publishing new Nugets.

Wrapping up…

There’s still more going on, but I’m done with productivity for the day! Hey, I don’t know if this was the slightest bit interesting to anyone but me, but I kind of enjoyed doing the little diary for today. I don’t plan for an encore though!