A couple weeks ago I wrote a blog post on the new “Async Daemon” feature in Marten. This post is a bit that I cut out of that post just describing the challenges I faced and what I did to slide around the problems. For all Marten users that have been asking me about writing their own subsystem to read and process events offline, you really want to read this post to understand why that’s much harder than you’d think and why you do probably want to just help make the async daemon solid.

The first challenge for the async daemon was “knowing” when there are new events that need to be processed by async projections. When a projection runs, it needs to process the events in the same order that they were captured in. Since the async daemon was inevitably going to use some sort of polling (NOTIFY/LISTEN in Postgresql was not adequate by itself) to read events out of the event table, we needed a very efficient way to be able to page the event fetching without missing events.

We started Marten with the thought that we would try to accomplish that by having the event store enqueue the events in a rolling buffer table that some kind of offline process would poll and read, but we were talked out of that approach in discussions with a Postgresql consultant who was helping us at work. Moreover, as I worked through other use cases to rebuild projections from scratch or add new projections later, we realized that the rolling buffer table would never have worked for the async daemon.

We also experimented with using sequential Guid’s as the global identifier for events in the event store with the idea that we would be able to use that to key off of for the projections by always querying for “Id > [last event id encountered].” In my testing I was unable to get the sequential Guid algorithm to accurately order the event id’s, especially under a heavy parallel load.

In the end, we opted to make the event store table in Marten use a sequential long integer as its primary key, and backed that with a database SEQUENCE. That gave us a more reliable way to “know” what events were new for each individual projection. In testing I figured out pretty quickly that the async daemon was missing events when there’s a lot of concurrent events streaming in because of event sequence id’s being reserved from in flight transactions. To counteract that problem, I ended up taking a two step process:

1. Limit the async daemon to only querying against events that were captured before some time of threshold (3 seconds is the default) to avoid missing events that are still in flight
2. When the async daemon fetches a new page of events, it actually tries to check that there are no gaps in the event sequence, and if there is, it pauses a little bit, and tries again until there are no gaps in the sequence or if the subsequent fetch turns up the exact same data (leading the async daemon to believe that the missing events were rejected).

Those two steps — as far as I can tell — have eliminated the problems I was seeing before about missing events in flight. It did completely ruin a family dinner at our favorite Thai restaurant when I couldn’t make myself stop thinking about how to slide around the problems in event ordering;)

The other killer problem was in trying to make the async daemon resilient in the face of potential connectivity problems and occasional projection failures without losing any results. I’ll try to blog about that in a later post.

I’m just thinking out loud here and hoping for some usable feedback.

I feel like Marten is getting very close to an official 1.0 release, and the latest Nuget is marked as 1.0-alpha. The Marten community voted on our minimum feature set for 1.0 earlier this year and we’ve finished everything on that list as of late July (right before I went on a long family vacation;)).

Some thoughts on the big 1.0:

• I’m a big believer in semantic versioning, so an OSS tool reaching 1.0 is a big deal because that starts the draconian versioning rules about backward compatibility. You want to get pretty close to a livable API before you throw that switch to 1.0.
• It’s a chicken and egg kind of conundrum. What we need right now is more users spawning yet more feedback about Marten. I’d love to have more usage before flipping Marten to 1.0, but we’ll get a lot more users after we release it as 1.0.
• In this day and age of package managers like Nuget, it’s a lot less friction to make more frequent releases and update your dependencies, so going 1.0 now knowing that 1.0.* bug fix releases and 1.* feature releases will be coming soon just isn’t that worrisome.
• I feel pretty good about the document database side of Marten, but the event store functionality is still churning and it’s less mature.
• We’re basically out of low hanging fruit kind of features on the document storage and Linq support
• My shop is doing the work right now to transition a very large web application from RavenDb to Marten. Right now I’m thinking that the first version of Marten that goes into production across all of that application will be declared to be 1.0.

All that being said, my best guess for an official Marten 1.0 release is around October 1st. Right now my biggest issues on my plate are really all around schema management and our database team’s requirements for the DDL generation. And more documentation, but that battle never ends. Plenty of pull requests are still flowing in, but I think I’m personally done with any kind of major feature work for awhile unless there’s noticeable demand from the community for specific features.

Marten 1.1 and Beyond

Based on our current issue list and requests from the Marten Gitter room, I think this list is where Marten goes next after the 1.0 release:

• Better support for child collections on documents
  • Support the Linq SelectMany() operator
  • Be able to do Include() against child collections
  • Maybe be able to use child collections in Select() transformations
• More types of event store projections — if you’re looking to get into doing some OSS work, I think these are our most approachable stories in the backlog
  • Project to a flat table for better reporting?
  • Projections that use the output of other projections
  • Arbitrary categorization of projected views (by customer, by region, etc.). Some of our users have already done this themselves, but it’s not in Marten itself yet
• Multi-tenancy support. My thinking right now is that we don’t directly put this into Marten, but make sure that there are adequate hooks to do this easily yourself. There’s a lot more information in the GitHub issue linked to above.
• Possibly try to support the Linq GroupBy() operator. That might also lead into some kind of map/reduce capability within Marten. We’ve had the feedback that “Marten isn’t a real document db because it doesn’t have map/reduce.” I think that’s nonsense, but we might very well need to have a better story for creating aggregated views into the document state — which may or may not be best done as some kind of formal map/reduce strategy.
• More support on document structural changes. Marten can already handle transformations of a single document type, but we’ll need to be able to later address document type names being changed, multiple document types getting combined (this is potentially a big deal for one of our systems), and whatever else we bump into next spring when we start optimizing a big system at work;)
• Being able to do document transformation with more than one document at time. This would mean being able to use related documents in the same Select() transformation. Also, we’ll probably need to be able to use Javascript transformations across multiple document types.

There’s some other things in the GitHub issue list, but the above is what I’m thinking about right now for 1.1 and beyond.

Thoughts? Concerns? Requests? Let us know either here, the GitHub issue list, or the Marten Gitter room.

This is an update to an earlier post on schema management using Marten.

At this point, I think the biggest challenges facing us at work for using Marten are strictly in the realm of database change management. To that end, we’re adding what will be a new package for command line tooling around Marten schema management and investigating possible usage of Sqitch to handle database migrations in our ecosystem. The command line usage shown in this post is in Marten master, but not pushed up to Nuget in any way yet. The Sqitch usage here is purely hypothetical.

When you’re using Marten, all the data definition language (DDL) for the underlying Postgresql database is generated to match by code within Marten. In development, you’d just run with the setting to auto-create database objects on the fly to match the code for faster iteration. For production deployment, however, you probably don’t get to do that and you’ll need some kind of database migration strategy to get the changes that your Marten application needs to the real database. That’s the gap that this post is trying to fill.

Command Line Tooling

My concept for supporting command line tooling suitable for build automation at this point is to publish a new library package called Marten.CommandLine that you can use to expose your own application and database through the command line. To use this tooling, follow these steps:

1. Create a new console application in your solution
2. Add the forthcoming Marten.CommandLine nuget
3. Add a reference to the projects in your system that would express the configuration for your Marten-enabled application
4. In the “Main()” entry point of your new console application, add code like this below to build up your Marten configuration via the StoreOptions class and then delegate to Marten to parse the command line arguments and execute the proper command:

    public class Program
    {
        public static int Main(string[] args)
        {
            var options = buildStoreOptions();

            return MartenCommands.Execute(options, args);
        }

        private static StoreOptions buildStoreOptions()
        {
            // build your own StoreOptions that 
            // establishes the configuration of your
            // Marten application
        }
    }

You can see an example of building the console application from the SampleConsoleApp project I used in the Marten codebase to test this functionality.

Once you have the code above, you’re actually ready to go. If you’re using the new dotnet CLI, running “dotnet run” in the root of your console application project yields this output listing the valid commands:

------------------------------------------------------------------------------------------------------------------------------------

  Available commands:
------------------------------------------------------------------------------------------------------------------------------------

   apply -> Applies all outstanding changes to the database based on the current configuration
  assert -> Assert that the existing database matches the current Marten configuration
    dump -> Dumps the entire DDL for the configured Marten database
   patch -> Evaluates the current configuration against the database and writes a patch and drop file if there are any differences
------------------------------------------------------------------------------------------------------------------------------------

If you’re not using the dotnet CLI yet, you’d just need to compile your new console application like you’ve always done and call the exe directly. If you’re familiar with the *nix style of command line interfaces ala Git, you should feel right at home with the command line usage in Marten.

For the sake of usability, let’s say that you stick a file named “marten.cmd” (or the *nix shell file equivalent) at the root of your codebase like so:

dotnet run --project src/MyConsoleApp %*

All the example above does is delegate any arguments to your console application. Once you have that file, some sample usages are shown below:

# Assert that the database matches the current database. This
# command will fail if there are differences
marten assert --log log.txt

# This command tries to update the database
# to reflect the application configuration
marten apply --log log.txt

# This dumps a single file named "database.sql" with 
# all the DDL necessary to build the database to
# match the application configuration
marten dump database.sql

# This dumps the DDL to separate files per document
# type to a folder named "scripts"
marten dump scripts --by-type

# Create a patch file called "patch1.sql" and
# the corresponding rollback file "patch.drop.sql" if any
# differences are found between the application configuration
# and the database
marten patch patch1.sql --drop patch1.drop.sql

In all cases, the commands expose usage help through “marten help [command].” Each of the commands also exposes a “–conn” (or “-c” if you prefer) flag to override the database connection string and a “–log” flag to record all the command output to a file.

My Current Thinking about Marten + Sqitch

Our team doing the RavenDb to Marten transition work has turned us on to using Sqitch for database migrations. From my point of view, I like this choice because Sqitch just uses script files in whatever the underlying database’s SQL dialect is. That means that Marten can use our existing “WritePatch()” schema management to tie into Sqitch’s migration scheme.

The way that I think this could work for us is first to have a Sqitch project established in our codebase with its folders for updates, rollbacks, and verify’s. In our build script that runs in our master continuous integration (CI) build, we would:

1. Call sqitch to update the CI database (or whatever database we declare to be the source of truth) with the latest known migrations
2. Call the “marten assert” command shown above to detect if there are outstanding differences between the application configuration and the database by examining the exit code from that command
3. If there are any differences detected, figure out what the next migration name would be based on our naming convention and use sqitch to start a new migration with that name
4. Run the “marten patch” command to write the update and rollback scripts to the file locations previously determined in steps 2 & 3
5. Commit the new migration file back to the underlying git repository

I’m insisting on doing this on our CI server instead of making developers do it locally because I think it’ll lead to less duplicated work and fewer problems from these migrations being created against work in progress feature branches.

For production (and staging/QA) deployments, we’d just use sqitch out of the box to bring the databases up to date.

I like this approach because it keeps the monotony of repetitive database change tracking out of our developer’s hair, while also allowing them to integrate database changes from outside of Marten objects into the database versioning.

EDIT 8/19: Couple other things came up about indexing yesterday that I added here.

For the purpose of this post, I’m only talking about the document database features in Marten. Our immediate need is to replace RavenDb before our busy season starts. Using the event store half of Marten probably won’t happen for us until next year.

The planets have finally aligned for us at work to begin converting our largest and most active application from RavenDb to Marten for persistence. I’m meeting with a couple of our teams this morning to talk over the transition, and this blog post is just an attempt to get my talking points prepared for them.

Moving to Marten

First off, Marten is really just a fancy data access library against the outstanding Postgresql database engine. Marten utilizes Postgresql’s JSONB type to efficiently store and query against our document data. We have deliberately based some of the most basic API usage on RavenDb where that made sense in order to make the transition to Marten easier for our teams, but Marten has deviated quite a bit in more advanced usage.

Here’s what I want our teams to know when we switch things over:

• Marten is ACID all the way down. No more WaitForNonStaleResults() nonsense, no more subtle bugs or unstable automated tests from stale data. Some folks have poked back at this in Marten by claiming that eventual consistency is necessary for performance or scalability. So far, all our experimentation suggests that Marten’s Postgresql-backed writes – with ACID – are measurably faster than RavenDb.
• Marten does not force you to declare which indexes you want to use for any given query. Postgresql itself can figure out the most efficient execution plan for itself. This is going to be advantageous for us in a couple ways. First by letting us rip a lot of RavenDb index code out. Secondly by making it much easier to optimize database performance without having to have so much impact on the code like it is today with RavenDb.
• We need more documentation and blog posts on this topic, but it is perfectly possible to use the relational database features of Postgresql where that’s still valuable.
• If it’s useful, it is possible to use Dapper in conjunction with Marten and even in the same unit of work/transaction.
• Just like RavenDb, Marten’s IDocumentSession is effectively the unit of work and should be scoped to a logical transaction. In most cases in our systems that effectively translates to an IDocumentSession per HTTP request or service bus message.
• There is no hard request throttling in Marten. You should be aware of how many network round trips you’re making during a single operation and there are diagnostics to track that, but Marten will not blow up in production because an operation happened to make too many requests.
• There’s no equivalent to RavenDb’s embedded data store option. That was the killer feature in RavenDb we’re going to miss the most. Fortunately, it’s pretty easy to spin up Postgresql on your own box. For automated testing scenarios where today we just use a brand new RavenDb data store, we’ll just take advantage of Marten’s “database cleaner” to wipe out state in between tests. In a way, this will simplify some of our testing against distributed systems. If this becomes a problem for test performance, we have a couple fallback plans to either host Postgresql in disposable Docker images or to enhance our testing harnesses to leapfrog clean schemas between tests.
• Most importantly, if there’s something in Marten you don’t like, you can either do a pull request or at least raise an issue in GitHub where I’ll see it and we can get it fixed. OSS FTW!
• We don’t use this in our internal systems (but we should), but the “Include()” feature in Marten for fetching related documents in one round trip is quite different than Raven’s.
• Batch querying in Marten is more explicit and different mechanically than RavenDb’s “Futures.” We should be using this feature to reduce network chattiness between applications and the database.
• I am highly recommending the usage of the Compiled Query feature in Marten that has no equivalent in RavenDb for better runtime performance and even as a declarative query model. This feature can be used in combination with “Include()” and batch querying to maximize the performance of your Marten backed persistence.
• You can use any tooling you want that’s compatible with Postgresql to poke and prod a Marten-ized database. I just use pgAdmin, but Datagrip or even just Visual Studio is useful.
• Marten has quite a few more useful diagnostic abilities you can use to analyze the SQL being generated or track database activity by session. In a later blog post, I’ll talk about the reusable recipe we’ve built for Marten integration into FubuMVC applications.

Why we’re getting off of RavenDb

I’ve been asked several times since we started working on Marten in public what it would take for us to change our minds and continue with RavenDb. I think it’s quite possible that Voron will make a positive difference, but as I’ll explain a little below, we just don’t trust RavenDb’s quality and software engineering practices.

So why are we wanting to move away from RavenDb?

• We’ve had multiple day+ outages due to RavenDb indexes getting corrupted and being unable to rebuild. That in a nutshell is more than enough reason to move on.
• We’ve been concerned for years with RavenDb’s internal quality. We’ve experienced a number of regression bugs when changing versions of RavenDb to the point where we’re unwilling to even try upgrading it.
• Their release and versioning strategies are not consistent with Semantic Versioning, so you never know if you’re going to get breaking changes in minor or revision level version changes
• Unresponsive support when we’ve had production issues with RavenDb
• We’ve not had a lot of success with the DevOps type tooling around RavenDb (replication, etc.) and we’re hopeful that adopting Postgresql helps out on that front.
• Resource utilization. RavenDb requires a lot of handholding to keep the memory utilization reasonable. Naive usage of RavenDb almost invariably leads to problems.
• The stale data issue as a result of RavenDb’s eventual consistency strategy has been a major source of friction for us