This is an update to an earlier post on schema management using Marten.

At this point, I think the biggest challenges facing us at work for using Marten are strictly in the realm of database change management. To that end, we’re adding what will be a new package for command line tooling around Marten schema management and investigating possible usage of Sqitch to handle database migrations in our ecosystem. The command line usage shown in this post is in Marten master, but not pushed up to Nuget in any way yet. The Sqitch usage here is purely hypothetical.

When you’re using Marten, all the data definition language (DDL) for the underlying Postgresql database is generated to match by code within Marten. In development, you’d just run with the setting to auto-create database objects on the fly to match the code for faster iteration. For production deployment, however, you probably don’t get to do that and you’ll need some kind of database migration strategy to get the changes that your Marten application needs to the real database. That’s the gap that this post is trying to fill.

 

Command Line Tooling

My concept for supporting command line tooling suitable for build automation at this point is to publish a new library package called Marten.CommandLine that you can use to expose your own application and database through the command line. To use this tooling, follow these steps:

1. Create a new console application in your solution
2. Add the forthcoming Marten.CommandLine nuget
3. Add a reference to the projects in your system that would express the configuration for your Marten-enabled application
4. In the “Main()” entry point of your new console application, add code like this below to build up your Marten configuration via the StoreOptions class and then delegate to Marten to parse the command line arguments and execute the proper command:

    public class Program
    {
        public static int Main(string[] args)
        {
            var options = buildStoreOptions();

            return MartenCommands.Execute(options, args);
        }

        private static StoreOptions buildStoreOptions()
        {
            // build your own StoreOptions that 
            // establishes the configuration of your
            // Marten application
        }
    }

You can see an example of building the console application from the SampleConsoleApp project I used in the Marten codebase to test this functionality.

Once you have the code above, you’re actually ready to go. If you’re using the new dotnet CLI, running “dotnet run” in the root of your console application project yields this output listing the valid commands:

------------------------------------------------------------------------------------------------------------------------------------

  Available commands:
------------------------------------------------------------------------------------------------------------------------------------

   apply -> Applies all outstanding changes to the database based on the current configuration
  assert -> Assert that the existing database matches the current Marten configuration
    dump -> Dumps the entire DDL for the configured Marten database
   patch -> Evaluates the current configuration against the database and writes a patch and drop file if there are any differences
------------------------------------------------------------------------------------------------------------------------------------

 

If you’re not using the dotnet CLI yet, you’d just need to compile your new console application like you’ve always done and call the exe directly. If you’re familiar with the *nix style of command line interfaces ala Git, you should feel right at home with the command line usage in Marten.

For the sake of usability, let’s say that you stick a file named “marten.cmd” (or the *nix shell file equivalent) at the root of your codebase like so:

dotnet run --project src/MyConsoleApp %*

All the example above does is delegate any arguments to your console application. Once you have that file, some sample usages are shown below:

# Assert that the database matches the current database. This
# command will fail if there are differences
marten assert --log log.txt

# This command tries to update the database
# to reflect the application configuration
marten apply --log log.txt

# This dumps a single file named "database.sql" with 
# all the DDL necessary to build the database to
# match the application configuration
marten dump database.sql

# This dumps the DDL to separate files per document
# type to a folder named "scripts"
marten dump scripts --by-type

# Create a patch file called "patch1.sql" and
# the corresponding rollback file "patch.drop.sql" if any
# differences are found between the application configuration
# and the database
marten patch patch1.sql --drop patch1.drop.sql

In all cases, the commands expose usage help through “marten help [command].” Each of the commands also exposes a “–conn” (or “-c” if you prefer) flag to override the database connection string and a “–log” flag to record all the command output to a file.

 

My Current Thinking about Marten + Sqitch

Our team doing the RavenDb to Marten transition work has turned us on to using Sqitch for database migrations. From my point of view, I like this choice because Sqitch just uses script files in whatever the underlying database’s SQL dialect is. That means that Marten can use our existing “WritePatch()” schema management to tie into Sqitch’s migration scheme.

The way that I think this could work for us is first to have a Sqitch project established in our codebase with its folders for updates, rollbacks, and verify’s. In our build script that runs in our master continuous integration (CI) build, we would:

1. Call sqitch to update the CI database (or whatever database we declare to be the source of truth) with the latest known migrations
2. Call the “marten assert” command shown above to detect if there are outstanding differences between the application configuration and the database by examining the exit code from that command
3. If there are any differences detected, figure out what the next migration name would be based on our naming convention and use sqitch to start a new migration with that name
4. Run the “marten patch” command to write the update and rollback scripts to the file locations previously determined in steps 2 & 3
5. Commit the new migration file back to the underlying git repository

I’m insisting on doing this on our CI server instead of making developers do it locally because I think it’ll lead to less duplicated work and fewer problems from these migrations being created against work in progress feature branches.

For production (and staging/QA) deployments, we’d just use sqitch out of the box to bring the databases up to date.

I like this approach because it keeps the monotony of repetitive database change tracking out of our developer’s hair, while also allowing them to integrate database changes from outside of Marten objects into the database versioning.

 

 

EDIT 8/19: Couple other things came up about indexing yesterday that I added here.

For the purpose of this post, I’m only talking about the document database features in Marten. Our immediate need is to replace RavenDb before our busy season starts. Using the event store half of Marten probably won’t happen for us until next year.

The planets have finally aligned for us at work to begin converting our largest and most active application from RavenDb to Marten for persistence. I’m meeting with a couple of our teams this morning to talk over the transition, and this blog post is just an attempt to get my talking points prepared for them.

Moving to Marten

First off, Marten is really just a fancy data access library against the outstanding Postgresql database engine. Marten utilizes Postgresql’s JSONB type to efficiently store and query against our document data. We have deliberately based some of the most basic API usage on RavenDb where that made sense in order to make the transition to Marten easier for our teams, but Marten has deviated quite a bit in more advanced usage.

Here’s what I want our teams to know when we switch things over:

• Marten is ACID all the way down. No more WaitForNonStaleResults() nonsense, no more subtle bugs or unstable automated tests from stale data. Some folks have poked back at this in Marten by claiming that eventual consistency is necessary for performance or scalability. So far, all our experimentation suggests that Marten’s Postgresql-backed writes – with ACID – are measurably faster than RavenDb.
• Marten does not force you to declare which indexes you want to use for any given query. Postgresql itself can figure out the most efficient execution plan for itself. This is going to be advantageous for us in a couple ways. First by letting us rip a lot of RavenDb index code out. Secondly by making it much easier to optimize database performance without having to have so much impact on the code like it is today with RavenDb.
• We need more documentation and blog posts on this topic, but it is perfectly possible to use the relational database features of Postgresql where that’s still valuable.
• If it’s useful, it is possible to use Dapper in conjunction with Marten and even in the same unit of work/transaction.
• Just like RavenDb, Marten’s IDocumentSession is effectively the unit of work and should be scoped to a logical transaction. In most cases in our systems that effectively translates to an IDocumentSession per HTTP request or service bus message.
• There is no hard request throttling in Marten. You should be aware of how many network round trips you’re making during a single operation and there are diagnostics to track that, but Marten will not blow up in production because an operation happened to make too many requests.
• There’s no equivalent to RavenDb’s embedded data store option. That was the killer feature in RavenDb we’re going to miss the most. Fortunately, it’s pretty easy to spin up Postgresql on your own box. For automated testing scenarios where today we just use a brand new RavenDb data store, we’ll just take advantage of Marten’s “database cleaner” to wipe out state in between tests. In a way, this will simplify some of our testing against distributed systems. If this becomes a problem for test performance, we have a couple fallback plans to either host Postgresql in disposable Docker images or to enhance our testing harnesses to leapfrog clean schemas between tests.
• Most importantly, if there’s something in Marten you don’t like, you can either do a pull request or at least raise an issue in GitHub where I’ll see it and we can get it fixed. OSS FTW!
• We don’t use this in our internal systems (but we should), but the “Include()” feature in Marten for fetching related documents in one round trip is quite different than Raven’s.
• Batch querying in Marten is more explicit and different mechanically than RavenDb’s “Futures.” We should be using this feature to reduce network chattiness between applications and the database.
• I am highly recommending the usage of the Compiled Query feature in Marten that has no equivalent in RavenDb for better runtime performance and even as a declarative query model. This feature can be used in combination with “Include()” and batch querying to maximize the performance of your Marten backed persistence.
• You can use any tooling you want that’s compatible with Postgresql to poke and prod a Marten-ized database. I just use pgAdmin, but Datagrip or even just Visual Studio is useful.
• Marten has quite a few more useful diagnostic abilities you can use to analyze the SQL being generated or track database activity by session. In a later blog post, I’ll talk about the reusable recipe we’ve built for Marten integration into FubuMVC applications.

 

Why we’re getting off of RavenDb

I’ve been asked several times since we started working on Marten in public what it would take for us to change our minds and continue with RavenDb. I think it’s quite possible that Voron will make a positive difference, but as I’ll explain a little below, we just don’t trust RavenDb’s quality and software engineering practices.

So why are we wanting to move away from RavenDb?

• We’ve had multiple day+ outages due to RavenDb indexes getting corrupted and being unable to rebuild. That in a nutshell is more than enough reason to move on.
• We’ve been concerned for years with RavenDb’s internal quality. We’ve experienced a number of regression bugs when changing versions of RavenDb to the point where we’re unwilling to even try upgrading it.
• Their release and versioning strategies are not consistent with Semantic Versioning, so you never know if you’re going to get breaking changes in minor or revision level version changes
• Unresponsive support when we’ve had production issues with RavenDb
• We’ve not had a lot of success with the DevOps type tooling around RavenDb (replication, etc.) and we’re hopeful that adopting Postgresql helps out on that front.
• Resource utilization. RavenDb requires a lot of handholding to keep the memory utilization reasonable. Naive usage of RavenDb almost invariably leads to problems.
• The stale data issue as a result of RavenDb’s eventual consistency strategy has been a major source of friction for us

 

The road to Marten 1.0 continues with a discussion of the indexing options that we directly support against Postgresql.

We’re aiming to make Marten be usable for a wide range of application scenarios, and an obvious one is to make querying faster. To that end, Marten has direct support for adding a couple different flavors of indexes to optimize querying.

In all cases, Marten’s schema migration support can detect changes, additions, and removals of index definitions.

Calculated Index

After getting some feedback from a 2nd Quadrant consultant, the recommended path for optimizing queries against JSON documents in Marten is to use a Postgresql calculated index, which Marten can build for you with:

    var store = DocumentStore.For(_ =>
    {
        _.Connection(ConnectionSource.ConnectionString);
        _.Schema.For<Issue>().Index(x => x.Number);
    });

Marten creates this index behind the scenes against the Issue storage table:

CREATE INDEX mt_doc_issue_idx_number ON public.mt_doc_issue 
    ((CAST(data ->> 'Number' as integer)));

The advantages of using a calculated index are that you’re not duplicating storage and you’re causing fewer database schema changes as compared to our original “Duplicated Field” approach that’s described in the next section.

It’s not shown here, but there is some ability to configure how the calculated index is created. See the documentation on calculated indexes for an example of that usage.

I’ve been asked several times what it would take for Sql Server to add before it could support Marten. The calculated index feature as applicable to the JSONB data type isn’t explicitly necessary, but it’s a big advantage that Postgresql has over Sql Server at the moment.

Duplicated Field

Marten’s original approach was to optimize querying against designated fields by just duplicating the value within the JSON document into a separate database table column, and indexing that column. Marten does this behind the scenes when you use the Foreign Key option. Some of our users will opt for a duplicated field if they want to issue their own queries against the document table without having to worry about JSON locators.

To make a field duplicated, you can either use the [Duplicated] attribute:

    public class Team
    {
        public Guid Id { get; set; }

        [DuplicateField]
        public string Name { get; set; }
    }

Or you can specify the duplicated fields in the StoreOptions for your document store:

    using (var store = DocumentStore.For(_ =>
    {
        _.Connection(ConnectionSource.ConnectionString);

        _.Schema.For<User>().Duplicate(x => x.UserName);
    }))
    {

    }

If you decide to add a duplicated field to an existing document type, Marten’s schema migration support is good enough to add the column and fill in the values from the JSON document as part of patching. Even so, we will recommend the computed index approach in the section above to simplify your schema migrations.

It’s not shown here, but you have quite a bit of flexibility in configuring exactly what index type and applicability. See the documentation on duplicated fields for an example.

Gin Index

If you’re needing to issue a lot of variable adhoc queries against a Marten document, you may want to opt for a Gin index. A gin index against a Postgresql JSONB object creates a generalized index of key/value pairs and arrays within the parsed JSON document. To add a Gin index to a Marten document type, you need to explicitly configure that document type like this:

    var store = DocumentStore.For(_ =>
    {
        _.Schema.For<Issue>().GinIndexJsonData();
    });

You can also decorate your document class with the [GinIndexed] attribute. It’s not shown above, but there are options to customize the index generated.

When the DDL for the Issue document is generated, you would see a new index added to its table like this one:

CREATE INDEX mt_doc_issue_idx_data ON public.mt_doc_issue USING gin ("data" jsonb_path_ops);

Do note that using a Gin index against a document type will result in slightly slower inserts and updates to that table. From our testing, it’s not that big of a hit, but still something to be aware of.

Yet more content on new features in Marten leading up to the 1.0 release coming soon. These posts aren’t getting many reads, but they’ll be stuck in Google/Bing for later users, so expect a couple more of these.

As part of the 1.0 release work, Marten gained the capability last week (>0.9.8) to support the concept of “soft deletes.” Instead of just deleting a document completely out of the database, a soft delete would just mark a “deleted” column in the database table to denote that the row is now obsolete. The value of a “soft delete” is simply that you don’t lose any data from the historic record. The downsides are now you’ve got more rows cluttering up your database and you probably need to filter out deleted documents from your queries.

To see this in action, let’s first configure a document type in Marten as soft deleted:

    var store = DocumentStore.For(_ =>
    {
        _.Connection(ConnectionSource.ConnectionString);
        _.Schema.For<User>().SoftDeleted();
    });

By default, Marten does a hard delete of the row, so you’ll need to explicitly opt into soft deletes per document type. There is also a [SoftDeleted] attribute in Marten that you could use to decorate a document type to specify that it should be soft deleted.

When a document type is soft deleted, Marten adds a couple extra fields to the document storage table in the database called “mt_deleted” and “mt_deleted_at” just to track whether and when a document was deleted.

In usage, it’s transparent to the user that you’re doing a soft delete instead of a hard delete:

    // Create a new User document
    var user = new User();
    session.Store(user);
    session.SaveChanges();

    // Mark it deleted
    session.Delete(user);
    session.SaveChanges();

As I said earlier, one of your challenges is to filter out deleted documents in queries. Fortunately, Marten has you covered. As the following acceptance test from Marten shows, deleted documents are automatically filtered out of the Linq query results:

    [Fact]
    public void query_soft_deleted_docs()
    {
        var user1 = new User { UserName = "foo" };
        var user2 = new User { UserName = "bar" };
        var user3 = new User { UserName = "baz" };
        var user4 = new User { UserName = "jack" };

        using (var session = theStore.OpenSession())
        {
            session.Store(user1, user2, user3, user4);
            session.SaveChanges();

            // Deleting 'bar' and 'baz'
            session.DeleteWhere<User>(x => x.UserName.StartsWith("b"));
            session.SaveChanges();

            // no where clause, deleted docs should be filtered out
            session.Query<User>().OrderBy(x => x.UserName).Select(x => x.UserName)
                .ToList().ShouldHaveTheSameElementsAs("foo", "jack");

            var sql = session.Query<User>().OrderBy(x => x.UserName).Select(x => x.UserName).ToCommand().CommandText;
                _output.WriteLine(sql);

            // with a where clause
                session.Query<User>().Where(x => x.UserName != "jack")
                .ToList().Single().UserName.ShouldBe("foo");
        }
    }

Easy peasy. Of course you may want to query against the deleted documents or against all the documents. Marten’s got you covered there too, you can use these two custom Linq extensions in Marten to include deleted documents:

1. “IDocumentSession.Query<T>().MaybeDeleted()…” will include all documents in the query, regardless of whether or not they have been deleted

2. “IDocumentSession.Query<T>().IsDeleted()…” will only include documents marked as deleted

This is a logical follow up to my last post on Document Transformations in Marten with Javascript and yet another signpost on the way to Marten 1.0. Instead of having to write your own Javascript, Marten supplies the “Patch API” described here for very common scenarios.

Before I started working on Marten, I read an article comparing the performance of writing and querying JSON data between MongoDB and Postgresql (I couldn’t find the link when I was writing this post). Long story short, Postgresql very clearly comes out on top in terms of throughput, but the author still wanted to stick with MongoDB because of its ability to do document patching where you’re able to change elements within the persisted document without having to first load it into your application, change it, and persist the whole thing back. It’s a fair point and a realistic scenario that I used with RavenDb’s Patch Commands in the past.

Fortunately, that argument is a moot point because we have a working “Patch API” model in Marten for doing document patches. This feature does require PLV8 be enabled in your Postgresql database if you want to play with this feature in our latest nugets.

For an example, let’s say that you want to change the user name of a User document without first loading it. To update a single property or field in a document by its Id, it’s just this:

public void change_user_name(IDocumentSession session, Guid userId, string newName)
{
    session.Patch<User>(userId).Set(x => x.UserName, newName);
    session.SaveChanges();
}

When IDocumentSession.SaveChanges() (or its async equivalent) is called, it will send all the patching requests queued up with all of the pending document changes in a single database call.

I should also point out that the Set() mechanism can be used with nested properties or fields and non-primitive types.

Looking at another example, what if you just want to add a new role to an existing User? For that, Marten exposes the Append() method:

public void append_role(IDocumentSession session, Guid userId, string role)
{
    session.Patch<User>(userId).Append(x => x.Roles, role);
    session.SaveChanges();
}

In the case above, the new role will be appended to the “Roles” collection in the persisted JSON document in the Postgresql database. Again, this method can be used for nested or deep properties or fields and with non-primitive elements.

As a third example, let’s say that you only want to increment some kind of counter in the JSON document:

public void increment_login_count(IDocumentSession session, Guid userId)
{
    session.Patch<User>(userId).Increment(x => x.LoginCount);
    session.SaveChanges();
}

When the above command is issued, Marten will find the current numeric value in the JSON document, add 1 to it (the increment is an optional argument not shown here), and persist the new JSON data without ever fetching it into the client. The Increment() method can be used with int’s, long’s, double’s, and float’s.

Lastly, if you want to make a patch update to many documents by some kind of criteria, you can do that too:

public void append_role_to_internal_users(IDocumentSession session, Guid userId, string role)
{
    // Adds the role to all internal users
    session.Patch<User>(x => x.Internal)
        .Append(x => x.Roles, role);

    session.SaveChanges();
}

Other “Patch” mechanisms include the ability to rename a property or field within the JSON document and the ability to insert an item into a child collection at a given index. Other patch mechanisms are planned for later as well.

In all likelihood, Marten would garner much more rapid adoption if we were able to build on top of Sql Server 2016 instead of Postgresql. Hopefully, .Net folks will be willing to try switching databases after they see how many helpful capabilities that Postgresql has that Sql Server can’t match yet. This blog post, yet one more stop along the way to Marten 1.0, demonstrates how we’re taking advantage of Postgresql’s built in Javascript engine (PLV8). 

A Common .Net Approach to “Readside” Views

Let’s say that you’re building HTTP services, and some of your HTTP endpoints will need to return some sort of “readside” representation of your persisted domain model. For the purpose of making Marten shine, let’s say that you’re going to need to work with hierarchical data. In a common .Net technology stack, you’d:

1. Load the top level model object through Entity Framework or some other kind of ORM. EF would issue a convoluted SQL query with lots of OUTER JOIN’s so that it can make a single call to the database to fetch the entire hierarchy of data you need from various tables. EF would then proceed to iterate through the sparsely populated recordset coming back and turn that into the actual domain model object represented by the data with lots of internally generated code.
2. You’d then use something like AutoMapper to transform the domain model object into a “read side” Data Transfer Object (view models, etc.) that’s more suitable to going over the wire to clients outside of your service.
3. Serialize your DTO to a JSON string and write that out to the HTTP response

Depending on how deep your hierarchy is, #1 can be expensive in the database query. The serialization in #3 is also somewhat CPU intensive.

As a contrast, here’s an example of how you might approach that exact same use case with Marten:

    var json = session.Query<User>()
        .Where(x => x.Id == user.Id)
        .TransformToJson("get_fullname").Single();

In the usage above, I’m retrieving the data for a single User document from Marten and having Postgresql transform the persisted JSON data to the format I need for the client with a pre-loaded Javascript transformation. In the case of Marten, the workflow is to:

1. Find the entire hierarchical document JSON in a single database row by its primary key
2. Apply a Javascript function to transform the persisted JSON to the format that the client needs and return a JSON representation as a String
3. Stream the JSON from the Linq query directly to the HTTP response without any additional serialization work

Not to belabor the point too much, but the Marten mechanics are simpler and probably much more efficient at runtime because:

• The underlying database query is much simpler if all the data is in one field in one row
• The Javascript transformation probably isn’t that much faster or slower than the equivalent AutoMapper mechanics, so let’s call that a wash
• You don’t have the in memory allocations to load a rich model object just to immediately transform that into a completely different model object
• You avoid the now unnecessary cost of serializing the DTO view models to a JSON string

A couple additional points:

• Jimmy Bogard reviewed this and pointed out that in some cases you could bypass the Domain Model to DTO transformation by selecting straight to the DTO, but that wouldn’t cover all cases by any means. The same limitations apply to Marten and its Select() transformation features.
• To get even more efficient in your Marten usage, the Javascript transformations can be used inside of Marten’s Compiled Query feature to avoid the CPU cost of repetitively parsing Linq statements. You can also do Javascript transformations inside of batched queries – which can of course, also be combined with the aforementioned compiled queries;)

 

Now, let’s see how it all works…

 

Building the Javascript Function

The way this works in Marten is that you write your Javascript function into a single file and export the main function with the “module.exports = ” CommonJS syntax. Marten is expecting the main function to have the signature “function(doc)” and return the transformed document.

Here’s a sample Javascript function I used to test this feature that works against a User document type:

module.exports = function(doc) {
    return {fullname: doc.FirstName + ' ' + doc.LastName};
}

Given the persisted JSON for a User document, this transformation would return a different object that would then be streamed back to the client as a JSON string.

There is some thought and even infrastructure for doing Javascript transformations with multiple, related documents, but that feature won’t make it into Marten 1.0.

To load the function into a Javascript-enabled Postgresql schema, Marten exposes this method:

    var store = DocumentStore.For(_ =>
    {
        _.Connection(ConnectionSource.ConnectionString);

        // Let Marten derive the transform name
        // from the file name
        _.Transforms.LoadFile("get_fullname.js");

        // or override the transform name
        _.Transforms.LoadFile("get_fullname.js", "fullname");
    });

Internally, Marten will wrap a PLV8 function wrapper around your Javascript function like this:

CREATE OR REPLACE FUNCTION public.mt_transform_get_fullname(doc jsonb)
  RETURNS jsonb AS
$BODY$

  var module = {export: {}};

module.exports = function (doc) {
    return {fullname: doc.FirstName + ' ' + doc.LastName};
}

  var func = module.exports;

  return func(doc);

$BODY$
  LANGUAGE plv8 IMMUTABLE STRICT;

My intention with the approach shown above was to allow users to write simple Javascript functions and be able to test their transformations in simple test harnesses like Mocha. By having Marten wrap the raw Javascript in a generated PLV8 function, users won’t have to be down in the weeds and worrying about Postgresql mechanics.

Depending on the configuration, Marten is good enough to build, or rebuild, the function to match the current version of the Javascript code on the first usage of that transformation. The Javascript transforms are also part of our schema management support for database migrations.

 

Transformations

The persisted JSON documents in Marten are a reflection of your .Net classes. Great, that makes it absurdly easy to keep the database schema in synch with your application code at development time — especially compared to the typical development process against a relational database. However, what happens when you really do need to make breaking changes or additions to a document type but you already have loads of persisted documents in your Marten database with the old structure?

To that end, Marten allows you to use Javascript functions to alter the existing documents in the database. As an example, let’s go back to the User document type and assume for some crazy reason that we didn’t immediately issue a user name to some subset of users. As a default, we might just assign their user names by combining their first and last names like so:

module.exports = function (doc) {
    doc.UserName = (doc.FirstName + '.' + doc.LastName).toLowerCase();

    return doc;
}

To apply this transformation to existing rows in the database, Marten exposes this syntax:

    var store = DocumentStore.For(_ =>
    {
        _.Connection(ConnectionSource.ConnectionString);

        _.Transforms.LoadFile("default_username.js");
    });

    store.Transform
        .Where<User>("default_username", x => x.UserName == null);

When you run the code above, Marten will issue a single SQL statement that issues an UPDATE to the rows matching the given criteria by applying the Javascript function above to alter the existing document. No data is ever fetched or processed in the actual application tier.

Some blog posts you write for attention or self promotion, some you write just because you’re excited about the topic, and some posts you write just to try to stick some content for later into Google searches. This one’s all about users googling for this information down the road.

Out of the box, Marten uses Newtonsoft.Json as its primary JSON serialization mechanism. While Newtonsoft has outstanding customizability and the most flexible feature set, you can opt to forgo some of that flexibility in favor of higher performance by switching instead to the Jil serializer.

In the last couple months I finally made a big effort to be able to run Marten’s test suite using the Jil serializer. I had to make one small adjustment to our JilSerializer (turning on includeInherited), and a distressingly intrusive structural change to make the internal handling of Enum values (!) in Linq queries be dependent upon the internal serializer’s behavior for enum storage.

At this point, we’re not supplying a separate Marten.Jil adapter package, but the code to swap in Jil is just this class:

public class JilSerializer : ISerializer
{
    private readonly Options _options 
        = new Options(dateFormat: DateTimeFormat.ISO8601, includeInherited:true);

    public string ToJson(object document)
    {
        return JSON.Serialize(document, _options);
    }

    public T FromJson<T>(string json)
    {
        return JSON.Deserialize<T>(json, _options);
    }

    public T FromJson<T>(Stream stream)
    {
        return JSON.Deserialize<T>(new StreamReader(stream), _options);
    }

    public object FromJson(Type type, string json)
    {
        return JSON.Deserialize(json, type, _options);
    }

    public string ToCleanJson(object document)
    {
        return ToJson(document);
    }

    public EnumStorage EnumStorage => EnumStorage.AsString;
}

And this one line of code in your document store set up:

var store = DocumentStore.For(_ =>
{
    _.Connection("the connection string");

    // Replace the ISerializer w/ the JilSerializer
    _.Serializer<JilSerializer>();
});

A couple things to note about using Jil in place of Newtonsoft:

• The enumeration persistence behavior is different from Newtonsoft as it stores enum values by their string representation. Most Marten users seem to prefer this anyway, but watch the value of the “EnumStorage” property in your custom serializer.
• We’ve tried very hard with Marten to ensure that the Json stored in the database doesn’t require .Net type metadata, but the one thing we can’t address is having polymorphic child collections. For that particular use case, you’ll have to stick with Newtonsoft.Json and turn on its type metadata handling.

The Marten community has been making substantial progress toward a potential 1.0-alpha release early next week. As part of that effort, I’m going to be blogging about the new changes and features.

Recent versions of Marten (>0.9.5) have a new feature that allows you to enforce offline optimistic concurrency checks against documents that you are attempting to persist. You would use this feature if you’re concerned about a document in your current session having been modified by another session since you originally loaded the document.

I first learned about this concept from Martin Fowler’s PEAA book. From his definition, offline optimistic concurrency:

Prevents conflicts between concurrent business transactions by detecting a conflict and rolling back the transaction.

In Marten’s case, you have to explicitly opt into optimistic versioning for each document type. You can do that with either an attribute on your document type like so:

    [UseOptimisticConcurrency]
    public class CoffeeShop : Shop
    {
        // Guess where I'm at as I code this?
        public string Name { get; set; } = "Starbucks";
    }

Or by using Marten’s configuration API to do it programmatically:

    var store = DocumentStore.For(_ =>
    {
        _.Connection(ConnectionSource.ConnectionString);

        // Configure optimistic concurrency checks
        _.Schema.For<CoffeeShop>().UseOptimisticConcurrency(true);
    });

Once optimistic concurrency is turned on for the CoffeeShop document type, a session will now only be able to update a document if the document has been unchanged in the database since it was initially loaded.

To demonstrate the failure case, consider the following  acceptance test from Marten’s codebase:

[Fact]
public void update_with_stale_version_standard()
{
    var doc1 = new CoffeeShop();
    using (var session = theStore.OpenSession())
    {
        session.Store(doc1);
        session.SaveChanges();
    }

    var session1 = theStore.DirtyTrackedSession();
    var session2 = theStore.DirtyTrackedSession();

    var session1Copy = session1.Load<CoffeeShop>(doc1.Id);
    var session2Copy = session2.Load<CoffeeShop>(doc1.Id);

    try
    {
        session1Copy.Name = "Mozart's";
        session2Copy.Name = "Dominican Joe's";

        // Should go through just fine
        session2.SaveChanges();

        // When session1 tries to save its changes, Marten will detect
        // that the doc1 document has been modified and Marten will
        // throw an AggregateException
        var ex = Exception<AggregateException>.ShouldBeThrownBy(() =>
        {
            session1.SaveChanges();
        });

        // Marten will throw a ConcurrencyException for each document
        // that failed its concurrency check
        var concurrency = ex.InnerExceptions.OfType<ConcurrencyException>().Single();
        concurrency.Id.ShouldBe(doc1.Id);
        concurrency.Message.ShouldBe($"Optimistic concurrency check failed for {typeof(CoffeeShop).FullName} #{doc1.Id}");
    }
    finally
    {
        session1.Dispose();
        session2.Dispose();
    }

    // Just proving that the document was not overwritten
    using (var query = theStore.QuerySession())
    {
        query.Load<CoffeeShop>(doc1.Id).Name.ShouldBe("Dominican Joe's");
    }

}

Marten is throwing an AggregateException for the entire batch of changes being persisted from SaveChanges()/SaveChangesAsync() after rolling back the current database transaction. The individual ConcurrencyException’s inside of the aggregated exception expose information about the actual document type and identity that failed.