Author Archives: jeremydmiller

Marten 2.4.0 — now plays nicer with others

I was able to push a Marten 2.4.0 release and updated documentation yesterday with several bug fixes and some new small, but important features. The key additions are:

  1. Opening up a Marten session with a existing native Npgsql connection and/or transaction. When you do this, you can also direct Marten whether or not it “owns” the transaction lifecycle and whether or not Marten is responsible for committing or rolling back the transaction on IDocumentSession.SaveChanges(). Since this was a request from one of the Particular Software folks, I’m assuming you’ll shortly see Marten-backed saga persistence in NServiceBus soon;-)
  2. Opening a Marten session that enlists in the current TransactionScope. Do note that this feature is only available when either targeting the full .Net framework (> .Net 4.6) or Netstandard 2.0.
  3. Ejecting a document from a document session. Other folks have asked for that over time, but strangely enough, it got done quickly when I wanted it for something I was building. Weird how it works that way sometimes.
  4. Taking advantage of Marten’s schema management capabilities to register “feature schema objects” for additional database schema objects.

I don’t know the timing, but there were some new features that got left out because I got impatient to push this release, and we’ve had some recent feature requests that aren’t too crazy. Marten will return next in “2.5.0.”

 

 

 

 

 

 

 

Advertisements

Choosing Persistence Tooling, 2017 Edition

A couple years ago I wrote a long post title My Thoughts on Choosing and Using Persistence Tools about my personal decision tree on how I choose database and persistence tooling if left to my own devices.

This has been a common topic at work lately as our teams need to select persistence or database tooling for various projects in our new “microservice” strategy (i.e., don’t build more stuff into our already too large systems from here on out). As a lazy way of creating blog content, I thought I’d share the “guidance” that my team published a couple months back — even though it’s already been superseded a bit by the facts on the ground.

We’ve since decided to scale back our usage of Postgresql in new projects (for the record, this isn’t because of any purely technical reasons with Postgresql or Marten), but I think we’ve finally got some consensus to at least move away from a single centralized database in favor of application databases and improve our database build and test automation, so that’s a win. As for my recommendations on tooling selection, it looks like I’m having to relent on Entity Framework in our ecosystem due to developer preferences and familiarity.

 

Databases and Persistence

Most services will need to have some sort of persistent state, and that’s usually going to be in the form of a database. Even before considering which database engine or persistence approach you should take in your microservice, the first piece of advice is to favor application specific databases that are completely owned and only accessed by your microservice. 

The pattern of data access between services and databases is reflected by this diagram:

application-database

As much as possible, we want to avoid ever sharing a database between applications because of the implicit, tight coupling that creates between two or more services. If your service needs to query information that is owned by another service, favor using either exposing HTTP services from the upstream service or using the request/reply feature of our service bus tools.

Ideally, the application database for a microservice should be:

  • An integrated part of the microservice’s continuous integration and deployment pipeline. The microservice database should be built out by the normal build script for your microservice so that brand new developers can quickly be running the service from a fresh clone of the codebase.
  • Versioned together with any application code in the same source control repository to establish a strong link between the application code and database schema structure
  • Quick to tear down and rebuild from scratch or through incremental migrations

Database and Persistence Tooling Options

The following might be a lie. Like any bigger shop that’s been around for awhile, we’ve got some of everything (NHibernate, raw ADO.net, sproc’s, a forked copy of PetaPoco, etc.)

The two most common approaches in our applications is either:

  1.     Using Dapper as a micro-ORM to access Sql Server
  2.     Using Marten to treat Postgresql as a document database – slowly, but surely replacing our historical usages of RavenDb. 

Use Marten if your service data is complex or naturally hierarchical. If any of these things are true about your service’s persisted domain model, we recommend reaching for Marten:

  • Your domain model is hierarchical and a single logical entity used within the system would have to be stored across multiple tables in a relational database. Marten effectively bypasses the need to map a domain model to a relational database
  • There’s any opportunity to stream JSON data directly from the database to HTTP response streams to avoid the performance hit of using serializers, AutoMapper like tools, or ORM mapping layers
  • You expect your domain model to change rapidly in the future
  • You opt to use event sourcing to persist some kind of long running workflow

Choose Dapper + Sql Server if:

  • Your domain model is going to closely match the underlying database table structure, with simple CRUD-intensive systems being a good example.
  • Definitely use a more relational database approach if the application involves reporting functionality – but you can also do that with Marten/Postgresql
  • Your service will involve set-based logic that is more easily handled by relational database operations

If it feels like your service doesn’t fit into the decision tree above, opt for Sql Server as that has been our traditional standard.

Other choices may be appropriate on a case by case basis. Raw ADO.Net usage is not recommended from a productivity standpoint. Heavy, full featured ORM’s like Entity Framework or NHibernate are also not recommended by the architecture team. If you feel like EF would be advantageous for your domain model, then Marten might be an alternative with less friction.

The architecture team strongly discourages the usage of stored procedures in most circumstances.

For additional resources and conversation,

Jasper’s Getting Started Story – Take 1

IMG_1017

I’ve been kicking around the idea for a possible resurrection of FubuMVC as a mostly new framework with the codename “Jasper”  for several years with some of my colleagues. This year myself and several members of our architecture team at work have started making that a reality as the centerpiece of our longer term microservices strategy.

In the end, Jasper will be a lightweight service bus for asynchronous messaging, a high performance alternative to MVC for HTTP API’s, and a substitute for MediatR inside of ASP.Net Core applications (those three usages share much more infrastructure code than you might imagine and the whole thing is still going to be much, much smaller than FubuMVC was at the end). For the moment, we’re almost entirely focused on the messaging functionality.

I haven’t kicked out an up to date Nuget yet, but there’s quite a bit of documentation and I’m just hoping to get some feedback out of that right now. If you’re at all interested in Jasper, feel free to raise GitHub issues or join our Gitter room.

The only thing I’m trying to accomplish in this post is to get a sanity check from other folks on whether or not the bootstrapping looks usable.

Getting Started

This is taken directly from the getting started documentation.

Note! Jasper only targets Netstandard 1.5 and higher at this time, and we’ve been holding off on upgrading to ASP.Net Core v2.0.

Jasper is a framework for building server side services in .Net. Jasper can be used as an alternative web framework for .Net, a service bus for messaging, as a “mediator” type pipeline within a different framework, or any combination thereof. Jasper can be used as either your main application framework that handles all the configuration and bootstrapping, or as an add on to ASP.Net Core applications.

To create a new Jasper application, start by building a new console application:

dotnet new console -n MyApp

While this isn’t expressly necessary, you probably want to create a new JasperRegistry that will define the active options and configuration for your application:

public class MyAppRegistry : JasperRegistry
{
    public MyAppRegistry()
    {
        // Configure or select options in this constructor function
    }
}

See Configuring Jasper Applications for more information about using the JasperRegistry class.

Now, to bootstrap your application, add the Jasper.CommandLine library to your project and this code to the entrypoint of your console application:


using Jasper.CommandLine;

namespace MyApp
{
    class Program
    {
        static int Main(string[] args)
        {
            // This bootstraps and runs the Jasper
            // application as defined by MyAppRegistry
            // until the executable is stopped
            return JasperAgent.Run<MyAppRegistry>(args);
        }
    }
}

By itself, this doesn’t really do much, so let’s add Kestrel as a web server for serving HTTP services and start listening for messages from other applications using Jasper’s built in, lightweight transport:

public class MyAppRegistry : JasperRegistry
{
    public MyAppRegistry()
    {
        Http.UseKestrel().UseUrls("http://localhost:3001");
        Transports.Lightweight.ListenOnPort(2222);
    }
}

Now, when you run the console application you should see output like this:

Hosting environment: Production
Content root path: /Users/jeremill/code/jasper/src/MyApp/bin/Debug/netcoreapp1.1
Listening for messages at loopback://delayed/
Listening for messages at jasper://localhost:2333/replies
Listening for messages at jasper://localhost:2222/incoming
Now listening on: http://localhost:3001
Application started. Press Ctrl+C to shut down.

See Bootstrapping for more information about idiomatic Jasper bootstrapping.

That covers bootstrapping Jasper by itself, but next let’s see how you can add Jasper to an idiomatic ASP.Net Core application.

Adding Jasper to an ASP.Net Core Application

If you prefer to use typical ASP.Net Core bootstrapping or want to add Jasper messaging support to an existing project, you can use the UseJasper() extension method on ASP.Net Core’s IWebHostBuilder as shown below:

var host = new WebHostBuilder()
    .UseKestrel()
    .UseJasper<ServiceBusApp>()
    .Build();

host.Run();

See Adding Jasper to an ASP.Net Core Application for more information about configuring Jasper through ASP.Net Core hosting.

Your First HTTP Endpoint

The obligatory “Hello World” http endpoint is just this:

public class HomeEndpoint
{
    public string Get()
    {
        return "Hello, world.";
    }
}

As long as that class is in the same assembly as your JasperRegistry class, Jasper will find it and make the “Get” method handle the root url of your application.

See HTTP Services for more information about Jasper’s HTTP handling features.

Your First Message Handler

Let’s say you’re building an invoicing application and your application should handle an InvoiceCreated event. The skeleton for the message handler for that event would look like this:

public class InvoiceCreated
{
    public Guid InvoiceId { get; set; }
}

public class InvoiceHandler
{
    public void Handle(InvoiceCreated created)
    {
        // do something here with the created variable...
    }
}

See Message Handlers for more information on message handler actions.

 

 

Retrospective on Marten at 2 Years Old

I made the very first commit to Marten two years ago this week. Looking at the statistics, it’s gotten just shy of 2,000 commits since then from almost 60 contributors. It’s not setting any kind of world records for usage, but it’s averaging a healthy (for a .Net OSS project) 100+ downloads a day.

Marten was de facto sponsored by my shop because we intended all along to use it as a way to replace RavenDb in our ecosystem with Postgresql. Doing Marten out in the open as an open source project hosted in GitHub has turned out to be hugely advantageous because we’ve had input, contributions, and outright user testing from so many external folks before we even managed to put Marten into our biggest projects. Arguably — and this frustrates me more than a little bit — Marten has been far more successful in other shops that in my own.

I’ve been very pleasantly surprised by how the Marten community came together and how much positive contribution we’ve gotten on new features, documentation, and answering user questions in our Gitter room. At this point, I don’t feel like Marten is just my project anymore and that we’ve genuinely got a healthy group of contributors and folks answering user questions (which is contributing greatly to my mental health).

Early adopters are usually the best users to deal with because they’re more understanding and patient than the folks that come much later when and if your tool succeeds. There’s been a trend that I absolutely love in Marten where we’ve been able to collect a lot of bug reports as a pull request with failing tests that show you exactly what’s wrong. For a project that’s so vulnerable to permutation problems, that’s been a life send. Moreover, we’ve had enough users using it in lots of different things that’s led to the discovery and resolution of a lot of functionality and usability problems.

I’m a little bit disappointed by the uptake in Marten usage, because I think it’s hugely advantageous for developer productivity over ORM’s like Entity Framework and definitely more productive in many problem domains than using a relational database straight up. I don’t know if that’s mostly because the .Net community just isn’t very accepting of tools like this that are outside of the mainstream, we haven’t been able to break through in terms of promoting it, or if it just isn’t that compelling to the average .Net developer. I strongly suspect that Marten would be far more successful if it had been built on top of Sql Server, and we might test that theory if Sql Server ever catches up to Postgresql in terms of JSON and Javascript support (it’s not even close yet).

For some specific things:

  • Postgresql is great for developers just out of the sheer ease of installing it in developer or testing environments
  • I thought going into Marten that the Linq support would be the most problematic thing. After working on the Linq support for quite awhile, I now think that the Linq support is the most problematic and time consuming thing to work on and it’s likely that folks will never stop coming up with new usage scenarios
  • The Linq support would be so much easier and probably more performant when Postgresql gets their proposed JsonPath querying feature. Again, I don’t think that Sql Server’s JSON support is adequate to support Marten’s feature set, but they at least went for JsonPath in their Json querying.
  • A lot of other people contributed here too, but Marten has been a great learning experience on asynchronous code that’s helping me out quite a bit in other projects
  • The event sourcing feature has been a mixed bag for me. My shop hasn’t ended up adopting it, so I’m not dogfooding that work at all — but guess what seems to be the most popular part of Marten to the outside world? The event sourcing support wouldn’t be viable if we didn’t have so much constructive feedback and help from other people.
  • I think it was advantageous to have the documentation done very early and constantly updated as we went
  • After my FubuMVC flop, I swore that if I tried to do another big OSS project that I’d try much harder to build community, document it early, and promote it more effectively. To that end, you can see or hear more about Marten on DotNetRocks, the NoSQL podcast, the Cross Cutting Concerns podcast, a video on Channel 9Herding Code, a recent conversation on Hanselminutes, and a slew of blog posts as we went.

Let my close by thanking the Marten community. I might fight burnout occasionally or get grumpy about the internal politics around Marten at work, but y’all have been fantastic to interact with and I really enjoy the Marten community.

Introducing Oakton — Command line parsing minus the usual cruft

As the cool OSS kids would say, “I made another thing.” Oakton is a library that I maintain that I use for command line parsing in the console applications I build and maintain. For those who’ve followed me for a long time, Oakton is an improved version of the command line parsing in FubuCore that now targets Netstandard 1.3 as well as .Net 4.5.1 and 4.6 on the full framework.

What sets Oakton apart from the couple dozen other tools like this in the .Net ecosystem is how it allows you to cleanly separate the command line parsing from your actual command parsing so that you can write cleaner code and more easily test your command execution in automated tests.

Here’s the quick start example from the documentation that’ll have prettier code output. Let’s say you just want a command that will print out a name with an optional title and the option to override the color of the text.

A command in Oakton comes in two parts, a concrete input class that just establishes the required arguments and optional flags through public fields or settable properties:

    public class NameInput
    {
        [Description("The name to be printed to the console output")]
        public string Name { get; set; }
        
        [Description("The color of the text. Default is black")]
        public ConsoleColor Color { get; set; } = ConsoleColor.Black;
        
        [Description("Optional title preceeding the name")]
        public string TitleFlag { get; set; }
    }

The [Description] attributes are optional and embed usage messages for the integrated help output.

Now then, the actual command would look like this:

    [Description("Print somebody's name")]
    public class NameCommand : OaktonCommand
    {
        public NameCommand()
        {
            // The usage pattern definition here is completely
            // optional
            Usage("Default Color").Arguments(x => x.Name);
            Usage("Print name with specified color").Arguments(x => x.Name, x => x.Color);
        }

        public override bool Execute(NameInput input)
        {
            var text = input.Name;
            if (!string.IsNullOrEmpty(input.TitleFlag))
            {
                text = input.TitleFlag + " " + text;
            }
            
            // This is a little helper in Oakton for getting
            // cute with colors in the console output
            ConsoleWriter.Write(input.Color, text);


            // Just telling the OS that the command
            // finished up okay
            return true;
        }
    }

Again, the [Description] attributes and the Usage property in the constructor function are all optional, but add more information to the user help display. You’ll note that your command is completely decoupled from any and all text parsing and does nothing but do work against the single input argument. That’s done very intentionally and we believe that this sets Oakton apart from most other command line parsing tools in .Net that too freely commingle parsing with the actual functionality.

Finally, you need to execute the command in the application’s main function:

    class Program
    {
        static int Main(string[] args)
        {
            // As long as this doesn't blow up, we're good to go
            return CommandExecutor.ExecuteCommand&lt;NameCommand&gt;(args);
        }
    }

Oakton is fairly full-featured, so you have the options to:

  1. Expose help information in your tool
  2. Support all the commonly used primitive types like strings, numbers, dates, and booleans
  3. Use idiomatic Unix style naming and usage conventions for optional flags
  4. Support multiple commands in a single tool with different arguments and flags (because the original tooling was too inspired by the git command line)

 

So a couple questions:

  • Does the .Net world really need a new library for command line parsing? Nope, there’s dozens out there and a semi-official one somewhere inside of ASP.Net Core. It’s no big deal on my part though because other than the docs I finally wrote up this week, this code is years old and “done.”
  • Where’s the code? The GitHub repo is here.
  • Is it documented, because you used to be terrible at that? Yep, the docs are at http://jasperfx.github.io/oakton.
  • If I really want to use this, where can I ask questions? You can always use GitHub issues, or try the Gitter room.
  • Are there any real world examples of this actually being used? Yep, try Marten.CommandLine, the dotnet-stdocs tool, and Jasper.CommandLine.
  • What’s the license? Apache v2.

Where does the name “Oakton” come from?

A complete lack of creativity on my part. Oakton is a bustling non-incorporated area not far from my grandparent’s farm on the back way to Lamar, MO that consists of a Methodist church, a cemetery, the crumbling ruins of the general store, and maybe 3-4 farmhouses. Fun fact, when I was really small, I tagged along with my grandfather when he’d take tractor parts to get fixed by the blacksmith that used to be there.

Proposal for StructureMap 5

EDIT 9/12: Meh, I had a couple twitter exchanges and Gitter questions today that reminded me why I’ve been wanting to walk away from StructureMap. Having to support not just SM but really how SM is used internally inside of tools like ASP.Net MVC Core, MediatR, and half a hundred other frameworks is just wearing me down too much. If anyone else is interested in taking on any of this, I’ll happily help out, but otherwise I think I’m just going to leave this alone. Besides, there’s the new built in IoC in ASP.Net and 30 or so other OSS competitors.

 

So I’ve been more or less burned out on StructureMap development and support for quite some time (it’s been better lately though). That being said, there’s a ton of people using it (it averages just shy of a 1,000 downloads a day).  It has also been put through the ringer from a lot of users, which remarkably enough, exposes and leads to fixing a lot of bugs and usability problems — and if you don’t believe me, check out this folder of all the tests for bug regressions and fixes.

StructureMap 4.* has a couple ongoing issues that should get addressed some day if the project is going to keep going on:

  1. StructureMap has fallen behind many or most of the other IoC containers in the public performance benchmarks. I think those benchmarks are mostly over simplified BS, but still, there’s the pride factor
  2. ASP.Net Core DI compliance has been a huge pain in the ass to the point where I’ve openly wondered if the ASP.Net team has purposely tried to sabotage and wipe out all the existing ecosystem of IoC containers in .Net
  3. The child container behavior (which I don’t personally use) has been problematic as StructureMap’s more creative users have found several permutations of this and that where the child container model has broken down a bit

So, here’s my thoughts about a possible direction for a future StructureMap 5.0:

  • Keep API backwards almost completely backward compatible with StructureMap 4.* except for a few places impacted by the next bullet point
  • Completely redesign the internal data structures as a performance optimization. The current structure isn’t terribly different from the very earliest StructureMap versions and there’s absolutely room to cut out some performance fat there
  • Take a dependency on Microsoft.Extensions.DependencyInjection.Abstractions and merge in the functionality that today is in StructureMap.Microsoft.DependencyInjection so that it’s easier to get the compliance against ASP.Net Core right by having everything in one place. My thought here too is that we would somehow use their configuration abstractions, but supplemented with the existing StructureMap configuration options somehow as a kind of buddy class extension. Not sure how that one’s gonna work out yet.
  • Look for opportunities to make the dynamic Expressions that are built up to actually create objects be more efficient by inlining some operations and generally reducing the number of times it bounces through dictionary structures. I know a lot more about building dynamic Expressions that I did several years ago when I moved StructureMap off of IL generation, so surely there’s some opportunity there

Alright, so my personal conundrum is simply wondering do I care enough to do this as an exercise in crafting performant data structures and micro-optimization, or call StructureMap 4.5 the end of the road and just continue to try to address bugs and user questions for the time being.

I’d kind of like to hear from some StructureMap users or contributors to see how much they’d want the performance and ASP.Net Core compatibility, and then see if anyone would want to help out if we go with this.

 

How we did (and did not) improve performance and efficiency in Marten 2.0

Marten 2.0 was released yesterday, and one of the improvements is somewhat significantly runtime performance and far better memory utilization in applications that use Marten. For today’s blog post, here’s what we did and tried to get there:

  • Avoiding Json strings whenever possible. Some time last year Ayende wrote a “review” of Marten on his blog before almost immediately retracting it. While I didn’t agree with most of his criticisms, he did call out Marten for being inefficient in its Json serialization by reading and writing the full Json strings instead of opting for more efficient mechanisms of reading or writing via byte arrays or Stream’s. The “write” side of this problem was largely solved in Marten 2.0, but after some related changes in the underlying Npgsql library, the “read” side of Marten uses TextReader’s as the input to Json serialization, therefore bypassing the need to create then immediately tear down string objects. These changes reduced the memory allocations in Marten almost by half, with maybe a 15-20% improvement in performance.
  • StringBuilder for all SQL command build up. I know what you’re thinking, “duh, StringBuilder is way more efficient than string concatenation,” but Marten got off the ground by mostly using string interpolation and concatenation. For 2.0, I went back over all that code and switched to StringBuilder’s, which has the nice impact of reducing memory utilization quite a bit (it didn’t make that much difference in performance). I absolutely don’t regret starting with simpler, cruder mechanisms to get things working before pulling in this optimization.
  • FastExpressionCompiler – Marten heavily uses dynamically generated Expression’s that are then compiled to Func or Action’s for document persistence and loading. The excellent FastExpressionCompiler library from Maksim Volkau replaces the built in Expression compilation with a new model that results both in delegates that are faster in runtime, and also reduces the compilation time of these expressions. Using FastExpressionCompiler makes Marten bootstrap faster, which made a huge improvement in Marten’s test suite execution. I measured about a 10% throughput performance in Marten’s benchmarks just by using this library
  • Newtonsoft.Json 9 to 10 and back to 9 – Newtonsoft.Json 10 was measurably slower in the Marten benchmarks, so we reverted back to 9.0.1. Bummer. You can always opt for Jil or other alternatives for considerably faster json serialization, but we found too many cases where Jil errored out on document types that Newtonsoft.Json handled just fine, so we stuck with Newtonsoft as the default based on the idea that the code should at least work;)

 

What’s left to do for performance?

  • I’m sure we could get better with our mechanics for byte[] or char[] pooling and probably some buffering in the ADO.Net manipulation during async methods
  • We know there are some places where the Linq provider generates Sql that isn’t as efficient as it could be. We might try to tackle this tactically in use case by use case, but I’m hoping for the version of Postgresql after 10 to get their improved Json querying functionality based on JsonPath before we do anything big to the Linq support.