Lamar v3 is Released: Faster, smaller, quicker cold starts, internal type friendly

Lamar has had some churn lately with me almost giving up on it, changing my mind, and look at that, publishing a very significantly improved version up to Nuget just now.

I just published Lamar v3.0 with some bug fixes and big internal improvements. This release is mostly significant because it eliminates Lamar’s previous reliance on Roslyn for runtime code generation in favor of a more common model that builds Expression trees and compiles dynamic lambdas — inevitably with the excellent FastExpressionCompiler for maximum performance and faster cold starts.

All told, Lamar v3.0:

  • Has much faster “cold start” times compared to ❤
  • Is faster across the board, but especially when internal types are involved. The very early user feedback is “holy moses that’s much faster”
  • I think that we’re through with compatibility with internal types being registered into the container because the dynamic Lambdas don’t have to care about the type visibility
  • Will use way less memory without the Roslyn dependency tree in memory
  • Has far fewer dependencies now that some users complained about

There are no API changes to Lamar itself, but there’s now a different set of Nuget libraries:

  1. LamarCompiler — only has Lamar’s old helper for executing the Roslyn runtime compilation, but this is no longer referenced by Lamar itself
  2. LamarCodeGeneration — the Frame / Variable model that Lamar uses to generate code and now Expression trees in memory. This is used by Lamar itself, could be used as a stand alone library, and has no direct coupling to LamarCompiler
  3. Lamar itself

A touch of history…

I’ve been absolutely exhausted for years with maintaining StructureMap and dealing with user questions and oddball problems. I also knew it had some pretty ugly performance issues and it had been a massive pain in the ass force fitting StructureMap’s long standing functionality into ASP.Net Core’s new compliance requirements (that appear to have been made up by the ASP.Net team out of whole cloth with ZERO input from the major OSS IoC tool authors). I also knew from some experimentation that I wasn’t going to be able to fix StructureMap very easily, so…

Lamar was an attempt by me to throw together a new IoC tool that would mostly mimic StructureMap‘s public API (everybody has opinions, but there are actually quite a few people who like StructureMap’s usability and it’s had ~6 million downloads) and a large subset of its functionality. At the same time, I did change its lifecycle semantics and behavior to be ASP.Net Core-compliant from the very get go to head off more problems with that.

Lamar’s original model was based around using dynamic code generation and compilation with Roslyn. Woohoo, it was easy to use and I was able to get a large subset of StructureMap’s old functionality working over Christmas break a couple years ago. Unfortunately, that model doesn’t play well at all with internal types and my limited workarounds were pretty clumsy. It turns out that lots and lots of people and common frameworks like to put internal types into IoC containers (looking at you ASP.Net Core). So here we are. Lamar switched over to using Expressions compiled into Funcs like basically every other IoC container I’m aware of, but hopefully it starts fulfilling my real goal for Lamar:

Get folks complaining about StructureMap off of that and make my life easier overall by reducing user problems.

Advertisements

Jasper v0.9.9 is Released!

Jasper is an open source project I’ve been furiously working on for the past couple years as what I have to admit is mostly a chance to resurrect the best parts of the earlier FubuMVC framework while solving its technical and usability shortcomings. Along the way, Jasper has evolved a bit away from its FubuMVC roots to be more consistent and compatible with ASP.Net Core. At this point, Jasper is a fancy command execution pipeline that lives inside of the ASP.Net Core ecosystem. Jasper can be used as any mix of a lightweight messaging framework, an in-memory service bus, or handling HTTP requests as an alternative ASP.Net Core framework.

I was just able to push a new v0.9.9 release of Jasper that I want to effectively be the last alpha release. At this point, I think the public API surface is pretty well set and only a handful of features left before making the big ol’ 1.0 release. I think the most important thing for Jasper is to try to get folks to take it for a spin, or glance through tutorials, and generally try to get some feedback and visibility about the project.

Here’s some links to get you started:

There’ll hopefully be plenty of blog posts on Jasper in the next couple weeks, starting with how Jasper’s usability contrasts with MVC Core in the HTTP space or NServiceBus/MediatR/etc. for messaging and command execution.

The Road to 1.0

I’ve got to put Jasper down for awhile to focus on some Lamar and help out a lot more with Marten for probably a couple months before I do much more on Jasper, but I’d still love to get 1.0 out by at least the end of the summer.

I’m thinking out loud in this section, so everything is subject to change. Before I flip the switch to the big, giant 1.0, I think these things might need to happen:

  • There are some optimizations I want to make to the database backed message persistence I couldn’t quite get to for v0.9.9.
  • I’m very tempted just to wait until the netcoreapp3.0 wave of updates comes out. It’s a near guarantee that ASP.Net Core v3.0 will break some of Jasper’s internals, and it’d be very helpful to slim Jasper’s package dependency tree down if Jasper could depend on the new generic host model instead of IWebHostBuilder when 3.0 unifies that model somewhat.
  • Jasper heavily depends on the Task Parallel Library from Microsoft, but that seems to be somewhat deprecated now. I might look to rewire Jasper’s internals to use the newer System.Threading.Channels instead. I haven’t done any research into this one yet.
  • If the HTTP support is going to live on, it’ll need some kind of Swagger/Swashbuckle integration for the Jasper HTTP API support
  • I’d like to spend some time on supporting the Techempower Benchmarks for Jasper and some performance optimization. My goal here is to make Jasper the fastest HTTP application framework for .Net Core and be just barely slower than the raw ASP.Net Core benchmarks (not sure how feasible that is, but let me dream on).
  • Jasper is going to have to adjust to whatever becomes of Lamar. I don’t think this is going to change Jasper at development time, but might introduce a new production build step to optimize Jasper application’s “cold start” times for better hosting in Docker kind of worlds.

Lamar stays and how that enfolds

I wrote a blog post at the end of last week about Lamar when I just happened to be feeling discouraged about a couple things (it happens sometimes, and I swear that having to support IoC tools exposes me to more difficult people than every other project I work on combined). I got some rest this weekend, a bit of positive reinforcement from other folks, and actually thought through how to fix the issues. Long story short, I’m not giving up on anything and here’s what I think the very doable game plan is for Lamar (and the closely related Jasper project):

Lamar

  • Short term: Get a small bug fix release out soon that has some options to “force” all the compilation upfront in one dynamic assembly. That’s gonna hurt the cold start time, but should help the memory usage. We’ll also look to see if there’s any places where Lamar is holding on unnecessarily to the Roslyn compilation objects to ensure that they can be garbage collected
  • Medium term: Introduce an alternative compilation model based on Expressions compiled to Lambdas with FastExpressionCompiler. This model will kick in automatically whenever there’s one or more internal types in the “build plan”, and could be opted into globally through a container level switch. I didn’t want to do this originally because the model just isn’t very fun to work with.  After thinking it through quite a bit over the weekend, I think it won’t be bad at all to retrofit this alternative to Lamar’s existing Frame and Variable model. This will knock out the performance issues with internal types and address all the memory issues.
  • Long term: Probably split up LamarCompiler a little bit to remove the actual code compilation to significantly slim down Lamar’s dependency tree and move Lamar to a purely Expression based model. Introducing the Expression model will inevitably make the exception stacktrace messages coming out of Lamar explode, so there might have to be an effort to rewrite them to make them more user friendly (I had to do this in StructureMap 3 several years ago).

 

Jasper

I’m very close to pulling the trigger on a Jasper v1.0, with the understanding that it’s inevitable that there will be a Jasper v2.0 later in the year to incorporate .Net Core 3.0. I don’t think that Jasper will have any issues with memory usage related to Lamar because it uses Lamar very differently than MVC in any flavor. The changes to Lamar will impact Jasper though, so:

Short term: Jasper v1.0 with Lamar as it is.

Medium term: Jasper gets a model where you can happily use the runtime codegen and compilation during development time while things are churning, but for production usage you have the ability to just drop the code that would be generated to disk, have that compiled into your system in the first place, and let Jasper use those types. Ultra fast production time cold start times, no worries at all about Roslyn doing bad things from memory. I’ve already done successful proof of concept development on this one.

Long term: profit.

As an aside, I got quizzed quite a bit about why Jasper has to be specific to Lamar as its IoC container and can’t just support whatever tool folks want to use. The reason is that Jasper uses Lamar’s very specific code generation in its pipeline to avoid using an IoC container at runtime whatsoever and also to avoid forcing users to have to conform to all kinds of Jasper specific adapter interfaces. I could maybe force Jasper to still pull this off with the built in DI container or another IoC container with Jasper-centric adapters to expose all of its metadata in a way such that the codegen understands it, but just ick.

If you took Lamar’s runtime codegen away, I think Jasper inevitably looks like a near clone of NServiceBus or Brighter both in its usability and runtime pipeline and why does the world need that?

 

 

Lamar asks: Do I stay or do I go?

I happened to look into the Lamar issue list on GitHub late this afternoon at the end of a very long work day. What I saw was nothing to feel good about. A huge memory usage problem (probably related to Roslyn but I don’t know that for sure), a user who’s been very difficult in the past telling me he switched to another tool because it was easier to use and faster, bugs that unfortunately point to a huge flaw in Lamar’s basic premise, and folks really wanting a feature I’ve put off because it’s going to be a lot of work.

Lamar started its life as a subsystem of Jasper, the project that’s been my main side-project for the past couple years and my biggest ambition for several years before that. Jasper’s special sauce is the way that it uses Lamar to generate and compile code at runtime to glue together the runtime pipeline in the most efficient way possible. At the same time, it was killing me to support StructureMap and I wanted out of it, but didn’t want to leave a huge mess of folks with an abandoned project, so I turned Lamar into an IoC tool that was purposely meant to be a smooth transition for StructureMap users to an improved and more efficient IoC tool.

Right now, it’s becoming clear that the basic model of Lamar to use the runtime code generation isn’t working out. First because of what looks like some severe memory issues related to the runtime compilation, and second because it turns out that it’s very, very common for folks to use internal .Net types in IoC containers which pretty well defeats the entire code generation model. Lamar has workarounds for this to fall back to dynamic expressions for internal types, but it’s not optimized for that and Lamar’s performance degrades badly when folks use internal types.

As I see it, my choices with Lamar are to:

  • Retool it to use dynamic Expression compilation all the way down in the IoC engine usage like basically every other IoC tool including the older StructureMap library. That takes care of the performance issues most likely and knocks out the bugs due to internal types, but just sounds miserable.
  • Deal with the memory problems by trying to engage with the actual Roslyn team to figure out what’s going on and work around it. Which doesn’t solve the internal type issue.
  • Use pre-generated code build by Lamar during testing time baked into the real application for production deployments. There’s a bit more than a proof of concept already baked in, but this still doesn’t fix the internal problem

On the Jasper side I could:

  • Try to make it work with the built in IoC container but still try to do its “no IoC at runtime because it’s all codegen’d” model with a cutdown version of Lamar

Or, just call the whole thing off because while I would argue that Jasper shows a lot of promise technically and I still believe in its potential, it has zero adoption and I most likely won’t get to use it myself in any projects in the next year. If I give up on Jasper, I’m likely to do the same to Lamar. In that case, I’ll write off Lamar as a failed project concept, apologize profusely to all the folks that started to use it to replace StructureMap, and walk away.

Nobody needs to feel sorry for me here, so no need to try to cheer me up about any of this, but, I wouldn’t mind some feedback on whether it’s worth keeping Lamar around and what you think about the proposed improvements.

My integration testing challenges this week

EDIT 3/12: All these tests are fixed, and it wasn’t as bad as I’d thought it was going to be, but the extra logging and faster change code/run test under debugger cycle definitely helped.

This was meant to be a short post, or at least an easy to write post on my part, but it spilled out from integration testing and into some Storyteller mechanics, semi-advanced xUnit.Net usage, ASP.Net Core logging integration, and even a tepid defense of compositional architectures wrapped around an IoC container.

I’ve been working feverishly the past couple of months to push Jasper to a 1.0 release. As (knock on wood) the last big epic, I’ve been working on a large overhaul and redesign of the message persistence behind Jasper’s durable messaging. The existing code was fairly well covered by integration tests, so I felt confident that I could make the large scale changes and use the big bang integration tests to ensure the intended functionality still worked as designed.

I assume that y’all can guess how this has turned out. After a week of code changes and fixing any and all unit test and intermediate integration test failures, I got to the point where I was ready to run the big bang integration tests and, get this, they didn’t pass on the first attempt! I know, shocking right? Moreover, the tests involve all kinds of background processing and even multiple logical applications (think ASP.Net Core IWebHost objects) being started up and shut down during the tests, so it wasn’t exactly easy to spot the source of my test failures.

I thought it’d be worth talking about how I first stopped and invested in improving my test harness to make it faster to launch the test harness and to capture a lot more information about what’s going on in all the multithreading/asynchronous madness going on but…

First, an aside on Debugger Hell

You probably want to have some kind of debugger in your IDE of choice. You also want to be reasonably good using that tool, know how to use its features, and conversant in its supported keyboard shortcuts. You also want to try really hard to avoid needing to use your debugger too often because that’s just not an efficient way to get things done. Many folks in the early days of Agile development, including me, described debugger usage as a code or testing “smell.” And while “smell” is mostly used today as a pejorative to put down something that you just don’t like, it was originally just meant as a warning you should pay more attention to in your code or approach.

In the case of debugger usage, it might be telling you that your testing approach needs to be more fine grained or that you are missing crucial test coverage at lower levels. In my case, I’ll be looking for places where I’m missing smaller tests on elements of the bigger system and fill in those gaps before getting too worked up trying to solve the big integration test failures.

Storyteller Test Harness

For these big integration tests, I’m using Storyteller as my testing tool (think Cucumber,  but much more optimized for integration testing as opposed to being cute). With Storyteller, I’ve created a specification language that lets me script out message failover scenarios like the one shown below (which is currently failing as I write this):

JasperFailoverSpec

In the specification above, I’m starting and stopping Jasper applications to prove out Jasper’s ability to fail over and recover pending work from one running node to another using its Marten message persistence option. At runtime, Jasper has a background “durability agent” constantly running that is polling the database to determine if there is any unclaimed work to do or known application nodes are down (using advisory locks through Postgresql if anybody would ever be interested in a blog post about just that). Hopefully that’s enough description just to know that this isn’t particularly an easy scenario to test or code.

In my initial attempts to diagnose the failing Storyteller tests I bumped into a couple problems and sources of friction:

  1. It was slow and awkward mechanically to get the tests running under the debugger (that’s been a long standing problem with Storyteller and I’m finally happy with the approach shown later in this post)
  2. I could tell quickly that exceptions were being thrown and logged in the background processing, but I wasn’t capturing that log output in any kind of usable way

Since I’m pretty sure that the tests weren’t going to get resolved quickly and that I’d probably want to write even more of these damn tests, I believed that I first needed to invest in better visibility into what was happening inside the code and a much quicker way to cycle into the debugger. To that end, I took a little detour and worked on some Storyteller improvements that I’ve been meaning to do for quite a while.

Incorporating Logging into Storyteller Results

Jasper uses the ASP.Net Core logging abstractions for its own internal logging, but I didn’t have anything configured except for Debug and Console tracing to capture the logs being generated at runtime. Even with the console output, what I really wanted was all the log information correlated with both the individual test execution and which receiver or sender application the logging was from.

Fortunately, Storyteller has an extensibility model to capture custom logging and instrumentation directly into its test results. It turned out to be very simple to whip together an adapter for ASP.Net Core logging that captured the logging information in a way that can be exposed by Storyteller.

You can see the results in the image below. The table below is just showing all the logging messages received by ILogger within the “Receiver1” application during one test execution. The yellow row is an exception that was logged during the execution that I might not have been able to sense otherwise.

AspNetLoggingInStoryteller

For the implementation, ASP.Net Core exposes the ILoggerProvider service such that you can happily plug in as many logging strategies as you want to an application in a combinatorial way. On the Storyteller side of things, you have the Report interface that let’s you plug in custom logging that can expose HTML output into Storyteller’s results.

Implementing that crudely came out as a single class that implements both adapter interface (here’s a gist of the whole thing):

public class StorytellerAspNetCoreLogger : Report, ILoggerProvider

The actual logging just tracks all the calls to ILogger.Log() as little model objects in memory:

public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
{
    var logRecord = new LogRecord
    {
        Category = _categoryName,
        Level = logLevel.ToString(),
        Message = formatter(state, exception),
        ExceptionText = exception?.ToString()
    };

    // Just keep all the log records in an in memory list
    _parent.Records.Add(logRecord);
}

Fortunately enough, in the Storyteller Fixture code for the test harness I bootstrap the receiver and sender applications per test execution, so it’s really easy to just add the new StorytellerAspNetCoreLogger to both the Jasper applications and the Storyteller test engine:

var registry = new ReceiverApp();
registry.Services.AddSingleton<IMessageLogger>(_messageLogger);

var logger = new StorytellerAspNetCoreLogger(key);

// Tell Storyteller about the new logger so that it'll be
// rendered as part of Storyteller's results
Context.Reporting.Log(logger);

// This is bootstrapping a Jasper application through the 
// normal ASP.Net Core IWebHostBuilder
return JasperHost
    .CreateDefaultBuilder()
    .ConfigureLogging(x =>
    {
        x.SetMinimumLevel(LogLevel.Debug);
        x.AddDebug();
        x.AddConsole();
        
        // Add the logger to the new Jasper app
        // being built up
        x.AddProvider(logger);
    })

    .UseJasper(registry)
    .StartJasper();

And voila, the logging information is now part of the test results in a useful way so I can see a lot more information about what’s happening during the test execution.

It sucks that my code is throwing exceptions instead of just working, but at least I can see what the hell is going wrong now.

Get the debugger going quickly

To be honest, the friction of getting Storyteller tests running under a debugger has always been a drawback to Storyteller — especially compared to how fast that workflow is with tools like xUnit.Net that integrate seamlessly into your IDE. You’ve always been able to just attach your debugger to the running Storyteller process, but I’ve always found that to be clumsy and slow — especially when you’re trying to quickly cycle between attempted fixes and re-running the tests.

I made some attempts in Storyteller 5 to improve the situation (after we gave up on building a dotnet test adapter because that model is bonkers), but that still takes some set up time to make it work and even I have to always run to the documentation to remember how to do it. Sometime this weekend the solution for a quick xUnit.Net execution wrapper around Storyteller popped into my head and it honestly took about 15 minutes flat to get things working so that I could kick off individual Storyteller specifications from xUnit.Net as shown below:

StorytellerWithinXUnit

Maybe that’s not super exciting, but the end result is that I can rerun a specification after making changes with or without debugging with nothing but a simple keyboard shortcut in the IDE. That’s a dramatically faster feedback cycle than what I had to begin with.

Implementation wise, I just took advantage of xUnit.Net’s [MemberData] feature for parameterized tests and Storyteller’s StorytellerRunner class that was built to allow users to run specifications from their own code. After adding a new xUnit.Net test project and referencing the original Storyteller specification project named “StorytellerSpecs, ” I added the code file shown below in its entirety::

// This only exists as a hook to dispose the static
// StorytellerRunner that is hosting the underlying
// system under test at the end of all the spec
// executions
public class StorytellerFixture : IDisposable
{
    public void Dispose()
    {
        Runner.SpecRunner.Dispose();
    }
}

public class Runner : IClassFixture<StorytellerFixture>
{
    internal static readonly StoryTeller.StorytellerRunner SpecRunner;

    static Runner()
    {
        // I'll admit this is ugly, but this establishes where the specification
        // files live in the real StorytellerSpecs project
        var directory = AppContext.BaseDirectory
            .ParentDirectory()
            .ParentDirectory()
            .ParentDirectory()
            .ParentDirectory()
            .AppendPath("StorytellerSpecs")
            .AppendPath("Specs");

        SpecRunner = new StoryTeller.StorytellerRunner(new SpecSystem(), directory);
    }

    // Discover all the known Storyteller specifications
    public static IEnumerable<object[]> GetFiles()
    {
        var specifications = SpecRunner.Hierarchy.Specifications.GetAll();
        return specifications.Select(x => new object[] {x.path}).ToArray();
    }

    // Use a touch of xUnit.Net magic to be able to kick off and
    // run any Storyteller specification through xUnit
    [Theory]
    [MemberData(nameof(GetFiles))]
    public void run_specification(string path)
    {
        var results = SpecRunner.Run(path);
        if (!results.Counts.WasSuccessful())
        {
            SpecRunner.OpenResultsInBrowser();
            throw new Exception(results.Counts.ToString());
        }
    }
}

And that’s that. Something I’ve wanted to have for ages and failed to build, done in 15 minutes because I happened to remember something similar we’d done at work and realized how absurdly easy xUnit.Net made this effort.

Summary

  • Sometimes it’s worthwhile to take a step back from trying to solve a problem through debugging and invest in better instrumentation or write some automation scripts to make the debugging cycles faster rather than just trying to force your way through the solution
  • Inspiration happens at random times
  • Listen to that niggling voice in your head sometimes that’s telling you that you should be doing things differently in your code or tests
  • Using a modular architecture that’s composed by an IoC container the way that ASP.net Core does can sometimes be advantageous in integration testing scenarios. Case in point is how easy it was for me to toss in an all new logging provider that captured the log information directly into the test results for easier test failure resolution

And when I get around to it, these little Storyteller improvements will end up in Storyteller itself.

Jasper meets Azure Service Bus and gets better with Rabbit MQ

This is just a super quick update on Jasper so I can prove to the world that it’s actually moving forward. I’ll hopefully have a lot more on Jasper in the next couple weeks when I push the next big alpha.

I posted a new release of Jasper last week (v0.9.8.6) that has some better transport support for using Jasper commands with real messaging infrastructure:

I’d definitely love any feedback on the configuration of either transport option. I’m aiming to allow users to have access to every last bit of advanced features that either transport exposes, while also enabling users to quickly swap out transport mechanisms between executing locally, on test, and in production as necessary.

 

What’s Next

I really want to get to a 1.0 release soon because this has dragged on so much longer than I’d hoped. Right now I’m focused on improving the message persistence options and add automatic messaging idempotent rules.

 

My OSS Plans for 2019

I wrote a similar post last year on My OSS Plans for 2018, and it wasn’t too far off what actually happened. I did switch jobs last May, and until recently, that dramatically slowed down my rate of OSS contributions and my ambition level for OSS work. I still enjoy building things, I’m in what’s supposed to be a mostly non-coding role now (but I absolutely still do), and working on OSS projects is just a good way to try to keep my technical skills up. So here we go:

Marten — I’m admittedly not super active with Marten these days as it’s a full fledged community driven project now. There has been a pile up of Linq related issues lately and the issue backlog is getting a bit big, so it really wouldn’t hurt for me to be more active. I happen to love Stephen King’s Dark Tower novels (don’t bother with the movie), but he used to complain that they were hard for him to get into the right head space to write more about Roland and his Ka-tet. That’s basically how I feel about the Linq provider in Marten, but it’s over due for some serious love.

Lamar (was BlueMilk) — Users are reporting some occasional memory usage problems that I think point to issues in Roslyn itself, but in the meantime I’ve got some workarounds in mind to at least alleviate the issue (maybe, hopefully, knock on wood). The one and only big feature idea for this year I have in mind is to finally do the “auto-factory” feature that I hope will knock out a series of user requests for more dynamic behavior.

Alba — I took some time at CodeMash this year and did a pretty substantial 3.0 release that repositioned Alba as a productivity helper on top of TestServer to hopefully eliminate a whole lot of compatibility issues with ASP.Net Core. I don’t know that I have any other plans for Alba this year, but it’s the one tool I work on that I actually get to use at work, so there might be some usability improvements over time.

Storyteller — I’d really love to sink into a pretty big rewrite of the user interface and some incremental — but perfectly backward compatible — improvements to the engine. After a couple years of mostly doing integration work, I suddenly have some UI centric projects ahead of me and I could definitely use some refresher on building web UIs. More on this in a later post.

Jasper — I think I’m finally on track for a 1.0 release in the next couple months, but I’m a little unsure if I want to wait for ASP.Net Core 3.0 or not. After that, it’s mostly going to be trying to build some community around Jasper. Most of my side project time and effort the past three years has been toward Jasper, I have conceptual notes on its architecture that go back at least 5 years, and counting its predecessor project FubuMVC, this has been a 10 year effort for me. Honestly, I think I’m going to finish the push to 1.0 just to feel some sense of completion.

Oakton — I feel like it’s basically done

StructureMap — I answer questions here and there, but otherwise it’s finished/abandoned

FubuMVC — Dead, but little bits and pieces of it live on in Oakton, Alba, and Jasper