Jasper meets Azure Service Bus and gets better with Rabbit MQ

This is just a super quick update on Jasper so I can prove to the world that it’s actually moving forward. I’ll hopefully have a lot more on Jasper in the next couple weeks when I push the next big alpha.

I posted a new release of Jasper last week (v0.9.8.6) that has some better transport support for using Jasper commands with real messaging infrastructure:

I’d definitely love any feedback on the configuration of either transport option. I’m aiming to allow users to have access to every last bit of advanced features that either transport exposes, while also enabling users to quickly swap out transport mechanisms between executing locally, on test, and in production as necessary.

 

What’s Next

I really want to get to a 1.0 release soon because this has dragged on so much longer than I’d hoped. Right now I’m focused on improving the message persistence options and add automatic messaging idempotent rules.

 

Advertisements

My OSS Plans for 2019

I wrote a similar post last year on My OSS Plans for 2018, and it wasn’t too far off what actually happened. I did switch jobs last May, and until recently, that dramatically slowed down my rate of OSS contributions and my ambition level for OSS work. I still enjoy building things, I’m in what’s supposed to be a mostly non-coding role now (but I absolutely still do), and working on OSS projects is just a good way to try to keep my technical skills up. So here we go:

Marten — I’m admittedly not super active with Marten these days as it’s a full fledged community driven project now. There has been a pile up of Linq related issues lately and the issue backlog is getting a bit big, so it really wouldn’t hurt for me to be more active. I happen to love Stephen King’s Dark Tower novels (don’t bother with the movie), but he used to complain that they were hard for him to get into the right head space to write more about Roland and his Ka-tet. That’s basically how I feel about the Linq provider in Marten, but it’s over due for some serious love.

Lamar (was BlueMilk) — Users are reporting some occasional memory usage problems that I think point to issues in Roslyn itself, but in the meantime I’ve got some workarounds in mind to at least alleviate the issue (maybe, hopefully, knock on wood). The one and only big feature idea for this year I have in mind is to finally do the “auto-factory” feature that I hope will knock out a series of user requests for more dynamic behavior.

Alba — I took some time at CodeMash this year and did a pretty substantial 3.0 release that repositioned Alba as a productivity helper on top of TestServer to hopefully eliminate a whole lot of compatibility issues with ASP.Net Core. I don’t know that I have any other plans for Alba this year, but it’s the one tool I work on that I actually get to use at work, so there might be some usability improvements over time.

Storyteller — I’d really love to sink into a pretty big rewrite of the user interface and some incremental — but perfectly backward compatible — improvements to the engine. After a couple years of mostly doing integration work, I suddenly have some UI centric projects ahead of me and I could definitely use some refresher on building web UIs. More on this in a later post.

Jasper — I think I’m finally on track for a 1.0 release in the next couple months, but I’m a little unsure if I want to wait for ASP.Net Core 3.0 or not. After that, it’s mostly going to be trying to build some community around Jasper. Most of my side project time and effort the past three years has been toward Jasper, I have conceptual notes on its architecture that go back at least 5 years, and counting its predecessor project FubuMVC, this has been a 10 year effort for me. Honestly, I think I’m going to finish the push to 1.0 just to feel some sense of completion.

Oakton — I feel like it’s basically done

StructureMap — I answer questions here and there, but otherwise it’s finished/abandoned

FubuMVC — Dead, but little bits and pieces of it live on in Oakton, Alba, and Jasper

 

Marten 3.4: Full text search

Marten 3.4 was release this week with bug fixes, some performance improvements (by using ImTools internally in place of ConcurrentDictionary), and one giant new feature to support full text searching using Postgresql v10’s new full text searching features. Congratulations to the core Marten team (Babu Annamalai, Oscar Dudycz,  and Joona-Pekka Kokko) for all their hard work on this release.

Taken from the Marten docs, here’s some of the usage:

Postgres contains built in Text Search functions. They enable the possibility to do more sophisticated searching through text fields. Marten gives possibility to define Full Text Indexes and perform queries on them. Currently three types of full Text Search functions are supported:

  • regular Search (to_tsquery)
var posts = session.Query<BlogPost>()
    .Where(x => x.Search("somefilter"))
    .ToList();
  • plain text Search (plainto_tsquery)
var posts = session.Query<BlogPost>()
    .Where(x => x.PlainTextSearch("somefilter"))
    .ToList();
  • phrase Search (phraseto_tsquery)
var posts = session.Query<BlogPost>()
    .Where(x => x.PhraseSearch("somefilter"))
    .ToList();

All types of Text Searches can be combined with other Linq queries

var posts = session.Query<BlogPost>()
    .Where(x => x.Category == "LifeStyle")
    .Where(x => x.PhraseSearch("somefilter"))
    .ToList();

They allow also to specify language (regConfig) of the text search query (by default english is being used)

var posts = session.Query<BlogPost>()
    .Where(x => x.PhraseSearch("somefilter", "italian"))
    .ToList();

See also, Full Text indexes for information on setting up indexing on documents.

Building a Hybrid Jasper/MVC Core Application

This was originally a tutorial on Jasper’s documentation website. Assuming you even remember FubuMVC and ask me what’s going to be different about Jasper this time around, one of the first things I’ll bring up is that it’s perfectly possible to mix and match HTTP actions handled by either Jasper or MVC Core (or Carter for that matter) within the same application.

Jasper is trying very hard to be merely a citizen within the greater ASP.Net Core ecosystem than its own standalone framework. To that end, Jasper plays nicely with ASP.Net Core, as shown in this example that adds Jasper to an MVC Core application.

Start by creating a new project with this command line:

dotnet new webapi

Now, add a Nuget reference to Jasper. In your Program.Main() method, make these changes noted in comments:

public class Program
{
    // Return an int for a status code
    public static int Main(string[] args)
    {
        // Calling RunJasper() opts into Jasper's expansive
        // command line skeleton with diagnostics you probably
        // want
        return CreateWebHostBuilder(args).RunJasper(args);
    }

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .UseStartup<Startup>()

            // Add Jasper with all its defaults
            .UseJasper();
}

Next, go into the generated Startup class and make the changes shown here with comments:

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
        services.AddDbContext<UserDbContext>();
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IHostingEnvironment env, JasperOptions messaging)
    {
        // This is optional, but it's awfully helpful
        // to configure the message bus part of Jasper directly
        // from configuration
        messaging.ListenForMessagesFrom(Configuration["ListeningEndpoint"]);
        messaging.AddSubscription(Subscription.All(Configuration["OtherServiceUri"]));


        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }
        else
        {
            // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
            app.UseHsts();
        }

        app.UseHttpsRedirection();

        // The ordering here is meaningful, but we think that
        // Jasper's routing is more efficient, so let it try
        // first
        app.UseJasper();

        app.UseMvc();
    }
}

In this case, I’m assuming that there’s one single messaging listener and all published messages will get sent to the same location. Both of those Uri values are in the appsettings.json file shown below:

{
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ListeningEndoint": "tcp://localhost:2100",
  "OtherServiceUri": "tcp://localhost:2200"
}

Now, after running the application from a straight up call to dotnet run (and maybe remembering to force your csproj to copy the appsettings.json file around), you should get a lot of command line output like this:

dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[3]
      Hosting starting
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0]
      User profile is available. Using '/Users/jeremydmiller/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
dbug: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[37]
      Reading data from file '/Users/jeremydmiller/.aspnet/DataProtection-Keys/key-fdd38e0a-635d-4d73-93bf-7197b3bf71a8.xml'.
dbug: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[37]
      Reading data from file '/Users/jeremydmiller/.aspnet/DataProtection-Keys/key-506cdf5d-957b-4bdd-8b07-3edf5a109e9a.xml'.
dbug: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[37]
      Reading data from file '/Users/jeremydmiller/.aspnet/DataProtection-Keys/key-122b0ab4-2cb0-49ac-ae79-98a3ca730514.xml'.
dbug: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[18]
      Found key {fdd38e0a-635d-4d73-93bf-7197b3bf71a8}.
dbug: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[18]
      Found key {506cdf5d-957b-4bdd-8b07-3edf5a109e9a}.
dbug: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[18]
      Found key {122b0ab4-2cb0-49ac-ae79-98a3ca730514}.
dbug: Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver[13]
      Considering key {fdd38e0a-635d-4d73-93bf-7197b3bf71a8} with expiration date 2019-02-26 20:28:23Z as default key.
dbug: Microsoft.AspNetCore.DataProtection.TypeForwardingActivator[0]
      Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
dbug: Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ManagedAuthenticatedEncryptorFactory[11]
      Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
dbug: Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ManagedAuthenticatedEncryptorFactory[10]
      Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
dbug: Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider[2]
      Using key {fdd38e0a-635d-4d73-93bf-7197b3bf71a8} as the default key.
dbug: Microsoft.AspNetCore.DataProtection.Internal.DataProtectionStartupFilter[0]
      Key ring with default key {fdd38e0a-635d-4d73-93bf-7197b3bf71a8} was loaded during application startup.
dbug: Microsoft.AspNetCore.Mvc.MvcJsonOptions[0]
      Compatibility switch AllowInputFormatterExceptionMessages in type MvcJsonOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch AllowCombiningAuthorizeFilters in type MvcOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch AllowBindingHeaderValuesToNonStringModelTypes in type MvcOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch AllowValidatingTopLevelNodes in type MvcOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch InputFormatterExceptionPolicy in type MvcOptions is using compatibility value MalformedInputExceptions for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch SuppressBindingUndefinedValueToEnumType in type MvcOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch EnableEndpointRouting in type MvcOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch MaxValidationDepth in type MvcOptions is using compatibility value 32 for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcOptions[0]
      Compatibility switch AllowShortCircuitingValidationWhenNoValidatorsArePresent in type MvcOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.ApiBehaviorOptions[0]
      Compatibility switch SuppressMapClientErrors in type ApiBehaviorOptions is using default value False
dbug: Microsoft.AspNetCore.Mvc.ApiBehaviorOptions[0]
      Compatibility switch SuppressUseValidationProblemDetailsForInvalidModelStateResponses in type ApiBehaviorOptions is using default value False
dbug: Microsoft.AspNetCore.Mvc.ApiBehaviorOptions[0]
      Compatibility switch AllowInferringBindingSourceForCollectionTypesAsFromQuery in type ApiBehaviorOptions is using default value False
dbug: Microsoft.AspNetCore.Mvc.RazorPages.RazorPagesOptions[0]
      Compatibility switch AllowAreas in type RazorPagesOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.RazorPages.RazorPagesOptions[0]
      Compatibility switch AllowMappingHeadRequestsToGetHandler in type RazorPagesOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.RazorPages.RazorPagesOptions[0]
      Compatibility switch AllowDefaultHandlingForOptionsRequests in type RazorPagesOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.Razor.RazorViewEngineOptions[0]
      Compatibility switch AllowRecompilingViewsOnFileChange in type RazorViewEngineOptions is using explicitly configured value True
dbug: Microsoft.AspNetCore.Mvc.MvcViewOptions[0]
      Compatibility switch SuppressTempDataAttributePrefix in type MvcViewOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.MvcViewOptions[0]
      Compatibility switch AllowRenderingMaxLengthAttribute in type MvcViewOptions is using compatibility value True for version Version_2_2
dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ModelBinderFactory[12]
      Registered model binder providers, in the following order: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexTypeModelBinderProvider
dbug: Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer[2]
      Failed to locate the development https certificate at '(null)'.
dbug: Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer[0]
      Using development certificate: CN=localhost (Thumbprint: 81C087389BE208DFD502D193565DB7503A96E805)
dbug: Microsoft.AspNetCore.Server.Kestrel[0]
      No listening endpoints were configured. Binding to http://localhost:5000 and https://localhost:5001 by default.
dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[4]
      Hosting started
dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[0]
      Loaded hosting startup assembly MvcCoreHybrid, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Running service 'MvcCoreHybrid'
Application Assembly: MvcCoreHybrid, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Hosting environment: Development
Content root path: /Users/jeremydmiller/code/jasper/src/MvcCoreHybrid
Hosted Service: Jasper.JasperActivator
Hosted Service: Jasper.Messaging.Logging.MetricsCollector
Hosted Service: Jasper.Messaging.BackPressureAgent
Listening for loopback messages

Active sending agent to tcp://localhost:2200/
Active sending agent to loopback://retries/

Application started. Press Ctrl+C to shut down.

If you stop that, and now see what other command line utilities Jasper has added to your project, type:

dotnet run -- help

And you’ll see a lot of new options:

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------
    Available commands:
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------
        code -> Display or export the runtime, generated code in this application
    describe -> Preview the configuration of the Jasper application without running the application or starting Kestrel
         run -> Runs the configured Jasper application
    services -> Display the known Lamar service registrations
    validate -> Validate the configuration and environment for this Jasper application
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------

See Jasper Command Line Support for more information about Jasper’s command line support.

Jasper as Lightweight Service Bus

I’ve suddenly got some momentum going on Jasper, and I’m trying to keep up a steady pace of blog posts about it to try to get some visibility and see about possibly attracting some community around it. This tutorial is admittedly just taken from the documentation website.

Using its own built in TCP Transport, Jasper can be used as a very lightweight service bus to send messages asynchronously between Jasper applications. In this tutorial, we’re going to create two Jasper services that exchange messages with each either.

Note! While this was originally meant for production applications — and its predecessor implementation from FubuMVC has been in production under significant load for 5-6 years, we think this functionality will be mostly useful for testing scenarios where your production transport is unusable locally. Looking at you Azure Service Bus.

For the purpose of this tutorial, we’re going to heavily leverage the dotnet command line tools, so if you would, open your favorite command line tool.

The first step is just to ensure you have the latest Jasper project templates by installing JasperTemplates as shown below:

dotnet new -i JasperTemplates

Next, let’s build up a new .Net solution with these three projects:

  1. Messages — just a class library that holds shared message types we’ll be sending back and forth.
  2. Pinger — a Jasper service that will send PingMessage messages every 5 seconds
  3. Ponger — a Jasper service that will receive PingMessage messages, and send a corresponding PongMessage back to the original sender service

Note! You do not have to use shared libraries of message DTO classes in order to use messaging between Jasper applications, but it’s the simplest possible way to get started, so here it is.

To build out this new solution, you can use the following command line script and then open the newly created PingAndPong.sln file:

mkdir PingAndPong
cd PingAndPong

dotnet new classlib --name Messages
dotnet new jasper.service --name Pinger
dotnet new jasper.service --name Ponger

dotnet add Pinger/Pinger.csproj reference Messages/Messages.csproj

dotnet new sln

dotnet sln PingAndPong.sln add Messages/Messages.csproj Pinger/Pinger.csproj Ponger/Ponger.csproj

In the Messages Project

All we need to do in the Messages project is to add these message types for later:

public class PingMessage
{
    public Guid Id { get; set; }
    public string Name { get; set; }
}

public class PongMessage
{
    public Guid Id { get; set; }
    public string Name { get; set; }
}

In the Ponger Project

We need to make a couple changes to Ponger. First, open the generated JasperConfig class and add the line of code shown below to tell Ponger to listen for messages on port 3051:

    internal class JasperConfig : JasperRegistry
    {
        public JasperConfig()
        {
            Transports.LightweightListenerAt(3051);
        }
    }

Next, we need to handle the PingMessage message, and send a PongMessage back to the original sender:

public class PingHandler
{
    public Response Handle(PingMessage message)
    {
        ConsoleWriter.Write(ConsoleColor.Cyan, "Got a ping with name: " + message.Name);

        var response = new PongMessage
        {
            Name = message.Name
        };

        // Don't know if you'd use this very often,
        // but this is a special syntax that will send
        // the "response" back to the original sender
        return Respond.With(response).ToSender();
    }
}

In the Pinger Project

In Pinger, we need a handler for the PongMessage messages coming back, so add this class:

public class PongHandler
{
    public void Handle(PongMessage message)
    {
        ConsoleWriter.Write(ConsoleColor.Cyan, "Got a pong back with name: " + message.Name);
    }
}

We also need something that just runs in the background and sends PingMessage messages out. For that, we’ll use an implementation of ASP.Net Core’s IHostedService:

    // In this case, BackgroundService is a base class
    // for the IHostedService that is *supposed* to be
    // in a future version of ASP.Net Core that I shoved
    // into Jasper so we could use it now. The one in Jasper
    // will be removed later when the real one exists in
    // ASP.Net Core itself
    public class PingSender : BackgroundService
    {
        private readonly IMessageContext _bus;

        public PingSender(IMessageContext bus)
        {
            _bus = bus;
        }

        protected override Task ExecuteAsync(CancellationToken stoppingToken)
        {
            var count = 1;

            return Task.Run(async () =>
            {
                while (!stoppingToken.IsCancellationRequested)
                {
                    await Task.Delay(1000, stoppingToken);

                    await _bus.Send(new PingMessage
                    {
                        Name = "Message" + count++
                    });
                }
            }, stoppingToken);
        }
    }

Lastly in Pinger, open the JasperConfig class and add a single line of code shown below that directs Jasper to listen for messages using its TCP Transport at port 3050. We’ll also make a static publishing rule to publish all messages to the Ponger service. Lastly, we need to register the PingSender with the system so that it runs continuously when the application is started.

All that is shown below:

    internal class JasperConfig : JasperRegistry
    {
        public JasperConfig()
        {
            
            // Directs Jasper to use the TCP listener and
            // bind to port 3050. 
            Transports.LightweightListenerAt(3050);
            
            // Send all published messages to this location
            Publish.AllMessagesTo("tcp://localhost:3051");

            Services.AddSingleton<IHostedService, PingSender>();
        }
    }

Running Pinger and Ponger

Now you’ll want a pair of command windows open, one to the root directory of Pinger and one to Ponger.

Since it’s a little bit cleaner, start up Ponger first with a simple call to dotnet run from the Ponger directory and you should see output like this:

ComputerName:Ponger user$ dotnet run
dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[3]
      Hosting starting
Jasper 'Nullo' startup is being used to start the ASP.Net Core application
dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[4]
      Hosting started
dbug: Microsoft.AspNetCore.Hosting.Internal.WebHost[0]
      Loaded hosting startup assembly Ponger, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Running service 'JasperConfig'
Application Assembly: Ponger, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Hosting environment: Production
Content root path: /Users/jeremydmiller/code/PingAndPong/Ponger/bin/Debug/netcoreapp2.1/
Hosted Service: Jasper.JasperActivator
Hosted Service: Jasper.Messaging.Logging.MetricsCollector
Hosted Service: Jasper.Messaging.BackPressureAgent
Listening for loopback messages

Active sending agent to loopback://retries/
Handles messages:
  PingMessage: HandlerType: Ponger.PingHandler, Method: Jasper.Messaging.Runtime.Invocation.Response Handle(Messages.PingMessage)

Application started. Press Ctrl+C to shut down.

Then start Pinger with the same dotnet run command from the root of the Pinger project and well, you’ll get a boatload of logging like this:

dbug: Jasper.Messages[101]
      Received PongMessage#0168be78-ddcd-43b6-8fc4-15cc855354c3 at tcp://localhost:3050/ from tcp://localhost:3051/
Got a pong back with name: Message18
dbug: Jasper.Messages[102]
      Started processing PongMessage#0168be78-ddcd-43b6-8fc4-15cc855354c3
dbug: Jasper.Messages[103]
      Finished processing PongMessage#0168be78-ddcd-43b6-8fc4-15cc855354c3
info: Jasper.Messages[104]
      Successfully processed message PongMessage#0168be78-ddcd-43b6-8fc4-15cc855354c3 from tcp://localhost:3051/
dbug: Jasper.Messages[100]
      Sent PingMessage#0168be78-e0ed-4eec-8081-05e7b1f36006 to tcp://localhost:3051/
info: Jasper.Transports[204]
      Sending agent for tcp://localhost:3051/ has resumed
dbug: Jasper.Transports[200]
      Successfully sent 1 messages to tcp://localhost:3051/
dbug: Jasper.Transports[202]
      Received 1 message(s) from tcp://localhost:3051/
dbug: Jasper.Messages[101]
      Received PongMessage#0168be78-e1b9-4bc8-bc58-98cdccf9824b at tcp://localhost:3050/ from tcp://localhost:3051/
dbug: Jasper.Messages[102]
      Started processing PongMessage#0168be78-e1b9-4bc8-bc58-98cdccf9824b
Got a pong back with name: Message19
dbug: Jasper.Messages[103]
      Finished processing PongMessage#0168be78-e1b9-4bc8-bc58-98cdccf9824b
info: Jasper.Messages[104]
      Successfully processed message PongMessage#0168be78-e1b9-4bc8-bc58-98cdccf9824b from tcp://localhost:3051/
dbug: Jasper.Messages[100]
      Sent PingMessage#0168be78-e4da-4c36-95e3-aa9d15ee0ff2 to tcp://localhost:3051/
info: Jasper.Transports[204]
      Sending agent for tcp://localhost:3051/ has resumed
dbug: Jasper.Transports[200]
      Successfully sent 1 messages to tcp://localhost:3051/
dbug: Jasper.Transports[202]
      Received 1 message(s) from tcp://localhost:3051/
dbug: Jasper.Messages[101]
      Received PongMessage#0168be78-e5a7-4e03-9894-4c3f61ed7413 at tcp://localhost:3050/ from tcp://localhost:3051/
dbug: Jasper.Messages[102]
      Started processing PongMessage#0168be78-e5a7-4e03-9894-4c3f61ed7413
Got a pong back with name: Message20
dbug: Jasper.Messages[103]
      Finished processing PongMessage#0168be78-e5a7-4e03-9894-4c3f61ed7413
info: Jasper.Messages[104]
      Successfully processed message PongMessage#0168be78-e5a7-4e03-9894-4c3f61ed7413 from tcp://localhost:3051/
dbug: Jasper.Messages[100]
      Sent PingMessage#0168be78-e8c5-4130-a919-81bfe235becb to tcp://localhost:3051/
info: Jasper.Transports[204]
      Sending agent for tcp://localhost:3051/ has resumed
dbug: Jasper.Transports[200]
      Successfully sent 1 messages to tcp://localhost:3051/
dbug: Jasper.Transports[202]
      Received 1 message(s) from tcp://localhost:3051/
dbug: Jasper.Messages[101]
      Received PongMessage#0168be78-e991-41b9-b1f0-16ee2fa75377 at tcp://localhost:3050/ from tcp://localhost:3051/
dbug: Jasper.Messages[102]
      Started processing PongMessage#0168be78-e991-41b9-b1f0-16ee2fa75377
Got a pong back with name: Message21

Alright, you’ve got a fully functional system of two services who constantly chat with each other. For more information about the topics we covered in this tutorial, see the documentation for Messaging

New Jasper Alpha for HTTP Services

Jasper is the name of an OSS project I’ve been working on for a disturbingly long time that’s the spiritual successor to FubuMVC. In more concrete terms, Jasper is built around what I think is a unique Roslyn-powered command execution pipeline that can be used as any combination of:

  1. An alternative to MVC Core for authoring HTTP services
  2. An in-memory service bus or command executor ala MediatR, but with more functionality
  3. A lightweight service bus
  4. A full fledged service bus when used in conjunction with real messaging infrastructure like Rabbit MQ or Azure Service Bus (Jasper fills the mediator rule between your code and the underlying transport)

To get you started on Jasper and to see what it’s capable, there’s actually a set of tutorials so far:

  1. Getting Started
  2. Jasper as an In Memory Command Executor / Mediator
  3. Using Jasper as a Lightweight Service Bus
  4. Building HTTP Services with Jasper
  5. Hybrid Jasper/MVC Core Application

 

What’s changed recently?

I changed Jasper’s direction a little bit starting last September. I’ve gone through the internals and tried very hard to replace any bit of code or infrastructure that was specific to Jasper that I thought could be adequately replaced by stuff built into ASP.Net Core. To that end, a lot of Jasper’s custom bootstrapping code was thrown out and replaced by fully embracing ASP.Net Core’s IWebHostBuilder infrastructure — even when Jasper is running “headless” with no real HTTP server.

I also replaced Jasper’s built in error handling and retry policies it had inherited from FubuMVC with a few extensions to Polly so users have the full power of Polly available to them — and I and any eventual Jasper community have a lot less code to document and support;)

I had quite a bit of free time at CodeMash and NDC London recently, so was able to work quite a bit on Jasper’s HTTP service support, including the new “MVC Extender” functionality that allows you to mix and match MVC Core elements into Jasper endpoints.

 

What’s Next

There’s going to be another pre-1.0 alpha within the next two weeks that will focus mostly on the messaging functionality. I can justify some day job time on it in the next couple weeks where I’ll be working on improving the Rabbit MQ integration, finishing the Azure Service Bus integration, and testing out the custom Azure AppInsights extension.

Beyond that? Performance optimization maybe, but I’d like to push for a 1.0 release soon and try to start building some interest and community around it. The only thing that will slow that down is to see if I wanna try to make Jasper 1.0 wait for ASP.Net Core 3.0 because every single OSS tool I maintain that depends on ASP.Net Core has been broken by every single ASP.Net Core release.

 

I’m very interested in whatever feedback you might have, either here, on twitter, or the Gitter room for Jasper.

 

Metaprogramming in C# with LamarCompiler

I gave a talk last week at NDC London called “Dynamic Runtime Code with Roslyn” that featured an example of using LamarCompiler and Lamar to create a minimal equivalent of the Refit library. If you’re not familiar with Refit, it allows you to express the intended service proxy to a web service with a .Net interface or abstract class, apply a few attributes to explain Url patterns or other HTTP request items, and then allow Refit to dynamically generate a class that implements your interface that deals with all the repetitive gunk of building Urls, serializing or deserializing JSON, and calling HttpClient to actually make the requests.

Cool, right? I’d argue that Refit is an example of metaprogramming inside of C# where a tool is able to generate and eventually execute code gleaned from data (an interface and a handful of attributes in this case) within the application. I tend to think of metaprogramming as something specific to the Ruby and other dynamic languages arena, but now that we have the ability to compile C# code on the fly with Roslyn, I think it’s going to become something much more commonly done in .Net systems.

Today in the Lamar codebase, there’s a little project called LamarRest that contains all the code I’m going to show here if you’re curious to see more about how it all fits together. I should say that I intended this code strictly as a demonstration of what LamarCompiler can do and do not have any plans right now to make this a production capable library.

The Problem and Conceptual Solution

Okay, let’s get concrete and write a very small web service that manages users. The whole API surface in this case is implemented by the UserController class shown below (the full source code for this web service is here):

    public class UserController : Controller
    {
        private readonly UserRepository _repository;
        private readonly ILogger _logger;

        public UserController(UserRepository repository, ILogger logger)
        {
            _repository = repository;
            _logger = logger;
        }

        [HttpPost("/user/create")]
        public IActionResult Create([FromBody] User user)
        {
            _logger.LogInformation($"I created a new user with the name '{user.Name}'");
            _repository.Users.Fill(user);
            return Ok();
        }
        
        [HttpGet("/user/{name}")]
        public User GetUser(string name)
        {
            return _repository.Users.FirstOrDefault(x => x.Name == name);
        }

        [HttpGet("/users")]
        public User[] GetUsers()
        {
            return _repository.Users.ToArray();
        }
    }

Now, back in the codebase for an application that needs to communicate with the user service, we ideally want to just write an interface to act as a proxy to the User web service that looks like this:

    public interface IUserService
    {
        [Get("/user/{name}")]
        Task GetUser(string name);

        [Post("/user/create")]
        Task Create(User user);
    }

In our application, we just want to interact with the IUserService and not really have to know where it comes from. To that end, I’m going to go a step farther than Refit does and say that our solution should automatically resolve the generated type for IUserService from our application’s IoC container on demand.

As you’ll notice, I did have to add a couple attributes ([Get] and [Post]) that just specify the Url pattern for the two proxy methods. In this case as well though, the presence of the route marker attributes tells the code we’re about to write that these methods should be generated to call HttpClient.

Implementing Our Refit Clone

To generate the concrete HTTP proxy classes, I’m going to leverage LamarCompiler’s Frame model. If you write out implementation of IUserService above, you’ll notice discrete set of logical steps, each of which becomes a LamarCompile Frame in this model:

  1. Resolving an HttpClient from the HttpClientFactory for our application.  That became BuildClientFrame.
  2. Fill the real Url by substituting any route arguments with method parameters from our interface method signatures. That’s implemented (crudely and incompletely) in LamarRest by the FillUrlFrame.
  3. If necessary, serialize the input boy argument to Json using Newtonsoft.Json’s default settings (Refit being a real library allows you to configure the Json serialization, LamarRest being a demo worked up the day before a conference talk, does not). For that purpose, I created SerializeJsonFrame.
  4. To actually send an Http request, we need to create an HttpRequestMessage object with the right Url, Http method, and the serialized Json if there is any. This one is a little easier, as I just subclass Lamar’s built in support for calling a class constructor with any mix of arguments and setters in the BuildRequestFrame class.
  5. To actually invoke HttpClient.SendAsync() , there’s just a built in MethodCall Frame inside of LamarCompiler.
  6. If there is a response body expected, as in the IUserService.GetUser() method, we need a Frame that can deserialize the contents of the HTTP response body into the desired model. I implemented that in DeserializeObjectFrame.

Okay, so that’s a lot of steps, but once these Frame classes were built, it was easy to assemble them to match an interface method. In LamarRest, there’s a fair sized class called GeneratedServiceType that handles all the code generation and compilation for these proxy classes. Most applicable to this discussion is the method that builds out the concrete proxy methods by assembling the necessary list of Frame objects as shown below:

public static void BuildOut(
    // The contract type
    Type interfaceType, 
    
    // The MethodInfo from Reflection that describes
    // the Url structure through attributes, the input type
    // if any, and the .Net type of the response (if any)
    MethodInfo definition, 
    
    // This models the method being generated by LamarCompiler
    GeneratedMethod generated)
{
    var path = definition.GetCustomAttribute();

    // Get the right HttpClient from IHttpClientFactory
    generated.Frames.Add(new BuildClientFrame(interfaceType));

    // Build out the Url from the method arguments and route
    // pattern
    var urlFrame = new FillUrlFrame(definition);
    generated.Frames.Add(urlFrame);

    
    // See if there is an input type that should be serialized
    // to the HTTP request body
    var inputType = DetermineRequestType(definition);
    if (inputType == null)
    {
        // Just build the HttpRequestMessage
        generated.Frames.Add(new BuildRequestFrame(path.Method, urlFrame, null));
    }
    else
    {
        // Add a step to serialize the input model to Json
        generated.Frames.Add(new SerializeJsonFrame(inputType));
        
        // Just build the HttpRequestMessage
        generated.Frames.Add(new BuildRequestFrame(path.Method, urlFrame, new SerializeJsonFrame(inputType)));
    }

    // Actually call the HttpClient to send the request
    generated.Frames.Call(x => x.SendAsync(null));

    // Is there a response type that should be serialized out of the HTTP
    // response body?
    var returnType = DetermineResponseType(definition);
    if (returnType != null)
    {
        // Deserialize the response JSON into a new variable
        var deserialize = new DeserializeObjectFrame(returnType);
        generated.Frames.Add(deserialize);
        
        // Return that deserialized object from the method
        generated.Frames.Return(deserialize.ReturnValue);
    }
}

I should note that LamarCompiler does some work of connecting variables, method arguments, and class fields by type and sometimes by name — which is why you see no explicit mention of how IHttpClientFactory comes into play. LamarCompiler is smart enough to see that a Frame object needs an IHttpClientFactory object, know that that service is a singleton-coped object in the application’s IoC container, and therefore generate a field for that service.

For the IUserService interface, LamarRest will generate this code for the concrete class, but I warn you now, it’s generated code and hence will be as ugly as sin:

    public class UserService : LamarRest.Testing.IUserService
    {
        private readonly System.Net.Http.IHttpClientFactory _httpClientFactory;

        public UserService(System.Net.Http.IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;
        }



        public async System.Threading.Tasks.Task GetUser(string name)
        {

            // From BuildClientFrame
            var httpClient = _httpClientFactory.CreateClient("IUserService");

            // From FillUrlFrame
            var url = $"/user/{name}";
            var httpRequestMessage = new System.Net.Http.HttpRequestMessage(new System.Net.Http.HttpMethod("GET"), url);
            using (var httpResponseMessage = await httpClient.SendAsync(httpRequestMessage))
            {

                // From DeserializeObjectFrame
                var user = Newtonsoft.Json.JsonConvert.DeserializeObject(await httpResponseMessage.Content.ReadAsStringAsync());
                return user;
            }

        }


        public Task Create(UserApp.Controllers.User user)
        {

            // From BuildClientFrame
            var httpClient = _httpClientFactory.CreateClient("IUserService");

            // From FillUrlFrame
            var url = $"/user/create";

            // From SerializeJsonFrame
            var json = Newtonsoft.Json.JsonConvert.SerializeObject(user);
            var httpRequestMessage = new System.Net.Http.HttpRequestMessage(new System.Net.Http.HttpMethod("POST"), url){Content = new System.Net.Http.StringContent(json, System.Text.Encoding.UTF8, "application/json")};
            return httpClient.SendAsync(httpRequestMessage);
        }

Hooking it up to the Application

Lastly, we need to hook up our new LamarRest library to our application’s IoC container and application configuration. Assuming that you’re using Lamar as your IoC container, the configuration might be something like this:

            var container = new Container(x =>
            {
                x.AddHttpClient(
                    typeof(IUserService).Name, 
                    // You'd pull this from actual configuration in real life
                    c => c.BaseAddress = new Uri("http://localhost:5000"));
                
                // Adding a Lamar policy here that will auto-resolve LamarRest
                // proxies on demand
                x.Policies.Add<LamarRestPolicy>();
            });

In the code above, I’m heavily leveraging the standard ASP.Net Core AddHttpClient() registration to deal with base Urls and default HTTP request attributes like authentication credentials. I’d probably switch it to being strong typed later, but for now the convention is to name the HttpClient based on the interface name.

The second registration is Lamar specific to add a policy to allow Lamar to auto-resolve a concrete class for any interface that is otherwise unknown to the container and has methods marked with any kind of LamarRest route attribute.

Just for completeness sake, that’s shown below:

    public class LamarRestPolicy : Lamar.IFamilyPolicy
    {
        public ServiceFamily Build(Type type, ServiceGraph serviceGraph)
        {
            if (type.GetMethods().Any(x => x.HasAttribute()))
            {
                var rules = new GenerationRules("LamarRest");
                var generatedAssembly = new GeneratedAssembly(rules);
                var generatedType = new GeneratedServiceType(generatedAssembly, type);

                var container = (IContainer)serviceGraph.RootScope;
                
                container.CompileWithInlineServices(generatedAssembly);

                return new ServiceFamily(
                    type, 
                    new IDecoratorPolicy[0], 
                    new ConstructorInstance(type, 
                    generatedType.CompiledType, 
                    ServiceLifetime.Singleton
                ));
            }

            return null;
        }

The final result of all this setup is to be able to do this:

            
            // Build an implementation of this service as needed
            // I'm using service location here, but you'd be able
            // to inject IUserService as a constructor argument as well
            var userService = container.GetInstance<IUserService>();