Author Archives: jeremydmiller

Metaprogramming in C# with LamarCompiler

I gave a talk last week at NDC London called “Dynamic Runtime Code with Roslyn” that featured an example of using LamarCompiler and Lamar to create a minimal equivalent of the Refit library. If you’re not familiar with Refit, it allows you to express the intended service proxy to a web service with a .Net interface or abstract class, apply a few attributes to explain Url patterns or other HTTP request items, and then allow Refit to dynamically generate a class that implements your interface that deals with all the repetitive gunk of building Urls, serializing or deserializing JSON, and calling HttpClient to actually make the requests.

Cool, right? I’d argue that Refit is an example of metaprogramming inside of C# where a tool is able to generate and eventually execute code gleaned from data (an interface and a handful of attributes in this case) within the application. I tend to think of metaprogramming as something specific to the Ruby and other dynamic languages arena, but now that we have the ability to compile C# code on the fly with Roslyn, I think it’s going to become something much more commonly done in .Net systems.

Today in the Lamar codebase, there’s a little project called LamarRest that contains all the code I’m going to show here if you’re curious to see more about how it all fits together. I should say that I intended this code strictly as a demonstration of what LamarCompiler can do and do not have any plans right now to make this a production capable library.

The Problem and Conceptual Solution

Okay, let’s get concrete and write a very small web service that manages users. The whole API surface in this case is implemented by the UserController class shown below (the full source code for this web service is here):

    public class UserController : Controller
    {
        private readonly UserRepository _repository;
        private readonly ILogger _logger;

        public UserController(UserRepository repository, ILogger logger)
        {
            _repository = repository;
            _logger = logger;
        }

        [HttpPost("/user/create")]
        public IActionResult Create([FromBody] User user)
        {
            _logger.LogInformation($"I created a new user with the name '{user.Name}'");
            _repository.Users.Fill(user);
            return Ok();
        }
        
        [HttpGet("/user/{name}")]
        public User GetUser(string name)
        {
            return _repository.Users.FirstOrDefault(x => x.Name == name);
        }

        [HttpGet("/users")]
        public User[] GetUsers()
        {
            return _repository.Users.ToArray();
        }
    }

Now, back in the codebase for an application that needs to communicate with the user service, we ideally want to just write an interface to act as a proxy to the User web service that looks like this:

    public interface IUserService
    {
        [Get("/user/{name}")]
        Task GetUser(string name);

        [Post("/user/create")]
        Task Create(User user);
    }

In our application, we just want to interact with the IUserService and not really have to know where it comes from. To that end, I’m going to go a step farther than Refit does and say that our solution should automatically resolve the generated type for IUserService from our application’s IoC container on demand.

As you’ll notice, I did have to add a couple attributes ([Get] and [Post]) that just specify the Url pattern for the two proxy methods. In this case as well though, the presence of the route marker attributes tells the code we’re about to write that these methods should be generated to call HttpClient.

Implementing Our Refit Clone

To generate the concrete HTTP proxy classes, I’m going to leverage LamarCompiler’s Frame model. If you write out implementation of IUserService above, you’ll notice discrete set of logical steps, each of which becomes a LamarCompile Frame in this model:

  1. Resolving an HttpClient from the HttpClientFactory for our application.  That became BuildClientFrame.
  2. Fill the real Url by substituting any route arguments with method parameters from our interface method signatures. That’s implemented (crudely and incompletely) in LamarRest by the FillUrlFrame.
  3. If necessary, serialize the input boy argument to Json using Newtonsoft.Json’s default settings (Refit being a real library allows you to configure the Json serialization, LamarRest being a demo worked up the day before a conference talk, does not). For that purpose, I created SerializeJsonFrame.
  4. To actually send an Http request, we need to create an HttpRequestMessage object with the right Url, Http method, and the serialized Json if there is any. This one is a little easier, as I just subclass Lamar’s built in support for calling a class constructor with any mix of arguments and setters in the BuildRequestFrame class.
  5. To actually invoke HttpClient.SendAsync() , there’s just a built in MethodCall Frame inside of LamarCompiler.
  6. If there is a response body expected, as in the IUserService.GetUser() method, we need a Frame that can deserialize the contents of the HTTP response body into the desired model. I implemented that in DeserializeObjectFrame.

Okay, so that’s a lot of steps, but once these Frame classes were built, it was easy to assemble them to match an interface method. In LamarRest, there’s a fair sized class called GeneratedServiceType that handles all the code generation and compilation for these proxy classes. Most applicable to this discussion is the method that builds out the concrete proxy methods by assembling the necessary list of Frame objects as shown below:

public static void BuildOut(
    // The contract type
    Type interfaceType, 
    
    // The MethodInfo from Reflection that describes
    // the Url structure through attributes, the input type
    // if any, and the .Net type of the response (if any)
    MethodInfo definition, 
    
    // This models the method being generated by LamarCompiler
    GeneratedMethod generated)
{
    var path = definition.GetCustomAttribute();

    // Get the right HttpClient from IHttpClientFactory
    generated.Frames.Add(new BuildClientFrame(interfaceType));

    // Build out the Url from the method arguments and route
    // pattern
    var urlFrame = new FillUrlFrame(definition);
    generated.Frames.Add(urlFrame);

    
    // See if there is an input type that should be serialized
    // to the HTTP request body
    var inputType = DetermineRequestType(definition);
    if (inputType == null)
    {
        // Just build the HttpRequestMessage
        generated.Frames.Add(new BuildRequestFrame(path.Method, urlFrame, null));
    }
    else
    {
        // Add a step to serialize the input model to Json
        generated.Frames.Add(new SerializeJsonFrame(inputType));
        
        // Just build the HttpRequestMessage
        generated.Frames.Add(new BuildRequestFrame(path.Method, urlFrame, new SerializeJsonFrame(inputType)));
    }

    // Actually call the HttpClient to send the request
    generated.Frames.Call(x => x.SendAsync(null));

    // Is there a response type that should be serialized out of the HTTP
    // response body?
    var returnType = DetermineResponseType(definition);
    if (returnType != null)
    {
        // Deserialize the response JSON into a new variable
        var deserialize = new DeserializeObjectFrame(returnType);
        generated.Frames.Add(deserialize);
        
        // Return that deserialized object from the method
        generated.Frames.Return(deserialize.ReturnValue);
    }
}

I should note that LamarCompiler does some work of connecting variables, method arguments, and class fields by type and sometimes by name — which is why you see no explicit mention of how IHttpClientFactory comes into play. LamarCompiler is smart enough to see that a Frame object needs an IHttpClientFactory object, know that that service is a singleton-coped object in the application’s IoC container, and therefore generate a field for that service.

For the IUserService interface, LamarRest will generate this code for the concrete class, but I warn you now, it’s generated code and hence will be as ugly as sin:

    public class UserService : LamarRest.Testing.IUserService
    {
        private readonly System.Net.Http.IHttpClientFactory _httpClientFactory;

        public UserService(System.Net.Http.IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;
        }



        public async System.Threading.Tasks.Task GetUser(string name)
        {

            // From BuildClientFrame
            var httpClient = _httpClientFactory.CreateClient("IUserService");

            // From FillUrlFrame
            var url = $"/user/{name}";
            var httpRequestMessage = new System.Net.Http.HttpRequestMessage(new System.Net.Http.HttpMethod("GET"), url);
            using (var httpResponseMessage = await httpClient.SendAsync(httpRequestMessage))
            {

                // From DeserializeObjectFrame
                var user = Newtonsoft.Json.JsonConvert.DeserializeObject(await httpResponseMessage.Content.ReadAsStringAsync());
                return user;
            }

        }


        public Task Create(UserApp.Controllers.User user)
        {

            // From BuildClientFrame
            var httpClient = _httpClientFactory.CreateClient("IUserService");

            // From FillUrlFrame
            var url = $"/user/create";

            // From SerializeJsonFrame
            var json = Newtonsoft.Json.JsonConvert.SerializeObject(user);
            var httpRequestMessage = new System.Net.Http.HttpRequestMessage(new System.Net.Http.HttpMethod("POST"), url){Content = new System.Net.Http.StringContent(json, System.Text.Encoding.UTF8, "application/json")};
            return httpClient.SendAsync(httpRequestMessage);
        }

Hooking it up to the Application

Lastly, we need to hook up our new LamarRest library to our application’s IoC container and application configuration. Assuming that you’re using Lamar as your IoC container, the configuration might be something like this:

            var container = new Container(x =>
            {
                x.AddHttpClient(
                    typeof(IUserService).Name, 
                    // You'd pull this from actual configuration in real life
                    c => c.BaseAddress = new Uri("http://localhost:5000"));
                
                // Adding a Lamar policy here that will auto-resolve LamarRest
                // proxies on demand
                x.Policies.Add<LamarRestPolicy>();
            });

In the code above, I’m heavily leveraging the standard ASP.Net Core AddHttpClient() registration to deal with base Urls and default HTTP request attributes like authentication credentials. I’d probably switch it to being strong typed later, but for now the convention is to name the HttpClient based on the interface name.

The second registration is Lamar specific to add a policy to allow Lamar to auto-resolve a concrete class for any interface that is otherwise unknown to the container and has methods marked with any kind of LamarRest route attribute.

Just for completeness sake, that’s shown below:

    public class LamarRestPolicy : Lamar.IFamilyPolicy
    {
        public ServiceFamily Build(Type type, ServiceGraph serviceGraph)
        {
            if (type.GetMethods().Any(x => x.HasAttribute()))
            {
                var rules = new GenerationRules("LamarRest");
                var generatedAssembly = new GeneratedAssembly(rules);
                var generatedType = new GeneratedServiceType(generatedAssembly, type);

                var container = (IContainer)serviceGraph.RootScope;
                
                container.CompileWithInlineServices(generatedAssembly);

                return new ServiceFamily(
                    type, 
                    new IDecoratorPolicy[0], 
                    new ConstructorInstance(type, 
                    generatedType.CompiledType, 
                    ServiceLifetime.Singleton
                ));
            }

            return null;
        }

The final result of all this setup is to be able to do this:

            
            // Build an implementation of this service as needed
            // I'm using service location here, but you'd be able
            // to inject IUserService as a constructor argument as well
            var userService = container.GetInstance<IUserService>();
            

 

Advertisements

Alba 3.0 – TestServer on steroids and HTTP contract testing for .Net

Hey all, I was just able to make a 3.0 release of Alba with quite a few improvements. As introduced a couple years ago, Alba is a library you can use in combination with xUnit/NUnit to much more easily author integration tests against ASP.Net Core applications by running HTTP requests in process.

Some salient things:

Big Changes:

  • The core Alba Nuget package supports ASP.Net Core v2.1 and v2.2. Likewise, I’ve tested Alba with netcoreapp2.0 and netcoreapp2.1
  • The Alba.AspNetCore2 nuget has been deprecated
  • All support for ASP.Net Core < 2.0 and targets < netstandard2.0 have been eliminated
  • The latest Alba uses the ASP.Net Core TestServer under the covers. When I first released Alba I got plenty of “why didn’t you just use TestServer?”. So A.) now Alba does and that’s less code we have to support and much less weird incompatibilities to worry about, so yay! B.) Alba provides a lot of value and removes a lot of repetitive grunt work you’d have to deal with if you only used TestServer as is.
  • I added some syntactical sugar for common usages of “I only need to send and receive JSON to and from .Net objects” you can see in the new Getting Started tutorial

Do note that the advent of the Microsoft.AspNetCore.All metapackage makes it unfortunately likely that you’ll bump into assembly binding conflicts at runtime by consuming libraries like Alba. There are tips in the same Getting Started page for preventing or fixing these issues as they arise when you try to use Alba in your tests.

Show me a sample!

Hey, it’s silly, but let’s say you’ve got a Controller in your ASP.Net Core system that looks like this (with its related message DTO types):

    public enum OperationType
    {
        Add,
        Subtract,
        Multiply,
        Divide
    }
    
    public class OperationRequest
    {
        public OperationType Type { get; set; }
        public int One { get; set; }
        public int Two { get; set; }
    }

    public class OperationResult
    {
        public int Answer { get; set; }
        public string Method { get; set; }
    }
    
    
    public class MathController : Controller
    {
        [HttpGet("/math/add/{one}/{two}")]
        public OperationResult Add(int one, int two)
        {
            return new OperationResult
            {
                Answer = one + two
            };
        }

        [HttpPut("/math")]
        public OperationResult Put([FromBody]OperationRequest request)
        {
            switch (request.Type)
            {
                case OperationType.Add:
                    return new OperationResult{Answer = request.One + request.Two, Method = "PUT"};
                
                case OperationType.Multiply:
                    return new OperationResult{Answer = request.One * request.Two, Method = "PUT"};
                
                case OperationType.Subtract:
                    return new OperationResult{Answer = request.One - request.Two, Method = "PUT"};
                
                default:
                    throw new ArgumentOutOfRangeException(nameof(request.Type));
            }
        }
        
        [HttpPost("/math")]
        public OperationResult Post([FromBody]OperationRequest request)
        {
            switch (request.Type)
            {
                case OperationType.Add:
                    return new OperationResult{Answer = request.One + request.Two, Method = "POST"};
                    
                case OperationType.Multiply:
                    return new OperationResult{Answer = request.One * request.Two, Method = "POST"};
                
                case OperationType.Subtract:
                    return new OperationResult{Answer = request.One - request.Two, Method = "POST"};
                
                default:
                    throw new ArgumentOutOfRangeException(nameof(request.Type));
            }
        }

Inside of an xUnit.Net project for your web project, you can write a test for that controller that exercises the full HTTP stack of your application like so:

        [Fact]
        public async Task get_happy_path()
        {
            // SystemUnderTest is from Alba
            // The "Startup" type would be the Startup class from your
            // web application. 
            using (var system = SystemUnderTest.ForStartup<WebApp.Startup>())
            {
                // Issue a request, and check the results
                var result = await system
                    .GetAsJson<OperationResult>("/math/add/3/4");
                
                result.Answer.ShouldBe(7);
            }
        }

        [Fact]
        public async Task post_and_expect_response()
        {
            using (var system = SystemUnderTest.ForStartup<WebApp.Startup>())
            {
                var request = new OperationRequest
                {
                    Type = OperationType.Multiply,
                    One = 3,
                    Two = 4
                };

                var result = await system
                    .PostJson(request, "/math")
                    .Receive<OperationResult>();
                
                result.Answer.ShouldBe(12);
                result.Method.ShouldBe("POST");
            }
        }

        [Fact]
        public async Task put_and_expect_response()
        {
            using (var system = SystemUnderTest.ForStartup<WebApp.Startup>())
            {
                var request = new OperationRequest
                {
                    Type = OperationType.Subtract,
                    One = 3,
                    Two = 4
                };

                var result = await system
                    .PutJson(request, "/math")
                    .Receive<OperationResult>();
                
                result.Answer.ShouldBe(-1);
                result.Method.ShouldBe("PUT");
            }
        }

Notes from the CodeMash 2019 Automated Testing w/ .Net Workshop

I gave a workshop on Succeeding with Automated Testing with .Net yesterday at CodeMash 2019. First off, as usual for CodeMash, the crowd was great and all the interaction made the workshop go a lot easier for me. Thank you all for coming yesterday.

As promised, here are the notes from the session:

  • The slide deck and most of the samples are in this GitHub repository
  • Alba was the library we were using to test ASP.Net Core by running requests through our application in process. Not coincidentally, I made an Alba 3.0 release just this morning with some improvements and documentation updates to reflect the latest stuff I added for the workshop.
  • Storyteller was what I was using for the business facing acceptance tests.
  • Respawn was the tool I mentioned from Jimmy Bogard that helps you reset test state in relational databases between tests

 

Further reading  (by me):

 

 

Succeeding with Automated Testing in .Net at CodeMash 2019

If anyone is interested, I’m giving a half day workshop tomorrow afternoon (January 8th) at CodeMash 2019. The materials are up on GitHub (or at least the Powerpoint slide deck is for now, but more will be there by tomorrow;)).

In this workshop we’re gonna look at:

  1. Automated testing best practices — according to me anyway
  2. How the automated testing pyramid informs our testing approach
  3. Designing for Testability
  4. Dealing with Databases in Tests
  5. How Docker can be used to enable isolated testing environments
  6. HTTP contract testing with ASP.Net Core using Alba
  7. Random advice on how to express tests and craft assertions, mostly using Storyteller

I’m looking forward to catching up with old friends and making new ones this year at CodeMash. I hope to see some of you in the workshop tomorrow and all around the giant Kalahari.

 

 

Lamar 2.0 is Out! StructureMap’s Successor Grows up a bit

I was just able to publish Lamar 2.0 to Nuget and it looks like it’s indexed, so here we go!

Along with a whole lot of user reported bug fixes, the two big changes are:

  • We split the dynamic code compilation and code generation (the Frames model) into its own separate library called LamarCompiler so that interested folks could easily reuse that functionality without having the IoC support bundled along with it (even though you never needed to use the Lamar IoC code to take advantage of the code compilation but .Net devs are finicky about dependencies some times)
  • After a couple user requests for it, Lamar 2.0 supports optional and mandatory setter injection based on StructureMap’s setter injection support.\

Many thanks to all the folks that helped move this along by reporting GitHub issues and providing pretty damn useful reproduction steps. Special thanks to Mark Warpool for all his help in making Lamar go. Lamar, like Marten before it, has been a very positive experience for me in my dealings with the greater OSS community.

For more information, here’s:

 

What next for Lamar?

I don’t know when, but I’ll eventually get around to the “AutoFactory” feature, because I think that’s the single biggest blocker for many folks moving from StructureMap to Lamar. If LamarCompiler takes off at all, there’s worlds of opportunities to add new C# code constructs to the code generation model.

Jasper as an In-Memory Bus Part 1

I just finished the work in Jasper I’d described in Changing Jasper’s Direction and released a new Jasper 0.9 Nuget and a matching dotnet new template package with the newly streamlined structure. Hopefully, this represents the first big step on the march to a 1.0 release and production readiness by the end of this year. I’m going to start blogging a lot more examples on Jasper in hopes of building some community and possibly attracting some other contributors.

Today I want to talk about using Jasper as an in memory bus (like MediatR) within an ASP.Net Core application. To make Jasper less scary, let’s start with an ASP.Net Core application built with the dotnet new webapi template. After adding a dependency to Jasper via Nuget, you can add Jasper to your new application with this code:

    public class Program
    {
        public static void Main(string[] args)
        {
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .UseStartup()
                
                // This is all you have to do to use Jasper
                // with all its default behavior
                .UseJasper();
    }

Do note that this quietly replaces the default ASP.net Core DI container with Lamar that is absolutely required for Jasper usage. Also note that this is choosing to use Jasper with all its default behavior. Lastly, this simplified configuration approach will only discover message handlers within the same assembly with your main application. See this page for more information on configuring Jasper.

Now, let’s say we need to have an HTTP endpoint that will receive a command to create a new user within our system like this:

    public class CreateUser
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Email { get; set; }
    }

Since we’ve decided that we want the actual work to be done by a Jasper message handler instead of just burying it within the normal MVC controller cruft, let’s write our first Jasper message handler that will handle this message by just writing a message to the console:

    public class UserHandler
    {
        public void Handle(CreateUser user)
        {
            Console.WriteLine("Hey, I got a new user!");
        }
    }

Or to stay with using async anytime the handler involved any kind of IO, use an asynchronous version:

    public class UserHandler
    {
        public Task Handle(CreateUser user)
        {
            return Console.Out.WriteLineAsync("Hey, I got a user");
        }
    }

Now, switching to the MVC Core side, let’s add the controller action that receives a JSON POST for CreateUser and delegates to Jasper:

    public class UserController : ControllerBase
    {
        private readonly IMessageContext _messages;

        public UserController(IMessageContext messages)
        {
            _messages = messages;
        }

        [HttpPost("user")]
        public async Task CreateUser([FromBody] CreateUser user)
        {
            // Process the message NOW MISTER
            await _messages.Invoke(user);

            // OR

            // Put the incoming message in the in
            // memory worker queues and Jasper
            // will process it when it can
            await _messages.Enqueue(user);

            // OR, not sure why you'd want to do this, but say
            // you want this processed in 5 minutes
            await _messages.Schedule(user, 5.Seconds());

            return Ok();
        }
    }

The IMessageContext interface is the entry point for all messaging within Jasper and is automatically registered in your application’s Lamar container as ContainerScoped().

Alrighty then, it’s not super exciting in functionality, but this is actually a working application that hopefully shows off a decent getting started story for adding Jasper to an ASP.net Core system. In my next set of posts we’ll embellish the message handling to show off Jasper’s durable messaging, “outbox” support, and middleware.

In the meantime, please feel free to kick the tires on Jasper and let the community know what you think works and what needs some improvement in the Gitter room.

 

Jasper vs. MediatR

As strictly an in-memory messaging bus, Jasper supports a superset of MediatR’s functionality, but with a very different runtime architecture. In terms of implementation, MediatR is purposely simplistic and mostly relies on Inversion of Control (IoC) containers’ support for discovering and resolving generic types implementing its ​adapter interfaces likeIRequestHandler. Jasper, on the other hand, does its own command discovery (using Lamar’s built in type scanning) and eschews the usage of adapter interfaces or base classes in favor of naming conventions.

Selfishly speaking, MediatR has been a big headache to me with StructureMap and Lamar support because it encourages users to be too ambitious with generics to work around MediatR’s limited support for runtime composition. I certainly hope that Jasper’s middleware strategy — if it ever gets significant usage in the wild — generates far less user confusion.

As far as advantages, MediatR is obviously more mature, heavily used, and well understood by folks (as long as they stay sane on the generics usage). Plus some folks will prefer MediatR’s more explicit interfaces versus Jasper’s usage of conventions. In my opinion, Jasper’s potential advantages are:

  • Cleaner code because Jasper adapts itself to your code rather than forcing you to use its constructs
  • Higher performance, especially when it’s idiomatic Jasper HTTP handlers versus the combination of ASP.Net Core controller actions + MediatR.
  • Jasper supports quite a bit more functionality, and gives you much more control over message priority, error handling, and durability
  • Jasper composition and middleware design allows users to intelligently order middleware and provides many more extensibility scenarios
  • It’s rudimentary right now (just dump out the generated code), but Jasper provides out of the box diagnostics to show you what middleware is applied for which message types

Talking Integration Testing w/ .Net at CodeMash 2019

I’m officially approved by all pertinent bosses and spouses to go, so I’ll be speaking at CodeMash 2019 (think this is the 6th time for me). I’m doing a pre-compiler workshop this time around that’s going to be all new material. If you’re reading this and it’s not already January, feel very, very free to make requests or suggestions on scenarios you’d be interested in seeing as part of the workshop;)

Successful Integration Testing in .Net

Description

In the very early days of Agile development and Test Driven Development, we strove to structure our code such that we could write isolated unit tests with little or no coupling to infrastructure. Great, and it helped, but frequently that led to more complicated code internals and you still had issues from interactions with said infrastructure. What if instead, we could more effectively do quick, reliable integration testing? Could that simplify some of our code — especially when combined with microservice architectures? Do we need to reconsider where our efforts go in the old “testing pyramid?” How does ASP.Net Core support far better server side testing than previous versions? Where does HTTP contract testing fit in? Can Docker help us out? Should we pursue other techniques besides just trying to use Selenium? What about using alternative NoSQL style databases? Or if I’m stuck with relational databases, what can we do to soften the aggravations?

I’ll try to address all these questions and tackle automated testing best practices in this workshop.