Final Thoughts on Nuget and Some Initial Impressions on the new KVM

There’s an index now for the FubuMVC Lessons Learned series of blogposts. I fully intend to keep going with more content in this series some day, but this post is going to wrap up my thoughts on DevOps kind of topics like Nuget, Ripple, continuous integration, multi-repository development, and build management. If you read this and say why is Jeremy being so negative about Nuget?, I’m writing this from three different perspectives:

1. As a consumer of Nuget for complicated dependency trees
2. As an author of value added tooling (Ripple) on top of Nuget
3. As just an armchair software architect who enjoys looking at tools like Nuget and thinking about how I’d build that code differently if it was mine

Some of the recommendations I’m making here are either already on the Nuget team’s stated vNext roadmap or something I know they’ve thought about themselves.

My Thoughts on Project K

I’ve been piddling around with the content on this post for so long that in the meantime Microsoft finally publicly announced their Project K work (ASP.Net vNext) to the unwashed masses who aren’t some sort of ASP Insider or MVP. While I’m mostly positive about their general direction with Project K, I really wish the ASP.Net team had been much more transparent with their technical direction. Since most of Project K feels like catching up with other development platforms more than anything original or innovative, I don’t know what they buy for themselves by being so opaque.

The Project K runtime looks like it would have improved the FubuMVC team’s development experience over the current .Net framework and tooling:

• The K runtime does not include strong naming, which was an almost unending source of unnecessary aggravation for me over the past 3 years or so. Bravo.
• It looks like the K tooling has some level of support for our Ripple tool’s floating dependency concept.
• Eliminating the csproj files should go a long way toward reducing the csproj merge hell problem, but I bet that the project.json file still ends up being the worst source of merge conflicts even so. Hopefully they carefully consider how they’re going to integrate Nuget into K development to avoid so much of the friction we had before we wrote Ripple.
• Again, by eliminating the heavyweight csproj file baggage, it’s going to be much easier to write project and item generation tools like “rails new” or our own “fubu new.” I had to spend an inordinate amount of time last year building out our FubuCsProjFile library for csproj/sln file manipulation and project templating.
• I’m still going to claim that FubuMVC with Bottles had the best modularity strategy of any .Net web framework, but much of what we did would probably be unnecessary or at least much easier if I’m correctly understanding what the Roslyn compiler is capable of. If it’s really more efficient to forgo distributing assemblies in favor of just letting Roslyn build everything into memory, then all that work we did to smuggle web content through assemblies for “feature bottles” is now unnecessary.
• In my last post I talked about the auto-reloading/auto-refreshing development web server we built for FubuMVC development. While it sounds like it’s not usable yet, the ASP.Net team looks to be building something similar, but using the Roslyn compiler to recompile on code file changes should be a faster feedback loop than we have with our “fubu run” tool. I’ll be interested to see if they can create an experience comparable to Golang or Node.js in this regard.

Overall, I think that Microsoft has probably rode the Visual Studio.Net horse for too long.  Project K starts the work of making .Net development much more productive with lighter weight tools and better command line friendliness that can only help the community over time.

I’ve been asked a couple times on Twitter if I would consider restarting FubuMVC work after the Project K runtime is usable. My answer is an emphatic no, and moreover, I would have ditched many of our technical efforts around FubuMVC much earlier if I’d known about Project K earlier. If anything, I’d like to start over from scratch when Project K stabilizes a bit, but this time with much less ambitious goals.

Suggestions for Nuget in .Net Classic

• Optionally remove strong naming during package restore. Who knows how long it’ll take to move all server side development to the new K runtime. In the meantime, strong naming is still a borderline disaster. Maybe the Nuget team should consider a function where the Nuget package restore functionality could either strip out all the strong naming of the downloaded nugets or strong name nuget assemblies on the  fly for hosting technologies that require naming. We discussed this functionality quite a bit for Ripple but it’s never gotten done.
• Make the Nuget server API less chatty and ditch oData. I’m not the world’s foremost expert on software performance, but the very first thing I learned about the subject was to minimize the number of network round trips in your software. The Nuget client makes far too many network round trips because it seems to be treating every single dependency as a separate workflow instead of treating the entire dependency graph as one logical operation. I strongly recommend (and I’ve said this to them in person) that they add documented API alternatives where you can batch up your requirements in one request — and I think they need to go beyond oData to get this done.
• Make the Nuget server API a standard. The Nuget server API wasn’t great to work with. We usually had to resort to using Fiddler to figure out what the built in clients were doing in order to accomplish things with Ripple. A packaged assembly to be a client to the Nuget server API would be a great way to promote value added tools on top of Nuget.
• Decouple Nuget.Core from Visual Studio.Net. This is a big one. I think it was a huge mistake to depend on Visual Studio automation to modify the csproj files to make the assembly references. We first used crude Xml manipulation then later the FubuCsProjFile to manipulate csproj files in Ripple. Since we had no hard coupling to Visual Studio, we were able to provide valuable command line support for installing and updating Nuget packages that enabled our continuation integration across repositories approach.
• Better Command Line Support. Just like Ripple, you should should be able to install, update, and remove nuget dependencies from the projects in your repository without Visual Studio being involved at all.
• Modularize Nuget.Core. We found the Nuget.Core codebase to be poorly factored. Ripple would have been much easier to build if the Nuget.Core code had been reasonably modular. At a minimum, I’d recommend splitting the Nuget.Core code up in such a way that you could easily reuse the logic for applying a Nuget package to a csproj file all by itself. I’d again recommend pulling out the interaction with the Nuget server API’s. Making Nuget.Core more modular would open up opportunities for community built additions to the Nuget ecosystem.
• Support Private Dependencies (Somehow). Several folks have mentioned to me on Twitter how Node.js’s NPM is able to isolate the dependencies of your dependencies.   Maybe Nuget could support something like ilrepack to inline assemblies that your dependencies depend on but your application itself doesn’t need otherwise.
• Ripple Fix. Maybe it’s better now, but what we found in the early days of Nuget is that things could easily go off the rails. One of the features I’m most proud of in Ripple was our “ripple fix” command. What this did was check the declared dependencies of every single project in your repository and make everything right. If assembly references were missing for some reason, add them. If a dependency is missing, go install it. Whatever it takes, just make things work. And no, it wasn’t perfect but it still made using Nuget more reliable for us.
• (Possibly) Adopt the Actor Model for Nuget Internals. I think that in order for batched Nuget operations across a complicated tree more efficient, Nuget needs to do a far better job of using parallelization to improve performance. Unfortunately, all the operations you do (find the latest of package A, download package B, etc.) are interrelated. The current architecture of Ripple is roughly an old-fashioned pipes and filters architecture. We’ve long considered using some kind of Actor model to better coordinate all the parallel actions and latch the code from performing unnecessary work.