I just started playing with MbUnit, or at least that’s what I had in mind — found out then that it comes in that Gallio package these days, with all sorts of stuff on board that doesn’t all have a clear purpose. Clear to me, that is. Well, I think at this point that I would prefer to have a simple package comparable to NUnit, but maybe when I see what’s in there, I’ll change my mind.
Anyway, so far, so good — I took a small test project with five tests in it, which was working with NUnit so far. Threw out the NUnit references and added MbUnit ones instead. I went for MbUnit as well as MbUnit35, since the project uses .NET 3.5, and I found I also had to add Gallio in order for everything to build. Then I ran my tests using the CodeRush Test Runner, and everything behaved exactly like before. Brilliant! All five tests run through in about 1.6 seconds, for future reference — roughly the same as I got with NUnit.
For some reason I had the idea to look around a bit, and I ran one of my tests through TestDriven.NET, which I’ve also still got installed (I mean in addition to the CodeRush thing, which I really use most of the time these days). And there’s a surprise: it took almost exactly 60 seconds (!) to run the one single test!
The process had two stages, which seemed to take roughly an equal amount of time. Stage one is up to the point where the Test tool window in VS says “Initializing the test runner. … Running the tests”, and it took about 33 seconds. The stage after that is when the test actually runs, and it took about 27 seconds. Wow. Now, stage 1 must have had something to do with first-run initialization, because on subsequent runs the time of stage 1 is now down to less than a second. The time for stage 2, however, is still the same as before, comes up with about 27-28 seconds every time I try it.
Obviously I was glad to have the CodeRush runner, but then I thought it might be interesting to see the standard runner that’s included in the package. It’s called Icarus, and here’s surprise number 2: that thing is equally slow! I’ve now started it a few times, so I guess it’s now faster than it was in the beginning. I just did a rough test, and it takes almost 30 seconds for the UI of Icarus to even show up on screen. During that time, I can see in task manager that a process called Gallio.Util.exe (or is it Utility?) pops up many times and goes away again – no idea what the … thing… is going on there. Well, when the UI finally comes up, a progress bar dialog appears, telling me that the program is now exploring my tests. It moves to 49% pretty quickly, then hangs around for >20 seconds before finishing after an overall 25 seconds or so. Phew.
This is the same 5 test assembly I was using before. So, final test: run everything. The progress bar at the bottom of the window comes up to 53% very quickly, then stops. After a good while it continues and makes its way up to the top. Overall outcome: rather close to those 27-28 seconds I got from TestDriven.
Okay, this is the weirdest thing I can imagine… I don’t suppose there’s anything intentional about all this, but I can’t imagine what the problem is either. Why do those delays happen? How does CodeRush manage to run the tests without the delays, something that neither the much more mature TestDriven nor the standard runner can do? Very odd. I might go hunt around for the right place to report a bug on this etc etc. But who knows.
Update: I added a bug report here
Another update: I just notice that running the Gallio Control Panel app shows the same startup behavior as Icarus — takes forever, with that Gallio.Utility.exe thing popping up probably ten times or so. What sort of a weird application architecture is this? Hm…
Whoa: Sorry for all the updating 🙂 But this is interesting… I thought I’d see what the Utility thingy even did, so I ran it just like that. Comes up with a console window, and, well…. does absolutely nothing for a really long time, perhaps something like those 30 seconds long. I had enough time to read the helpful output in the console and think about it, before the window suddenly closed because the program was finally done doing nothing. I thought it was waiting for a keystroke by then 🙂
In case somebody’s interested — I’m running in a VM with 2GB RAM, 2 cores available to the VM exclusively. Windows 7, VS 2008 and 2010 RC installed, working in 2008.