May 27, 2008 at 5:58 PM
I think we need to spend some serious time getting unit testing established. I'm starting to feel pretty guilty about the work I've been doing untested!

Should it be a priority for this release? Probably not, but I definitely think we need to get a more test driven approach for future development. The tests will become increasingly more helpful as we start delving into macros and what not.

What do we need to test?

Syntax highlighting
I haven't really touched our implementation of this, you got any thoughts on this Jeff? Would it be possible to create a wrapper for the highlighting code that we can push a document to and analyse the returned tokens?

Again, this could probably work by wrapping it and passing it a document with a faked caret position, and analyse the members it returns. Easy enough for single documents, but it'll get tricky when involving references. We can probably create our own proxies to the VS automation objects that we could stub out.

Everything else
I'd like to see as much as possible after 0.2 being unit tested, is this achievable?

May 27, 2008 at 8:21 PM
ive got some tests written already for the syntax highlighting.. but they definately need work.. and of course there is the issue of what we can test, feasibly, in the confines of the environment... mostly this comes down to me testing ScanTokenAndProvideInfoAboutIt() over and over for a line and examining output for desired behavior...

... of course it's not perfect because you can't test the fact that the IDE runs through lines in a non-linear fashion in the background thread (which is why the above-mentioned method has a state param.. some kind of magic memory thing going on so that if it encounters a block in a multi-line comment out-of-band down the road, it'll remember that it's in a ML_COMMENT region, etc)

That being said, YES... we definately need tests...

can you discreetly test the various ways to get intellisense via the ParseSource method (ie test calling into it with various parserequests (which should be where you can stuff in the source code to be parsed...) .. that was the approach that I thought to take.. am i too far off-track, here?

YES to more tests. Even more YES to if you can write the tests first, before implementation (which I, personally, haven't had a ton of success with, having spent most of my time feeling-out the API for all the crap we're wiring together)
May 27, 2008 at 8:23 PM
also, a quick note on the test structure, if you haven't already scoped it:

all of the current tests are housed in BooLangServiceSpecs (one project for the whole solution to keep project bloat down.. we can organize by functional component via folders, if you'd like?)

And the tests mostly follow a When->Should format, ie:

WhenParsingMultiLineCommentsWithInLineQuotes (test fixture/context)
DoubleQuoteStringsShouldBeParsedAsComments (test method)

does that make sense? It's sorta bdd'ish.. and I like it because it doesn't exactly limit me to testing a single class' functionality, but more too limiting chunks of functionality...

I'm curious what the thoughts of others are on this approach.
May 27, 2008 at 9:42 PM
I think I need to have a sit down and play with the parser again, I haven't done it since I started my project. I'm thinking it should be fairly simple to test single line highlighting. Multiline stuff will be tricker though... I guess it's a case of trying until we get it right.

Regarding intellisense, I think you've pretty much nailed how I was thinking of doing it. The ParseSource method takes a ParseRequest which tells it whether the user is doing completion or method suggestion, so there's two groups of tests. Then I think I'll just be able to pass it various chunks of source, setting the line and column of the caret.

I'd like to get a test first practice going, but I don't know how likely that is currently. I find it very hard to maintain when I'm constantly trying to figure out how to do things. Hopefully as we get more accustomed to the SDK we'll be able to take a more test first approach. My biggest problem with my current process is adding tests now feels like a chore, because it's done after. I get a feeling an evening writing tests for the night before's code is an evening I could've spent writing more features.

The structure for the tests suits me fine. It's the first time I've written tests in that format, but I have no problem with it. In the end, a test is a test, and I've always been a bit uncomfortable with the FooBarTests format.
May 29, 2008 at 7:55 PM
hi, does anyone have any objections to me switching the project's test framework dep to xunit.net?

I got a chance to talk to the creators of it last weekend and they made a pretty good case for the usefulness of it (use language features for stuff that other test frameworks are doing via attributes, easy extensibility ... not to mention Assert.Throws() )

Ultimately, I want to write the tests in boo/specter (much more compact), but for now I'm doing it in c#, so yeah...

So I'm gonna go ahead and make these changes in the syntax_cleanup branch I have going.. and if you guys are opposed to the change, i will omit them from when i merge back into master...

May 30, 2008 at 6:57 AM
I have had some experience using xunit.net and think it is great, the extensibility options and the available data extension are great, as are the .net 3.5 method extensions.

What about mocking? If we need a mocking framework I would suggest Rhino Mocks or Moq :)

May 30, 2008 at 7:28 AM
I have no problem with xunit.net, and seconding torkelo I was going to run including Rhino Mocks by everyone.