Research company Evans Data sent me a link this morning to its new Tool Grader service. This is a simple web application for reviewing and rating software tools. The same tool may rated separately rated for different platforms. For example, there is one entry for Eclipse under UNIX/Linux, and another separate one under Tools for Mobile.
I took a quick look and rate the site mostly useless. There are not many reviews, and most of the reviews are of little value, for example “This Is The Best Programming Tool i Have Ever Used,” from somebody who says that Eclipse “Must be used as a competitor for Java.”
The site would improve of course if a lot of people were to use it; but currently there is little incentive to do so, since most developers will take one look and never return. Evans Data could do better; it has a ton of data from surveys it has conducted and if it were to take some of the more useful data from those reports and integrate it with the Tool Grader the site would be more valuable. It will not do that I guess because its business model is to sell those reports, and because it would be a lot of work.
It gave me pause for thought though. What is the best way to choose a development tool? Part of the problem is that context is everything. The same tool will be great for one purpose and poor for another; it depends what you want it for, especially when it is a multi-faceted product like Eclipse or Visual Studio, both of which are really tool platforms.
If you are looking for information on which tool will be best for your project, I doubt that either Tool Grader or even purchasing an expensive report will help you much. One approach that has value is to install several candidates and try them out, but it takes considerable time and effort. Another idea is to go along to an active community like Stack Overflow, describe your project in some detail including any constraints like “our developers span three continents” or “the boss insists we use Rational ClearCase for source code management”, and ask for opinions from other users.
When I am assessing a tool I always try to visit forums where it is discussed and get a flavour of the types of problems and queries users have. If there is little discussion that suggests the tool is most likely little used, usually a bad thing. If the vendor has no open discussion on its site and emphasises the “contact support” route that suggests a weak community. I also look for potential showstoppers like instability or intractable problems such as difficulty wresting acceptable performance from either the tool or its output.
I do not pretend it is easy though. Tool choices are important because they have a significant impact on productivity, and it is hard to change your mind once you and your team have invested money, skills and code in a particular product.