 |
|
|
What Price Integrity How useful are independent test results? You may think this is a strange question coming from someone who makes a living from performing such tests. Obviously I think that they are extremely valuable�– when performed correctly. Rarely does today’s IT Manager have the time or resources to perform large-scale software and hardware tests on the products he has short listed for purchase. How many of you have a “clean” lab environment with rack upon rack full of networking hardware and identical PC’s upon which you can install, uninstall and reinstall software, or to which you can attach hardware devices for testing? How many of you have state-of-the-art benchmarking tools available that can allow you to accurately and repeatedly stress-test a network or a single device to determine performance and scalability? When I was an IT Manager we were lucky if we had a spare machine in the corner for testing. If we did, it was usually a couple of generations old. Every single purchase had to be fought for long and hard, and a test lab was a luxury that could only be dreamed of. So I used to rely heavily on independent reports from magazines or test labs in order to help me in my short-listing and purchasing decisions. I suspect many of you are in the same boat. That is why I get so annoyed when I see conflicting results coming out of the same test organisation. Some months ago, KeyLabs – a Utah-based independent test lab – was the subject of some unwelcome publicity when certain parties voiced the opinion that some tests it had performed comparing NetWare and NT Server were biased in NT’s favour. Much back-peddling was in evidence following this. You would think that KeyLabs would be doubly careful the next time it had some results to announce. Strange therefore that I received two press releases in the same week, one from Novell and one from Microsoft. Both were announcing the results of independent comparisons by KeyLabs between NDS eDirectory and Active Directory. Strange, why should both vendors feel they have something to announce? Quite simply because KeyLabs had stated in Microsoft’s report that AD was better than NDS, and in Novell’s report that NDS was better than AD. Novell announced that KeyLabs “found
that NDS eDirectory outperformed both Microsoft Active Directory and
iPlanet (formerly Netscape) Directory Server in real-world environments,
simulating common directory loads and search requests” (see the full
press release at www.novell.com/press/archive/2000/02/pr00026.html).
Almost exactly the same time, however, Microsoft issued a press release
that stated “Active Directory Delivers Leading LDAP Performance -
Tests by KeyLabs Show Active Directory Is as Much as Seven Times Faster
Than Novell's NDS Version 8 for Common Internet Operations” (see the
full press release at www.microsoft.com/PressPass/press/ Even
more strange is the fact that the Microsoft-commissioned report remains on
the KeyLabs Web site at www.keylabs.com/results/microsoft/ad/index.htm,� Delving
into the two reports is becomes obvious that the same set of tests were
not run in the same manner on AD and NDS. Without wanting to pass
judgement either way, I suggest you take a look at Microsoft’s response
to the situation at www.microsoft.com/ When trying to prove certain results, vendors can often put undue pressure on test labs to run tests in a certain way or to highlight certain results above others. Surely it would be better to walk away from such projects? Surely in this case it would have been better to have a true head-to-head with both vendors participating equally in the same set of tests? Did the two vendors know that the other had commissioned similar tests from the same lab? If so, what did they hope to gain from it? When you pit products against each other like that you are bound to come out with an overall winner in performance terms – somehow that failed to happen on this occasion. That is not always the most important issue, of course, and a good test report will try to place each product in the context of where it would be best deployed. Sometimes lower performance can be acceptable to an organisation for whom other features (i.e. scalability or security) are more important. These are the sort of points that can only be raised in a true head-to-head comparison. The launch of Windows 2000 is one of the most important events of the year. IT Managers the world over are faced with a difficult choice – do they implement Active Directory or not? Clearly Novell is going to do its best to muddy the waters here, but it is hard to condone the release of two apparently completely contradictory reports by the same organisation.
|
|
Send mail to [email protected] with
|
