Testing Protocol
Back to: VRC Approach | VRC Tool Box
Specific site selection (content-oriented tool categories)
Every content-oriented tool should be tested against the VRC control site.
Other "local" sites that have been suggested for testing include:
—DPM tutorial
—Digital Imaging tutorial
—VRC site
In some cases, randomly selected sites may suffice. Other potential test sites might include: Fortune 500, emerging technology businesses, computer businesses, state libraries and government agencies.
Specific site selection (event-oriented tool categories)
By their nature, these tools are designed to be tested against a limited number of sites, and a limited number of pages on each site (one to five, perhaps).
Local sites will be selected based on access to advanced notice of changes that will be taking place, for example, the IRIS site.
Specific testing procedures (content-oriented tool categories)
Review settings. Make sure the tool is set to be as unobtrusive, well-behaved, and resource-sparing (i.e. "nice") as possible.
Turn on all reporting capability; capture/log as much activity as possible. Set up a directory to save all output files.
Most other settings should be set to defaults, unless defaults are inappropriate (e.g. the default is to test only internal links, or to visit only the first 10 pages, etc.) Focus on features that have relevance to VRC.
Start a testing log. Include the tool name, site name, start time, end time, relevant settings, etc. for each test run.
Test against the control site.
Review the transaction logs (on irisresearch) of your test against the control site to make sure the tool was behaving as nicely as you thought. Most important is that it wasn't requesting pages at too rapid a rate.
Review other results from the control site. Are they reasonable? For example,
are the page counts, link counts, file sizes, etc. correct or at least in
the right ballpark?
—Are the control site results complete and accurate?
—Take a look at any reports or logs. Do they completely
and accurately record known errors?
—Record any anomalies that occur during testing, such
as premature termination, crashes, etc.
—If performance of the tool is an issue, try to minimize
variations in the test environment that might impact performance, such as
the speed of the machine being used, or the time of day testing is done.
—If the tool's testing against the control site is satisfactory,
it can be tested against other sites. It is probably best to start with
other local sites.
—If all seems well, continue testing against selected
external sites
—If time allows, do repeat tests on some sites after a
period of at least several days have elapsed.
Specific testing procedures (event-oriented tool categories)
Events being monitored (e.g. server outages) will have to be simulated.
Particular caution is needed in testing external sites, since event-based tools typically monitor a site closely and regularly.
Change-detection requires change to occur for there to be output, so use of the control site for that purpose will require changes to be made. Since content-oriented tools are best tested against a static control site, a second version of the site that can be changed at will is needed.
Report of Testing Results
Discuss specific test results from each tool. Summarize issues (note: some
of these should already be in the tool inventory):
—ease of learning (utility of documentation, tutorials,
help files, etc.)
—ease of use
—limitations encountered
—anomalies encountered
—accuracy of completeness
—strengths and weaknesses
—scalability
—your evaluation and recommendation for this tool
Conclusions
Describe the overall utility of this tool category for VRC purposes, based
on test results.
—Does it measure up to expectations?
—Does it provide potentially useful information for risk
management purposes?
—Does it have potential, but require certain changes to
be made?
—What are the prospects for integrating this kind of tool
into a toolkit?
Can the reports produced by this tool category be used to create a comprehensive overview of a site's risk profile when combined with reports from other tool categories?
Where in the VRC spectrum (from passive monitoring to full-scale agreement) might this tool category have relevance?
