Site Monitors

Back to:  VRC Categories  |  VRC Tool Box

Change Detection Icon

View tools in the Site Monitors category

Site Monitors Category Description

Introduction

The most fundamental function of any web site monitor is to quickly notify the party responsible for maintenance of a web site that an outage is in progress. The purpose, of course, is to minimize the amount of time the site is unavailable through an automated process that doesn't require the site's maintainers to constantly check its status.

Web site monitors seem to have sprung up in concert with the growth of the Web as a vehicle for online commerce. The dotcom boom of the late nineties provided consumers with abundant choices for online purchases. Vendors recognized that potential customers might not wait around for a down site to return when there were alternatives available. (One of the monitoring services we found is even called Dotcom-Monitor.)

In its simplest form, a web site monitor will check a specific URL (usually a site's home page) at least once an hour and record the site's availability status in a log. If an anomaly occurs, a contact person at the site is notified right away via email and/or pager. The checking is typically, though not always, done from a remote site, so the monitoring doesn't rely on the equipment being monitored, or even the same network.

Availability testing may be done using ping (which only verifies that the server computer is running and reachable on the network) or, more commonly, using an http/https HEAD or GET command, which is more specific to the availability of the web site itself. A large array of other testing may be offered along with web site monitoring, including keyword detection (a limited form of change detection designed to warn of unexpected changes to a site), performance testing (indicating not just when a site is down, but when it responding unusually slowly, possibly due to a denial of service attack), and link checking, a web site maintenance function often offered as a standalone product or service.

Site Monitors for VRC

An e-commerce site that is down for a day may spell disaster for its operator. Thus, the frequent checking (usually no less than once per hour) and potential for false alarms (a site may be unreachable for reasons that have no connections to and thus no implications for the safety of its content) can be tolerated. But for VRC purposes, unless a site is unreachable because the owners have decided to pull the plug without warning, any particular outage is likely to have little meaning. VRC is designed to monitor a valued site over time, with the hope of detecting signs of declining maintenance that might endanger the site's content, before any real loss occurs. In this regard, a pattern of outages over time may reveal a breakdown in maintenance, making these types of tools most useful in the strategy and detection VRC stages.

However, a consideration for the use of web site monitors in VRC practice is that unavailability of a site is not a subtle problem. It doesn't require a specialized tool to detect, since any site maintenance tool needs to be able to contact the site before doing its job and will report its inability to do so. In that sense, a separate web site monitor may be both overkill and redundant.

Many web site monitors have another limitation that makes many of them unsuitable for VRC. They come with a range of restrictions prohibiting their use on any site not owned or controlled by the product's purchaser. Here are some excerpts from the terms and conditions of several products and services, from most to least restrictive:

Clearly, many of these restrictions would prohibit use of the tools for the lowest level of passive VRC monitoring. About two thirds of those we looked at made specific mention of limitations in the use of the application or service on someone else's web site.

Another sign of the general assumption that this category of products was designed for use on one's own sites is the complete absence of and discussion of robots.txt exclusions and whether the protocols will be honored.

VRC should be designed to achieve its goals with a minimum of impact on both the monitor and the monitored. Some products in this category even include warnings against excessive load on others' servers, for example "you must not cause an unreasonable or unwanted load on any third party systems" and "you must not use Seller's services in a way that causes unreasonable load on Seller's systems or unwanted load on systems with which you direct Seller's services to interact."

Site Monitor Features

Perhaps more so than any other category, web site monitors tend to be offered as services rather than products. Two thirds of those we considered are marketed as subscription services. Service offerings in this category have the advantage of requiring no software installation and being managed from a simple browser interface, Monitoring is done from the vendor's computers, so an outage on the subscriber's computer or network won't cause the monitoring to fail. However, the monitoring services don't come cheaply. Though a few offer rudimentary monitoring of a single URL for free (sometimes as a limited time trial, sometimes in exchange for agreeing to accept advertising contained in newsletters or messages), most full-fledged services charge monthly, with costs escalating rapidly as the number of URLs to monitor or protocols to test increases. More frequent monitoring also drives up costs dramatically, as do more sophisticated means of notification. Prices can range from as little as $5/month to do http monitoring of a single URL once an hour to hundreds of dollars monthly to monitor a handful of sites via ping, http, and https every five minutes.

Web site monitoring applications are usually offered in different versions, with the lowest cost version (typically around $50-$75) limited to monitoring a fixed, small number of sites. Products with unlimited monitoring capability are more typically around $150.

Judging by the web sites for the products in this category, most of them are fairly well-maintained. Even the products coming from small outfits have seen regular updates and bug fixes. Most of the applications we saw in this category are offered by companies located outside of the United States (e.g. Canada, Australia, New Zealand and Great Britain). Those offered as services can be used on any computing platform that supports a web browser. However, all the applications we saw were designed for the Windows environment.

Basic features of web site monitors:

More advanced features of web site monitors:

Tool Evaluation Selection Process

Web site monitors are widely available. Lengthy listings and some product reviews are available on the web (see the resources listing). It was impossible for us to evaluate every product in this category, but we did at least look at the Web sites for 27 of them. We chose for initial evaluation, products that offered a wide range of features, both services and applications, and those restricted to the subscriber's sites as well as permissible to use on third party sites. Ultimately, we settled on the services Wired-2-Watch, and Page Patrol, and the applications Site Vigil, Atomic Watch, SPIS Webwatch and Are You There? In most cases we purchased the software or subscribed to the service, but in some instances we used trial versions or trial subscriptions.