Backup is one of the thornier issues facing the network administrator, but it is also one of those most likely to find itself at the bottom of the �to do� pile. Let�s face it - backup is not particularly glamorous at the side of investigating Gigabit Ethernet or load balancing technologies. Surely it is more important to keep our network running smoothly and at peak performance? 

The lure of the user�s accolade is also very strong. Improve the response time to the workstation and you are an instant hero. Improve the chances of recovering a file after a disaster, and all you are likely to get is �Why is the response time so poor?� After all, a disaster is something which only happens to someone else .... isn�t it?

So why spend lots of money on a boring old backup system? Cheap tape devices can be picked up from most mail order PC warehouses these days, and the operating system comes with a backup utility for free, doesn�t it? 

There is far more to backup than simply stringing together a PC, a tape drive and a piece of software, however. Although low-end tape devices have become commodity items, it is worth putting some thought into the requirements of your particular situation before spending money on something which may not satisfy your needs in the long term.

Backup software too has become extremely sophisticated and feature rich, and the utilities included �free� with your operating system offer little more than a �taster� of what you can expect from a full-blown package. If you truly value your data, the backup system you choose should offer the following features :

• Backup must be done within a narrow, predictable, inflexible window. The backup window cannot vary with the amount of file system activity, and is often non-existent in today�s 24x7 networks. 
  
• Backup must be done at maximum speed. 
  
• Backup must be completed even in the event of a cartridge or tape drive failure. 
  
• Restore must be accomplished at maximum speed. 
  
• Restore must be accomplished even if a cartridge is destroyed, mislabelled or missing from the original backup set. 
  
• Restore must allow individual file or mailbox restoration. 
  
• Simple backup management paradigm. 

For many organisations, the backup application serves as a cornerstone to an expanding data management strategy that involves archival and retrieval. It used to be that backup and secondary storage strategies were driven by the relatively high cost of disk - now they are driven by the sheer amount of data and its thousands of distribution points across multi-server networks.

That is why we have entitled this document �Data Storage Management�. This is not just a fancy term for �backup�, since in order to implement a really effective backup policy you need to be on top of your data management. Consider what would happen, for example, if you were to attempt to backup your 320GB server-based RAID array to a single DAT drive. Even with the best compression technology in the world, you are not going to fit this amount of data on a single tape. 

The result is a partially complete backup job, with the console requesting another tape when you return to work in the morning. What do you do then? Put the second tape in and keep your users off the system for the rest of the morning until the job is complete? Or abort the job and run the risk of operating with unprotected data?

Hopefully we are about to dispel many of the myths and explain some of the jargon surrounding the storage management industry. The first section of this document looks at the backup issues facing most administrators today - mainly centred around too much data and too little time. 

What features should you look for in a backup package - file by file or image backup; push agents; automated tape rotation; directory services support; open file support? And how can you best design your network to take advantage of the limited backup window you have available? We also examine the various standards applicable to the backup industry, and look at how you might go about implementing a comprehensive backup regime.

The second section covers some of the more advanced features that can be found in today�s high-end backup solutions. Anti-virus scanning, centralised management, fault tolerance, and for those of you already experiencing the trauma of a disk subsystem which is constantly creaking at the seams, we look at what data migration and Storage Area Networking (SAN) can offer you. Although these features are often perceived as too expensive for anything but the largest corporations, this is simply not the case with many of today�s systems. 

Finally, we explore the world of backup devices, sorting through the bewildering array of offerings and attempting to offer some guidance as to which might be the most suitable for your particular application.

The choices out there are bewildering and there are many pitfalls to trap the unwary on the way to implementing a comprehensive data storage management strategy. Disk grooming, data migration, SAN, NAS, tape arrays, autoloaders - these are all technologies which can help to keep your data under control, and each of them will be examined as part of this document.

Many thousands of businesses still do not operate an effective data security system. As the size of hard disks grow and prices come down, many people take the view that backing up is either extremely time consuming or extremely expensive, and so they take a risk. But as more and more critical data is stored on these massive hard drives, the cost of replacing such data will make the cost of the most sophisticated backup system pale in comparison.

Just sit back for a moment and try to imagine what the effect would be should your main file server go up in smoke today! Replacing the hardware would be the least of your problems - any reputable supplier or maintenance company should be able to help you get physically up and running within a few hours. But where would you start when faced with the task of replacing your data? 

How long would the system be unavailable while you were attempting the salvage operation? What effect would this have on your daily operations? 

And, horror of horrors, what if you simply could not retrieve the data at all? Do you keep manual records of your debtors? How would you progress orders and get the cash in? How would you pay your employees?

The list of questions goes on and on, and gets ever more horrifying as it does. The bottom line is that, assuming your company doesn't actually go out of business as a result of this disaster, the final cost in terms of time, temporary staff to cope with the increased manual workload and, possibly most important of all, lost orders and customer confidence, is far greater than the cost of implementing the most comprehensive backup solution.

Of course, total catastrophes are rare. But personal catastrophes can be almost as devastating. The plight of the secretary who wants to retrieve a missing file from a month ago might not seem too important, but if she is the chairman�s PA you could be in big trouble. 

The thing is, of course, that backup is pretty low down on the priority list of most users too. Quite simply, they expect you as the network administrator to be able to lay your hands on just about every copy of every file they have ever written - even if those files are stored on their local hard disk.

The explosion of networked data - ranging from traditional LAN storage (corporate documents, presentations, etc.) to distributed databases (e.g., inventory, financial, or personnel-related data) to application-specific data stores (such as e-mail message stores, directory services repositories, or internal Web sites) - has resulted in increasing storage demands and thus storage-related costs.  

The META Group projects LAN storage will increase at 40 per cent or more annually, and the massive growth of distributed databases will at least keep pace with file system data growth over the next several years.  However, this information resides and continues to be deployed on disparate systems, creating additional difficulties in managing enterprise data from a centralised location.

With the process of storage management encompassing so many disciplines (i.e., backup, recovery, data migration, tape management, and archiving), IT administrators are finding themselves devoting an increasing (and sometimes inordinate) amount of time to managing this data. Scope of duties may range from labelling and tracking tapes to restoring a lost file or directory to rebuilding entire servers. 

It is also becoming increasingly difficult to pinpoint the �owner� of a particular set of data in an organisation. That pile of documents and spreadsheets sitting over there in the corner of the disk is probably the responsibility of the LAN administrator. But what about the payroll and HR databases? They may belong to the database administrator. And what about the e-commerce system - a mixture of files and databases? Who takes responsibility for that one?

Trying to accurately assess the true cost of rebuilding lost data is also extremely difficult. You have to include the physical cost of replacing media, time taken by the MIS team to perform the rebuild and - most important of all - lost production time while the end users wait for their data to be recovered.

One thing is for certain, the more mission critical the application - as with engineering and production systems - the higher the cost involved in rebuilding. Whilst the actual cost of the data in terms of bits and bytes remains the same across all types of applications, the cost of not being able to ship your product out of the door can be enormous.

Mainframe solutions rely on the huge reel-to-reel tape drives for backup, of the kind you usually see whirring away in the background on sci-fi B-movies. Far too expensive for PC use, most standalone users will resort to the humble floppy disk to backup small amounts of essential data. But with the much larger multi-gigabyte volumes available on today�s file servers � and even local desktops - a more efficient means of backup is required. 

Even client workstations have local hard disks running to tens of gigabytes these days, and with the latest server-based software allowing centralised backup of network clients, the destination device needs to be capable of storing many gigabytes of data in a single pass.

We begin our journey into the world of Data Storage Management with a look at the backup issues facing most of today�s administrators

When was the last time you backed up your hard disk ?

What's more important is, when are you planning to do your next one ? Because Murphy's First Law of Computing states that your hard disk will fail just minutes before you planned to start it.

There are many ways to back up your hard disk, ranging from the humble floppy through to the more expensive DAT's, optical disks and even second hard disk drives. Which method you choose depends on how important your data is, how much data you have to back up, and in what sort of time �window�. 

If you have a machine at home with a 6GB hard disk which you use mainly for completing jobs you started at work, you are hardly likely to want to spend thousands of pounds on a sophisticated backup system, especially since most of the files will exist on your work computer as well. However, if you are the manager responsible for a large network whose file server sports several 20GB drives, and on which your company relies totally, you are hopefully not going to trust that data to a few hundred 1.4MB floppy disks.

No matter what your requirements, there is a backup solution to suit you. A combination of hardware and software which should not only be totally reliable, but should be as easy as possible to use - or else you simply will not use it!

Not too long ago, you could split the backup software market into roughly four categories :

This consisted of stand alone PC-based products which could handle network drives if they were mapped as �local� drives in Windows Explorer, but were not able to back up any network-specific information such as the Access Control Lists (security information) and system files (the Windows Registry or the NDS or Active Directory repositories).

In this extremely wide-ranging category were all the packages which had been designed with networks firmly in mind. They were capable of backing up files complete with all their ACL assignments, and were also capable of backing up the system files. 

Most of these packages were designed to be run from a workstation, though a few were written as native Windows services or NetWare NLM modules, meaning that they were running on the file server itself.

This contained all the features found in the previous category, but included additional concepts such as centralised control of backups, databases containing details of all backup sessions, media management facilities  and built-in, rules-based, tape rotation systems. Products in this category usually consist of both server-based and workstation-based components.

This category covers not just backup but archival and data migration, and now encompasses that area of technology which used to be known as Hierarchical Storage Management (HSM).

As with all industries, things have progressed quite rapidly storage management world. Category one has virtually disappeared as manufacturers clamour to move up into the more lucrative network backup arena, and operating system suppliers begin to include fully-featured backup solutions as standard.

Despite the exodus from category one, category two has not expanded. Plain old network backup is old hat now for the major players who are now firmly entrenched in categories three and four.

Whichever backup package you choose, it should be able to handle workstation-based data just as easily as that residing on the network file server. 

Given the high capacities of the hard disk on even entry-level machines these days, it is inevitable that critical data is as likely to reside on the user�s local hard disk as on the network. It is important, therefore, that your backup software is capable of scheduling jobs centrally which will back up disks on any or all workstations. It may also be useful to allow users to schedule backup and restore operations themselves in order to reduce some of the load on the network administrator, and your system must be capable of restricting access to the data on tape to the user to whom it belongs.

The ability to back up critical system files - such as the Windows 9x, the Windows NT Registry, or the Active Directory/NDS repository - from the workstation should also be there in order to minimise the disruption to the user following a major crash of their machine.

And what about those users who work on a PC at home? Unless you want to perform backups over slow PSTN or ISDN lines, you will need to provide a stand alone backup solution for them. In such cases, it would be sensible to select similar - or at least compatible - hardware and software for the home user as that used in the office. Then, if there is a major crash at home, the user can simply bring the latest backup tape into work and restore to a machine there in order to minimise down time.

Naturally, the majority of the data to be protected will reside on central file servers, and this is where most of the high-end packages are aimed. 

Although a network drive is intrinsically the same as a local drive there are other considerations to bear in mind. 

Network servers often have hidden files which are used by the Network Operating System (NOS) itself, which need to be backed up regularly. The file and directory structures are also likely to be different to those found on local disks, since they must cater for additional security - or trustee - information, all of which needs to be backed up effectively. 

Finally, many servers will also support a number of separate �name spaces� to cater for files from other operating systems - these too must be backed up on the same tape as the �normal� network data.

There is now far more choice of server-based backup solutions, and the debate as to whether to install your tape drive in your server or a workstation hots up. There are several key areas to consider :

The first thing to realise is that the options open to you depend entirely on the software you have chosen. A significant amount of the software available today can only be installed in a workstation, for instance. It is meant for backing up stand alone PCs, but just happens to recognise network drives when the PC is logged on as a workstation. Some software even allows you to back up the local hard drives of other PCs attached to the network. 

File server-based software, on the other hand, is capable of being loaded as an NT Service or NetWare NLM, and requires that the tape unit is physically resident in the file server. 

Of course, with networking being such an integral part of the desktop operating system these days, thanks to the various flavours of Windows, the distinction between server- and workstation-based backup products begins to blur somewhat. What is to stop a workstation-based product from backing up any available share it can find in its own Network Neighbourhood? The key differentiator between workstation- and server-based is to be found in the next section.

As far as performance is concerned, there is no competition. A device which is installed in the file server will transfer data at bus speed between disk and tape rather than at network speed. The file server is usually a much more powerful machine than the workstations � both in terms of number and power of CPU�s, and the amount of installed memory - which will also make the backup process faster.

Server-based backup loses its performance advantage, however, when there are multiple servers on the LAN. This is because only the server with the backup device in it will reap the performance benefit, whilst all other servers must still backup across the LAN. One argument to counter this, of course, is that most sites utilising multiple servers would probably have one backup device per server anyway. 

Other performance-enhancing techniques are often employed by server-based backup solutions in order to cope with the limited time available to back up huge amounts of data. See the section entitled The Backup Window.

Placing the tape unit in the file server means that any authorised user can use the central unit to back up his or her own local hard disks, as we discussed in the previous section.

It also allows multiple tape drives to be grouped logically together into a single �media pool� for maximum flexibility, even if those drives are actually located on several different file servers.

Most workstation-based products need to log into the network in order to perform unattended backup operations. This provides a potential loophole should a user be capable of aborting the backup procedure, which will leave him logged into the network with supervisor privileges. Hopefully, most systems will guard against this, but a server-based product eliminates the problem altogether, since a login is not required at all in order to perform its backup operations.

Security of the tape drive itself is another point. Since the file server is usually placed in a room inaccessible to the users, it is much more secure, and midnight backups are much more tamper proof than if they were carried out from a workstation. 

The biggest case against server based backup solutions seems to be crashes, which would render the tape drive useless until the server recovers. Another concern is that backup software placed on the server must compete for resources, which could cause some performance degradation.

In practice, providing the host server is well specified in terms of memory in particular, these are not usually serious problems and can be discounted. What is sorely missing from most server-based backup systems is the ability for the software to automatically check the server environment and inform the administrator if there are potential problems (say with insufficient memory) before the backup software itself is loaded.

Even when your backup software is server-based, it should be possible to schedule jobs either from the server console, or from any client on the network. This provides the maximum flexibility and ease of use for the administrator, who may need to create jobs and maintain the job queue from a remote site, perhaps many miles away from the intended backup server.

You know the problem only too well. You schedule your backup jobs to run overnight to ensure that they do not impact on network usage. Then every single morning you come in to find a list of files as long as your arm which couldn�t be backed up because someone had left their PC switched on and logged in. 

This can be a major irritation, but at least it should be curable. With today�s �always on� philosophy, and an abundance of international operations and e-commerce applications that demand 24x7 operation, there are a number of companies who find that all their key database and e-mail files are never in a state where they can be safely backed up. With no secure copies of critical data, these companies are exposing themselves to the worst possible disaster.

The problem with databases, of course, is that it is not sufficient to simply back up all the files, since most of them will be related to others in the system in some way. For instance, if the system contains a customer file, it will also have at least one related index file. Imagine a scenario where the backup program has been lucky enough to grab a perfect copy of the index file, yet before it can finish copying the data file someone deletes a customer record. If you should later restore these two files, they will obviously be out of step - corrupted. It would have been better not to have backed them up at all. 

This is why most backup systems will simply skip all open files. For some organisations, the only alternative was to actually change their business methods to accommodate the backup regime. Where database files remained open for 24 hours a day, seven days a week, the company was then faced with the unpalatable task of closing down the database one day per week in order to perform backups. The problem is, of course, that this could severely hit productivity, even when choosing a �quiet� day like Sunday. 

Today, this is no longer an option for many. The advent of e-commerce � where closure for even a few hours is noticed by your customers, not just your employees � makes any sort of down time unacceptable.

On systems where files are not subjected to constant use, there are products on the market which allow you to backup open files successfully. Some database systems also offer the built-in capability to �freeze� the database at any particular point in time, thus allowing the backup software to take a �snapshot� of the database. Whilst this is happening, all updates to the database are �buffered� to a temporary file, and these changes are written to the database once the backup operation is finished. 

This can often be a complicated, error-prone, manual procedure, however, which is difficult - if not impossible - to perform in an unattended manner. It is worth noting that today�s messaging systems should also be considered as a specialised form of database for backup purposes, given the size and complexity of their message stores. Likewise, the data repositories used by the main directory services offerings from Microsoft (Active Directory) and Novell (NDS) can also be considered as special types of database.

A better solution, therefore, is for the backup system itself to switch the database � or messaging system - into backup mode, capture the data whilst maintaining complete integrity, and then return the database to its normal state once finished. Some backup systems do offer such a capability - providing you are using one of the mainstream offerings such as Oracle, Sybase, Gupta, Lotus Notes, Microsoft Exchange, SQL Server, Ingres or SAP - in the shape of database �agents�.

Given the increasing use of applications servers on our networks, however, and even better solution is the �application agent�. Backup software has a tendency to operate at the file level � files are what backup systems know best, after all - and usually has little or no knowledge of the applications using those files

In the ideal world, backup software will be know something about the applications themselves and, through the use of  specific �applications agents�, be capable of providing intelligent on-line backup of data files whilst the application is running - and all while maintaining complete integrity of the underlying data.

One of the more important guidelines in choosing a cost-effective backup solution is determining the amount of time available for a backup - often called the �backup window�. 

Smaller organisations, or those which do not rely on their network for mission-critical operations, are fortunate in that they can often schedule backups overnight - or even in the lunch hour - to coincide with periods when the network is not in use. In some cases, in order to fit into the available time slot, it may be necessary to restrict the data backed up to the most critical files - but that won�t win you any popularity contests if your users regularly lose their own data. These files may not be critical to the company�s operation, but they are important to the individual users

Many organisations, therefore, quite rightly determine that all data should be backed up, and preferably on a daily basis. But with data volumes increasing on even the smallest networks, these companies are beginning to find that there are simply not enough hours in the day to perform a full backup of all their data. 

This problem is made worse as new servers are added to the corporate network, yet only the original server contains a tape drive. Now we are trying to back up many gigabytes of data across a relatively slow 10/100Mbps  network link. Quite apart from restricting the speed of backup, this can also have a detrimental effect on network performance as a whole, particularly if backups are performed during working hours. In such cases, it doesn�t matter how fast your tape drive is, since the limiting factor is the speed of the network over which you are backing up. 

In many cases, the backup window simply does not exist any more. We have already mentioned the increase in e-commerce and other on-line applications where a corporate Web site or e-shop front is expected to be open all hours of the day. Potential customers on the other side of the world are not going to be impressed with the fact that your e-commerce site is down for backup during their busiest shopping hours, just because that happens to be �overnight� for you. 

As we move towards the �instant gratification� economy it is a fact of life that there is no good time to bring your computer systems down for even a short space of time � the backup window as we used to know it is a thing of the past.

The relatively low cost and high capacity of today�s mass storage devices has led to an almost complete disregard for good data grooming techniques. If a disk fills up, simply add another one � it will be twice as big as the last one we bought and is likely to be half the price too. The other effect of this phenomenon is that any of the end-user�s workstations have hard disk capacities equal to those of yesterday�s main file servers. 

The downside of this is that even in relatively small organisations, disk storage is capable of growing to such a size that � even with the latest in application agents � it is no longer possible to backup every piece of critical data in a single 24 hour period. So the idea of the backup window is not quite dead � we still need to be able to make our critical data safe in as short a time as possible. However, if we are not careful, shifting such huge amounts of data around our network could quite easily bring it to its knees.

One way to alleviate backup bottlenecks is to install a tape device in every server on the network, which obviously provides the optimum performance, but can also be somewhat costly. The biggest advantage of this approach is, of course, that now every tape drive is operating at bus speed, allowing it to move data from disk to tape far more rapidly than it could across the network. With this scenario, the type of drive - and its native speed - will have a direct bearing on the speed of your backup operation. Some third party solutions can bypass the ASPI layer completely to provide even higher performance from your backup hardware - though at the expense of using a proprietary solution.

Another option is to create a separate �backup network�, with one or more dedicated backup servers on a segment of their own, thus ensuring that the backup traffic does not interfere with day to day operations, and vice versa. Multiple NIC's can be installed in the backup server, and by load sharing between them - possibly in conjunction with the use of switching or high-speed networking technologies - it is possible to increase performance of the backup server and reduce overall backup times to a manageable size. A similar effect is achieved by implementing Storage Area Networks (SAN�s).

Next we can introduce DAT autoloaders or tape arrays into the equation enabling a number of separate backup jobs to be scheduled over a number of days, allowing, say, one database to be backed up one day, another the next day, and so on. Obviously, your backup software needs to be able to control the autoloader directly to obtain maximum advantage from this technology. Another useful benefit of installing multiple drives is the ability to schedule several backup jobs simultaneously, or having a single backup job write to several tapes simultaneously (known as parallel streaming) - both of which reduces the overall amount of time required to back up the entire system.

There are solutions in the market place at the moment which combine high performance, parallel streaming tape arrays with sophisticated image backup software to meet the disaster recovery needs of a high capacity mission critical server that has a fixed, finite and inflexible backup and restore window.

So, we have established that various bottlenecks may exist in a network, ranging from server I/O speed to hard disk fragmentation to network bandwidth.  When the hardware and network design are being pushed to the limits, the ability of a backup product to rapidly store increasing volumes of data within a narrowing backup window is critical. While the data transfer rate is largely defined by the storage media, some of the more advanced products are able to accelerate the speed of a given backup operation via the use of high performance Push Agents.

Push Agents are remote server programs that minimise the �wait state� multiple servers often experience when attempting to interleave data to a single storage device. This can occur when the backup server has to calculate the contents of the data set on each server itself, and then request that data on a file by file basis. In order to get around this, the task of generating the actual backup set can be distributed to the remote servers. Then, each Push Agent identifies the files to be backed up on its server, and packages those files a into tape-ready format, leaving the central backup server to orchestrate the movement of this remote data to storage.  

Another useful technique is known as file interleaving, where files from several backup jobs are interleaved on the same tape simultaneously. This technique has the result of maintaining the streaming operation of the tape drive, which cannot usually be maintained by a single backup job. It is important that each file remains intact during the interleave process in order that the speed of restore operations are not adversely affected.

Using any of these techniques, or a mixture, the individual backup sessions can be reduced to manageable proportions once again.

In the section on server-based backup, we have already discussed the potential security problems inherent in any system which, by its very nature, demands unfettered access to all files on the system. But today�s backup software must also be capable of preserving the security regime created and maintained by the network administrator.

This involved the preservation and compliance with any and all trustee assignments created under the host operating system. For instance, should a user be restricted to backing up only his own data? Some operating systems will allow the creation of �backup operators� which are allowed to back up any data on the network, even though they are not permitted read access to those same files. 

When it comes to restoring data, therefore, the backup software should be capable of recognising whether the user submitting the job is permitted access to the data being restored. If he or she is not allowed to access that data on the network, then they should not be allowed to restore the data, say, to a local hard disk, which would clearly be a breach of the security policy.

Never forget that it is the responsibility of your backup system to protect data from prying eyes as much as from disaster, fire or flood.

We have already mentioned portability with regard to restoring home user�s data onto a machine at the office. But there are more serious issues at stake here, such as disaster recovery or migration. For instance, if the NetWare server goes up in smoke, an organisation may consider it a good time to move to Windows 2000 � or vice versa. The ability to restore the data from the old server to the new one � even though they have completely different data formats - is thus paramount

As with any other industry, standards are important to promote growth. The result is a more stable product - since it does not require re-engineering every time a new version of an operating system is released - and more choice for the end user.

One of the problems which used to trouble users of DAT drives in the past was the low probability that data written to a tape on one drive could be reliably read back by another. This used to cause many problems when shipping large amounts of data between sites with different drives, or simply when upgrading to a new drive, leaving frustrated (and sometimes suddenly unemployed) administrators unable to restore from their old tapes. 

In an attempt to prevent this problem from escalating, the concept of the Digital Data Storage (DDS) standard was introduced. Invented by Hewlett Packard and Sony, DDS has since been adopted as a formal standard by ANSI, ECMA and ISO and is supported by many manufacturers. More importantly, it is subject to thorough collaborative testing programs which ensure that tapes written by one maker�s drives can be read by those of other manufacturer�s. There are two collaborative testing programs, one for DDS drives and the other for DDS media.

Introduced in 1989 as a means of ensuring compatibility between DAT drives, the original idea was to merely double the bit density, which was made possible by higher performance drive heads and electronics. But by removing the track synchronisation areas left over from DAT�s heritage as an audio recording medium and replacing it with a digital equivalent, the format efficiency of the tape was improved from 66 per cent to 90 per cent. Coupled with the proposed increase in bit density, this provided a tripling of the original tape capacity to 12GB (plus data compression) on a 120 metre tape and a corresponding increase in the Data Transfer Rate (DTR). 

The DDS development path looks like this (though there is some doubt at the time of writing whether we will ever actually see a DDS-5 drive):

Note that the terms DDS and DAT have almost become interchangeable these days, although technically they are different animals. If you are investing in a digital tape drive today you would be well advised to ensure that it is DDS, since these drives are rigorously tested for format compliance and data interchange capabilities. 

Seamless data interchange

High levels of scalability, reliability and performance. 

The standard interface offers support for heterogeneous environments by providing a consistent view of heterogeneous file systems (allowing access to NetWare, DOS, OS/2, Windows, Macintosh and UNIX files) and non-file based system resources such as NetWare Directory Services, SQL databases, the NetWare bindery and other services.

The SMS architecture consists of three primary components :

Storage Management Engine (SME). This is the heart of the SMS architecture, communicating with network clients through a common interface that ensures broad platform support and software version independence. The most common SME application is a backup/restore product, and Novell include a basic SME - SBACKUP - with NetWare.

Target Service Agent (TSA). This module isolates the target operating system�s data from the functionality of the SME, thus allowing a single SME to backup a number of network services (servers, SQL databases, local client hard disks, etc.) without changing the core SME application. The basic functionality and capabilities of the SME can be extended by introducing new TSA�s, and some backup vendors write their own TSA modules to provide higher performance than is offered by the NOS vendors� own software.

System Independent Data Format (SIDF). This is an open standard for the format of data on storage media. TSA�s generate data streams in the SIDF format, so that they need  not actually be aware of the media being used to store the data. Since SIDF standard also defines an on-tape data format common to all drives of a particular type (i.e. DAT, QIC, etc.), tapes written under one SME should be readable under any other SIDF-compatible SME, signalling an end to the days of proprietary media formats. SIDF is an extensible format that also ensures compatibility between future and current SIDF-based products, thus hopefully ensuring that data archived today will be readable in 5 years time, even if you are no longer using the same backup and restore software.

In short, SMS provides a development specification that encourages software developers to write for NetWare, and allows then to produce data management solutions which are robust, reliable and as future-proof as possible. Most serious backup packages have either implemented or are planning to implement SMS compliance.

There are three major components to a MTF tape :

A Tape Data Block, otherwise known as the tape header

One or more Data Sets

On Tape Catalogue Information

The On Tape Catalogue Information provides a quick method of locating Data Sets and specific files and directories on the tape. 

Administrators can then find themselves implementing a number of different backup packages, and although each uses a �standard� tape format, it is apparent that the result is a number of different - and completely incompatible - data sets from the different platforms. This is one situation where, although the use of standards has helped to stabilise the backup market, those standards still do little - in practical terms - for the end user.

For this reason, mixed environments may be better advised to acquire backup software which offers a solution on each of the required platforms, and thus offers a single tape format throughout the organisation. The fact that a tape format is proprietary to a particular package is no longer a problem if that package is available on every platform used by the customer. 

The single product approach should also provide the means to control all backup and restore operations, across all platforms, from a single console, and a commitment to cross-platform compatibility from a vendor will provide the administrator with the assurance that platform changes in the future will not affect the ability to restore legacy data.

As with any other section of the industry, backup has its own set of jargon and terminology. Most backup software will boast support for full, differential, incremental, file-by-file, and image backup - but what do each of these terms mean?

Differential backups occupy the middle ground, with the best balance between speed of daily backup and ease of restore operation.

As the name suggests, however, each file is copied to the tape in the same format as it exists on disk, and will thus always allow individual files to be restored at a later date, no matter which backup software you are using.

• Selective file-by-file restore was not possible - restore the whole disk or nothing
• Destination volume must match the size of the source volume when restoring
• Bad blocks on the destination volume could prevent a restore

Although many of today�s image backup products still suffer from all of these problems, there are products which address them effectively by providing :

• The ability to store a representation of the file system along with an image backup. It is then possible to restore only those sectors which are necessary to restore an individual file.
• The ability to restore an entire image to any volume which is equal in size or larger than the original
• The ability to use built-in fault tolerance features of the host NOS in order to avoid bad areas on the destination volume.

Where such issues are addressed by the software, image backup provides the best of both worlds - the speed of image backup and the flexibility of file-by-file. 

We have actually come across users who boast proudly that they �back up every single day�. When questioned a little further, however, it turns out that they only own two tapes and they always back up to the same one (the other one is a �spare�). The advantage of this approach is obvious (even if it is the ONLY one) - they have a full backup performed on a daily basis. 

The problem obviously occurs when they find that the tape is damaged or corrupted (another of Murphy�s Laws states that you only ever find that out as you are trying to restore), or that you need to recover a file from last week - oops! Whatever method you choose, and however often you back up, you must always make sure that you have more than one generation of backups available.

The simplest method of backing up is to use a minimum of three tapes, performing a full backup to each in turn on a daily basis - Monday to tape 1, Tuesday to tape 2, Wednesday to tape 3, Thursday to tape 1 again, and so on. This is quite an effective method - sometimes mistakenly referred to as �Grandfather, Father, Son� because of the three tapes involved - but still suffers from the problem of being unable to recover a file from a week ago. Of course you could decide to use the appropriate number of tapes for the amount of time you might wish to go back - but if the required period is three months this can get quite expensive in terms of tapes and fire-proof storage space.

Another option is to do an incremental backup of some form. This would involve, say, a full backup on Monday on tape 1. On Tuesday you would append only those files which have changed since Monday's backup to tape 1. You would do the same on Tuesday to Friday. The following Monday to Friday you would follow the same routine but using tape 2, and again using tape 3 the following week. 

Three weeks worth of data on just 3 tapes - not bad. But hang on a minute. What happens if your disk crashes and has to be replaced? No problem - restore the full backup from the beginning of the last tape used and then restore the changed files until the end of the tape and you are back where you started before the crash (with a maximum of 24 hours of missing data). But what happens if the full backup is unreadable ? You cannot restore your disk and you will have to resort to the previous tape - you have now lost one week's worth of data (always a potential problem with any form of incremental backup routine).

Providing you have the time to do it (back to our backup window dilemma again) and your tape device has the capacity, the best method is to backup all your data onto a single (yet different!) tape (or set of tapes) each day, overwriting the tapes on a weekly basis. If you need to keep more than a week�s worth of data, simply add more tapes to the equation - 20 would give you a full month�s data. The advantage of this scheme is simplicity - to retrieve a file of a specified age you simply go to the appropriate tape. Likewise, if you wish to restore the entire server, it can be done using a single tape.

• Monday
• Tuesday
• Wednesday
• Thursday
• Friday 1
• Friday 2
• Friday 3
• Month 1
• Month 2
• Month 3 

For the first week, back up everything on each day to the appropriately named tape, and on Friday use Friday 1. On week 2, do the same but use Friday 2, and on week 3 use Friday 3. On week 4, do exactly the same, but on Friday use Month 1. Do the same for the next two months, but on the last Friday of each month use tapes Month 2 and Month 3. Then start the whole cycle again. 

Thus with just ten tapes, at any point in time you have full daily backups for the last week, full weekly backups for the last month, and full monthly backups for the last three months. Of course there are, inevitably, some �holes� in this backup system - for instance, if you wished to restore a file which had a life span of just three days from two months ago, it is possible that it no longer exists on any of your backup tapes.

Naturally, this form of Grandfather, Father, Son system is problematical to implement manually. Some of today�s more advanced backup software is capable of managing a completely automatic tape rotation system, informing the operator when a new blank tape is required, specifying how that tape should be labelled, ensuring that the correct tape is inserted for each backup operation, and informing us which tape set should be moved off-site for maximum protection. Complete automation is the key here, yet the ability to specify a custom set of tape rotation rules is also important to many.

Just a couple of points to remember. If you are backing up a network, make sure that your software is capable of backing up all the hidden system files and related security settings. Otherwise when you come to restore your file server, you will find you have to create your users and print queues all over again. Or worse � perhaps your system won�t even fire up following a full restore operation. Not a pleasant prospect.

Finally, never trust the read-after-write verify on your tape device. Always do a full verify after every backup - it is the only way to ensure that the data on tape is readable. Even better than this, if your software allows it, is to do a full comparison of what is on tape against the data stored on disk byte for byte. This ensures not only that the data can be read, but that it corresponds with what was backed up in the first place.

This means that those tapes should not be kept on the desk next to the file server. If your backup system creates multiple generations of your data, then make sure that at least one of those generations is always kept off-site - whether that is at another office or at an employee�s home - and the remainder should be stored in a fire-proof safe. A useful feature available with some systems is the ability to perform a tape-to-tape copy in order that off-site copies can be maintained whilst still keeping recent backup data handy in case it is needed on-site.

And once you have decided where to store your tapes, take another look at it. There is the story of one company who backed up everything religiously, even performing a full disk versus tape verification after each backup, only to find that when it came time to do the inevitable restore, every single one of their backup tapes were blank. How could this be? It turned out that the network administrator had a wonderful hi-fi system in his office with night club-grade speakers. The backup tapes were piled neatly on a shelf � right next to one of the speakers. As quickly as he was writing data to his tapes, he was wiping them clean again as he listened to his music.

This is an extreme example, but illustrates how careful you must be in storing your tapes. Strong magnetic fields are obviously a no-no, but you must be equally wary of extremes of heat or cold, damp or humidity. All of these can affect the longevity of tape, and even if you don�t expect to restore data from years ago, bear in mind that a badly handled tape that is a few years old may well fail with yesterday�s data on it. 

Be aware that however carefully you store your tapes, they do have a finite life. Check out your tape vendor�s Web site for data on expected life-span and storage requirements. For instance, the DDS Manufacturer�s Group has this to say on the life-span of DDS cartridges: 

�Approximately 2000 passes may be made over a section of tape before degrading its performance. Depending on the backup application and other factors, each backup session can cause up to 6 passes. A practical answer to the question is 100 insertions. If the cartridge was to be used weekly as part of a backup cycle, then it would last two years. However, as DDS cartridges are inexpensive, it is unwise to economise by using them too long.�

This is excellent advice - it is not worth risking your precious data to an old or damaged tape for the sake of a few pounds spent on new media.

Last but not least, make sure that you label your tapes correctly. There is nothing more frustrating than attempting to restore yesterday�s data and finding that you have actually gone back a week. That is usually when you find that it was yesterday�s data that was stored on that dodgy five year old tape covered in mildew, too. 

Labeling becomes even more important when you have implemented a complicated tape rotation system that relies on tape numbers rather than names. As we have already said, some automated tape rotation systems will tell you how to label each tape � make sure you do it there and then. Once those tapes are in the fire-proof safe they have a tendency to mix themselves up thoroughly � it�s another one of Murphy�s Laws.

Since by far the most common restore operation is the recovery of a single file, it should be easy to browse the tape library and select the required file. If multiple versions of a file exist on several tapes, these should be clearly identified along with the date of last access, file size, etc., allowing the user to choose the required version from a �pick list�. Since most backup systems allow users to submit their own backup jobs, it also makes sense that they be able to perform their own restores, within the confines of the security regime imposed by the system administrator, of course.

It should also be possible to specify the location to which you wish to restore a file. Perhaps you want to keep the existing data intact until you have examined the restored data, just to make sure the restored stuff is exactly what you want before you copy it to where it should be. 

Of course, we are back to the security angle again, since we do not want just anybody restoring critical company data � such as the payroll files - to their local hard drive to peruse at their leisure.

Where the backup system has the capability to backup system databases such as the directory services repository or the e-mail message store, it should also be possible to browse through and restore single directory containers or mailboxes.

The real test of a backup system, however, is how it copes with total disaster. When complicated Grandfather, Father, Son tape rotation systems are employed, it can be a nightmare to determine manually which tapes are to be used to perform a total restore. 

Having specified the point to which you wish to restore the system, the backup software should guide you carefully through the restore procedure, telling you at each stage which tapes need to be inserted in the drive, or else selecting the required tapes from an autoloader completely automatically. Obviously, in order to achieve this, the tape rotation database must be available to the backup system. Usually, this will have been stored on the very system which has been rendered unusable by the disaster, so make sure that your backup software takes adequate copies of its own databases as part of the normal backup procedure.

Some systems go one step further in providing a set of �disaster recovery� diskettes. These will contain a snapshot of the critical system files synchronised to a backup set, and sometimes even the means to restore the operating system itself from tape. It is thus possible to restore the base operating system quickly - using just a single floppy disk and tape - to the point where a full restore of all the data can take place.

One vendor has also introduced a �One Button Disaster Recovery� feature. All that is required to do a full system restore is to insert the most recent backup cartridge, hold down the tape drive's eject button, and power on the tape drive and computer system. This puts the tape drive into a special disaster recovery mode that allows it to restore the operating system and reboot. (The tape acts like a bootable CD-ROM.) Once it has restored the operating system, it returns to tape drive mode and users can restore data as normal.

Whilst basic backup and restore operations are all that is required for the home or small business user, today�s backup software should be capable of providing a number of other services if it is to be considered fit for the enterprise:

Watch out for software which merely boasts the capability to switch to an alternate drive should a job fail, but which does not actually support mirroring. In such cases, the backup job must be restarted from the beginning on the new drive, and thus the second backup job may not be able to finish within the allotted backup window.

RAID tape systems offer the same degree of fault tolerance as mirroring, as well as providing increased performance and capacity by spreading write operations across several drives. Faulty tape drives and media no longer prevent the completion of successful backup operations, and full recovery can be realised even if one of the original tapes is lost or destroyed

There are currently very few backup products which include an anti-virus scanner as part of the base product, and where it is included it is usually fairly rudimentary (though obviously better than nothing at all)

The ideal scenario would be the ability to integrate a full-blown, mainstream anti-virus scanner with the backup software. This would not only provide the appropriate level of protection, but would also guarantee that the anti-virus mechanism is complete and can be kept up to date as new viruses are discovered.

In mission-critical environments, it may not be acceptable to come in at 9am to find that the backup did not complete, so we are increasingly finding extensive notification options built in to backup software. Various types of notification can often be selected depending on the severity of the message to be conveyed. Simple e-mail is enough to inform you that a job has finished with no problems, for instance, whereas the aforementioned pager, telephone call, or even an SMS text message can be used to inform of a critical error at any time of the day or night. 

Some backup systems can even be integrated with a server-based fax system to provide fax notifications, and links to industry standard SNMP-based management systems are also becoming increasingly desirable.

Whereas regular housekeeping might have been the order of the day when they only had a 2GB local disk to play with, the huge amount of storage space available on the file server simply leads to bad habits or misguided ideas on data security. The latter results in the old �you never know when I might need that file again� syndrome, which in turn results in much of the available network storage being clogged with files which would be better kept off-line.

In many cases, you may find that the user does not even know the exact name of the file to be restored, causing you to sift through hundreds of the things, possibly even moving a significant number back to on-line storage for the owner to search through at leisure - back to square one.

In the meantime, more RAM is needed to cache the additional disk space (impacting on server performance), and more and more time and media space is required to perform adequate backups - perhaps even resulting in the purchase of additional backup hardware. All this is additional cost over and above the new disk drive itself.

This is a vicious spiral, with more and more data resulting in less and less control - and tight control of data storage is precisely what is required when enormous legacy applications are being migrated to the LAN. Fortunately, certain of the mainframe techniques and technologies are coming with them. One of these was HSM � or data migration - which effectively provided �unlimited� data storage together with exactly the sort of control we were looking for.

But why do such technologies eventually move down into the LAN world? Haven�t we got by perfectly well without them so far?

The problem with the LAN is that, to a certain degree, it is a victim of its own success. What began life as a means of sharing a few files and the odd printer has rapidly become the mainstay of the corporate computing environment. The LAN now plays host to the mission-critical applications and data vital to the well-being of the enterprise, and in order to protect it adequately, it seems only natural to resort to techniques employed for years by larger systems.

To achieve this, almost any number of �levels� may be introduced into your migration hierarchy. At its simplest, migrated data can be moved directly to a single tape drive if the amounts being migrated will fit on just one of your chosen tape media. If you require faster retrieval times of recent data, you can introduce a middle layer of optical (or other removable) storage, to which data from the hard disk will be migrated first. Only when it is very old or has been marked as ready to be archived will data be moved from optical to tape (thus freeing space on the optical drive). 

If larger amounts of secondary storage are required, additional drives may be added or, more usually, autoloaders can be installed. Although retrieval times can be adversely affected if they involve a disk or tape change, the ability to increase secondary storage space into the terabyte range far outweighs any disadvantages. A key feature of each lower level of hierarchical storage is that the cost per megabyte of storing data is lower than that used at the previous level - along with this is the assumption that retrieval times are correspondingly slower.

The ability to fine tune migration rules is essential, in order that files which have rules specifying �migrate after 1 week� and which are one month old, will actually be migrated before files whose rules state �migrate after 1 year� and which are two years old. 

As a final point, of course, you would need to be able to specify that certain files or file-types were never migrated, in order to prevent system executables or libraries from being moved from the server volumes.

A �high water mark� should also be set which, once exceeded, causes a �critical migration� to take place which migrates as much data as is necessary to reduce volume occupancy to an �acceptable level�. 

Some systems implement what is known as �pre-staging�, which ensures that inactive data is copied onto secondary storage even when primary storage levels are below the optimum level. Thus, if a critical migration is suddenly required during peak network usage, all that is necessary is to delete the copies of the pre-staged files from the primary, thus effecting the migration almost immediately, and without adversely affecting network performance with heavy data transfers.

To all intents and purposes, the file appears as if it still resides on the server volume, but when one of these �stub� files is accessed, retrieval agents running on the server inform the user that the file has been migrated, giving them the option to wait until the file has been recovered, continue working on something else or abort the de-migration altogether. 

Providing the de-migration is not aborted, the migration software determines the location of the file from its internal database, loads the relevant media and moves the file from secondary storage back to its original location on the server volume. Since only a copy of the original file is provided, when that file has aged enough to move back onto near-line storage and its original contents have not been altered, the copy on the hard drive is simply deleted, thus eliminating unnecessary migration processing and traffic.

Migration, on the other hand, is quite different. Rather than simply make a copy of the files on the secondary storage, the files are also deleted from the original location. The migration process can take place during scheduled, non-peak �sweeps�, as well as when thresholds are crossed or trigger events - such as the �age� of a file - occur. Files can also be migrated during idle processor cycles with no particular urgency.

De-migration, however, needs to be performed as quickly as possible, since generally a user is actually sitting in front of his PC screen waiting for it to happen. De-migration allows much quicker access than when restoring a backup file, though depending on the physical media on which the migrated file is located, some delay is inevitable.

Wherever possible, however, you should look for systems which are complementary - the tighter the integration between your migration and backup systems, the less chance there is of losing critical data. For instance, losing the zero length �stub� files can cause major problems for some systems, so the ability to protect them and retrieve them quickly from backup media is essential. Also, your backup system should be capable of recognising stub files for what they are, and backing them up as they stand. There is no point in de-migrating a file from secondary storage simply to back it up to tape.

Likewise, if your backup system includes sophisticated tape rotation and archival facilities, it would be beneficial for the migration routines to know when a file has been permanently archived as part of the backup routine, thus allowing it to free up its own near or off-line storage capacity. One significant advantage noted by many sites that implement data migration is that, because of the reduction in occupied disk space on the managed servers, the main backup operation can take considerably less time to run.

Another option is to read the data from the migration media without de-migrating it - the technical term for this is �Peek�. The only problem you have solved is that of space on your primary volume, however. The backup would still be extremely slow, because you are still reading from the migration media to get the data for backup. Additionally, as pointed out earlier, the order in which your backup software will backup files is unlikely to be the same as the order in which they were migrated.

The third option is that any file ever migrated is caught on a set of permanent tapes before they are migrated, a process referred to as a �catch basket� or �pre-migration� scheme. 

The biggest drawback with this scheme is that if an entire server was lost and it had to be completely restored, there may not be enough space on the primary volume to do it, since there is likely to be a large amount of migrated data. 

The aforementioned alternatives have the primary drawback that they are backing up migrated data. This means that there may never be enough primary storage space available to permit a full restore of all the data, as well as the fact that backups and restores will be extremely time-consuming. 

The ideal scenario is to only backup the keys (stub files) and put the responsibility of securing the migrated data on the migration system itself. This way, you will only restore the keys, and you will thus have enough space to restore data in its entirety. Almost as important is the fact that your backup will be quick, and you will be able to complete it in your permissible backup window. 

Some migration systems provide their own integrated backup software to protect the migrated data. Another option is for the system to safeguard the migrated data by providing incremental mirroring of migration media for optical and tape stages, coupled with tight integration with your chosen backup package. The magnetic stage would be a file server volume and thus can be backed up using your main backup software. 

The incremental mirroring should be triggered by your main backup software when it backs up a primary server. Since the mirroring is triggered by the main backup operation, the user does not have to worry about scheduling it. The backup operation is scheduled as normal and the mirroring will be automatically triggered whenever a primary server in the migration hierarchy is backed up. Because the mirroring is incremental, it is also very fast. Additionally, the mirror media is completely synchronised with the main backup tapes so that the user can put both backup tapes and the mirror media off-site on the completion of the backup. 

Almost as important is the considerable reduction (if not elimination) of the data management burden on the administrator, which can be almost as great on a 20GB system as on a 2000GB system!

In addition to data protection, organisations wishing to maintain their competitive edge require the ability to quickly restore or recover critical information ranging from customer data to internal operations. This task has become extremely difficult due to the sheer amount of data spread across WANs and distributed heterogeneous systems, while contending with bandwidth saturation and usability issues. And while in the past it may have been acceptable to have servers �down� for a period of time, this is no longer the case. Today, system downtime often results in loss of business, decreased market share, and possible disaster for an organisation - regardless of its size or industry.

Many enterprises have taken a distributed approach to storage management. Backups are either performed over the LAN, where several systems are backed up to a central storage device, or locally, where a system has a backup device directly connected to it. The SCSI bus, while a mainstay of storage connectivity for over two decades, still has some limitations. Ultra Wide SCSI today only delivers up to 40Mbps and sustains up to 15 devices on the chain.

Fibre Channel � for so long a technology with no applications � is the critical enabler for the SAN. SAN's utilise high-speed fibre optic or copper cabling to interconnect between server and storage devices, resulting in data transfer speeds of up to 200 Mbps in a dual loop configuration or 100 Mbps in redundant mode. 

Storage sharing � or storage consolidation � enables multiple computers across a corporate network to access a common set of storage devices such as disk arrays, tapes, optical drives and autoloaders. Think back to the early days of the LAN when we were sold the idea on the back of the promise of sharing expensive resources across multiple users on the network. Those resources � big disks, tapes, printers � were installed on the central file server, from where they could be accessed by everyone with the appropriate authorisation.

SAN storage sharing simply introduces another level of abstraction. Now, those same resources (only much bigger) are moved out of the server and attached directly to the network, thus allowing them to be addressed directly by multiple servers. For instance, if your backup software expects to find a tape drive in the local server, you would normally have to install a drive in every server on your LAN. Now, fewer tape drives can be installed in a central array and attached to the SAN, making them accessible to every server on the network

With storage sharing, exclusive access is provided while a device is assigned. Primary storage may be assigned to a computer for a long time because the data and applications on the storage become integral parts of the computer. Secondary storage, like tape drives, may be assigned to a computer for much shorter periods of time, often only as long as is necessary to back up the computer�s data files.

This exclusive access is important to preserve data integrity, especially when the same disk device is shared between two completely different operating systems. Allowing both systems to access the disk simultaneously could cause unpredictable � even disastrous � results. The SAN must therefore be capable of hiding mount points from end users, and even preventing the OS itself from recognising the presence of locked devices on the SAN.

The SAN also improves the concept of data sharing. Although a typical LAN enables applications and end users to access data held in a central location, the SAN moves that data onto a much faster infrastructure. This allows multiple computers to transfer large files concurrently at rates comparable to locally attached disks over the SAN without adversely affecting the corporate LAN.

Usually, of course, it is the host operating system that controls access to local hard drives, and can thus preside over access privileges and file locks when more than one application attempts to use the file at the same time. Once the disk storage is removed physically from the server, however, the SAN itself must take over and secure access to files through a volume lock manager or distributed file system software.

Data sharing requires that the participating computers be able to find and use the contents of a file. Hence, computers with different operating systems must use protocol translation modules and other software to establish a common communication dialect.

Ordinarily, data sharing is associated with primary storage devices. But it can be done with secondary storage. Tape devices are linearly accessible file systems managed by the backup software. Robotic tape libraries containing multiple tape devices and a large quantity of media can be accessed by more than one computer using SAN�s, which makes it possible to share data on secondary storage devices.

But while instituting a local backup solution might seem the obvious answer, it is impractical from both a logistical and economic point of view to, for example, simply attach a tape drive or library to each application or database server. This type of fragmented solution presents an administrative nightmare, with limited reporting, management, access, and control for each distributed server. Furthermore, the SCSI bus - the thick, awkward cable that provides server to storage connectivity - has numerous limitations, including its length, the number of devices to which it can attach, and its bandwidth and burst rate.

SAN's uncouple the front end of the IT infrastructure - applications, operating system, and processors - from the back-end storage. This enables businesses to meet their expanding storage requirements while still maintaining rapid response at the application, business process, and user level. 

However, SAN's don't completely free servers from their back-end tasks. Businesses not only store information they also have to back it up. Business servers still have to execute backup functions, meaning they have to read data from the storage device and write it to the backup device. When you consider the enormous, multi-terabyte databases of major enterprises and eCommerce websites, it is not hard to see how this can compromise server performance and hence SAN performance. 

Serverless backup is a simple, elegant solution to the problem of how to safeguard massive and exponentially growing amounts of data without compromising system performance or limiting the bandwidth available for enterprise communications. It is therefore important that the backup system can support the SAN directly, otherwise it could force data to travel from SAN-based storage via a file server on the LAN just to enable the backup software to write the data to tape. In such cases, the file server would act as a bottleneck, slowing down the backup process and once again threatening the backup window. 

Serverless backup over SAN's requires three major components: 

• The backup application itself
• The Extended SCSI Copy Command standard
• A protocol-aware, intelligent SAN appliance that can recognise protocols from many heterogeneous systems and transmit data at high speeds to the tape and DLT libraries.

With serverless backup, the data flows across the SAN directly from the disk drive to the tape device, with no data moving through the server. The enterprise servers only need to host the backup application, and the backup application determines what needs to be backed up and sends the command to a �copy agent� embedded in an intelligent SAN appliance. The intelligent SAN appliance detects the source and destination parameters, retrieves the data from the storage devices, writes it to the tape or DLT libraries, and reports completion (or status) back to the backup application.

With backup now handled by the SAN devices, enterprise servers can continuously process applications and information, and not concern themselves with �housekeeping� tasks like data movement. Routine backup can be performed regularly during peak business times rather than during off-peak backup windows, or backup can be performed offline altogether with zero impact on the enterprise.

In addition to off-loading the LAN, the backup software provides storage sharing capabilities, so multiple computers can use a set of tape devices. Advanced media and file management capabilities should also be included in backup software packages, so secondary storage data sharing is available.

In a conventional storage architecture, the storage sub-system is accessible only to the server or the platform the sub-system is attached to. As storage requirements grow in the environment, the administrator is forced to reallocate storage capacity, denying accessibility of those resources to other platforms. The answer to these challenges is to share resources between multiple platforms. SAN connectivity enables resource sharing between multiple backup servers, enables administrators to consolidate backups to one storage sub-system. This simplifies management and enables efficient use of storage capacity.

This same replication capability can also be used as a tool to remotely mirror data to an alternative site at local, metropolitan, or worldwide locations, providing further levels of data protection and redundancy.

Several storage devices, including servers, disks and tape devices, constitute the SAN environment. Storage Resource Management (SRM) defines applications that monitor and manage physical and logical resources. SRM includes capacity management, configuration management, event and alert management, volume management and policy management. Effective SAN management necessitates SRM tools be integrated with the SAN management solutions.

The ideal enterprise management tool includes the SAN as a storage network topology and as a sub-network to the communication LAN or WAN. The SAN is not isolated, and SAN solutions should not be built in isolation from other IT disciplines. Management solutions should offer SAN discovery topology mapping and end-to-end management for fibre channel devices in the loop. 

hile the tools for centralised management can be fairly expensive, the cost savings realised by improved reliability and ease of management more than offset the infrastructure costs.

In addition, SAN�s provide total cost of ownership benefits such as:

• Minimised down time
• Improved LAN performance
• Ability to connect to existing LANs
• Reduced administrative costs
• Leveraging of existing hardware
• Improved fault tolerance
• Maximised storage resources through load balancing
• Total SAN management

SAN awareness today is becoming widespread. As more and more organisations integrate SAN�s into their environments, they need reassurance that their current storage management solution takes full advantage of both existing storage technology and SAN technology. 

Open standards are thus increasingly important to allow a single solution across multi-vendor heterogeneous networks.

The backup device vendors have an array of bewildering terms to confuse you when you go looking for the ideal solution:

Traditionally, linear tape methods have offered superior reliability, capacity and transfer rates. Advances in helical scan have resulted in a drop in the mean time between failure, but for traditional backup and archive solutions, linear tape probably still has the edge

It is worth taking such claims with a pinch of salt however. Not that the drives are not theoretically capable of such capacities, but you would have to have exactly the right kind of data to achieve them. Most people would be happy with a 50 per cent increase in capacity with normal data profiles.

It is also worth remembering that drives which employ hardware data compression may appear to have a higher transfer rate than those which do not. For instance, if a ratio of 2 to 1 is assumed (which is the norm), whereas a normal drive will have to back up 8MB of data, the drive which uses compression will be actually backing up 4MB, but will report that 8MB has been transferred in a fraction (though probably not a half) of the time of the other drive. Most vendors will quote DTR for both uncompressed and compressed data.

Some newer devices - such as AIT-2 - include a memory chip in the media which provides rapid access to indexing data without even having to consult the tape.

Automated tape libraries can significantly reduce the cost of backup over traditional manual methods, making libraries a compelling choice for organisations seeking cost-effective solutions to their backup problems. One point against them is the temptation to leave a full tape rotation set within the autoloader, and let the backup software simply cycle through them. Whilst this is an incredibly efficient means of providing instant backups and restores, you may have problems if the computer room goes up in smoke along with all your on-line AND off-line data. 

Once you have a basic grasp of the terms used by the hardware manufacturers, you can begin to examine the numerous technologies on offer:

There is a huge choice of Quarter Inch Cartridge (QIC) drives to choose from with varying prices and build quality. Both 5�25� and 3.5� form factor drives are available, and capacities range from less than a megabyte to a massive 50GB (MLR).

One problem with basic QIC devices is poor performance when it comes to restoring. Since the QIC is a purely serial device, it is necessary to search the tape from beginning to end for the required files, which can result in lengthy restore times if you are unfortunate enough to be searching for the last file that was backed up!

Despite the increase in capacity, initial offerings required no change in media formulation and were able to utilise existing drive electronics and available head technology. Capacities range from 400MB (800MB compressed) to 10GB (20GB compressed), making it ideal for the home and small business user.

4mm DAT uses helical scan technology resulting in high capacities - the typical capacity of a 60 metre DAT tape is 1.2GB, whilst the 90m tapes can hold 2GB when used with DDS-1 drives. DDS-2 drives offer an uncompressed capacity of 4GB on a 120 metre tape, and DDS-3 drives provide a native 12GB capacity on the same length of tape. The new DDS-4 drives raise the bar yet again, offering a native (uncompressed) capacity of 20GB on a 150 meter tape.

With data compression, however, the maximum capacity of each of these is usually doubled, taking the current maximum for a DDS drive to a very respectable 40GB (DDS-4). Quoted data transfer rates range from 11MB per minute for DDS-1 to 180MB per minute for DDS-4, whilst individual file access times are an average of just 20-50 seconds, much faster than 8mm drives. 

4mm DAT was designed from the outset as a digital medium, hence it does not require a specially modified analogue tape transport like 8mm. Given the lower wrap angle and fewer moving parts, together with the three-level error correction as opposed to 8mm's two-level, 4mm drives should offer greater long term reliability than 8mm. Cartridge life is expected to be in the region of 2,000 passes, or 100 full backups.

Whilst DDS is still common because of its low cost per gigabyte and large installed base, the format is reaching the end of its life, and DDS-4 is likely to be the final generation. 

Performance is enhanced by the use of an innovative Memory In Cassette (MIC) feature, consisting of a 16Kbit EEPROM that is mounted within the data cartridge. Using a serial interface to the memory chip, the AIT drives are able to store and retrieve drive, index and user-generated information directly from the chip, resulting in faster data access.

The two formats currently available are: 

MLR1 - 13GB native capacity (26GB compressed)

MLR3 - 25GB native capacity (50GB compressed)

MLR1 is read compatible with all existing QIC drives down to QIC-120, although MLR tapes can only be read by MLR drives. A single 5.25 inch MLR3 cartridge provides 25GB native data capacity (50GB with hardware compression) with a native data transfer rate of up to 2MB/sec (4MB/sec with compression). 

One final feature of MLR drives which is of huge significance in the enterprise data storage market is the smooth auto-load and eject mechanisms, which makes them suitable for huge auto-loading tape library implementations. This provides a much more scaleable solution than DAT, whilst retaining a significantly lower price point than DLT. Data retention is also a significant feature of MLR tapes, with quoted life-spans in excess of ten years.

With capacities like that and prices far in excess of a reasonable QIC solution, and even 4mm DAT, 8mm drives are obviously aimed at organisations with large data storage capabilities. Even more so when you realise that there are automatic cartridge changing mechanisms available which provide a total backup capability measured in terabytes! For these reasons, 8mm drives tend to be more popular with mini-computer users rather that PC users.

One problem which has dogged the 8mm tape manufacturers is that of reliability. Using a modified analogue tape transport means that it has more moving parts than its 4mm brothers and many more than QIC drives. In addition to that, the wrap angle - the number of degrees the tape wraps around the spinning head assembly - is 210 degrees compared with just 90 degrees on 4mm. This means that there is more strain on the tape itself, more contact with the heads and rollers, and thus more friction and shorter tape life on 8mm drives.

One final point to bear in mind is size of the drive. 8mm drives will always be limited to a 5.25" form factor because of the size of the bulky 8mm video cartridge. Most of these are full height too, and only the smaller capacity (2.5mb) drives are currently available as half height.

Due to an agreement with Sony, the manufacturer of the 8mm camcorder drive, Exabyte is the only other company allowed to resell the specially modified drive. This means that prices are more or less fixed no matter where you shop.

The next generation of DLT, Super DLTtape, will offer 100GB per cartridge and a transfer rate of 10 or 20MB per second. SuperDLTtape uses Laser Guided Magnetic Recording (LGMR), which employs optical and magnetic methods to greatly increase the data storage capacity of cartridges.

DLT also features low tape tension, and sophisticated error detection and correction methods for the maximum reliability. DLT drives are able to recover data even if up to two inches of tape is damaged within an eight inch tape segment. DLT is also highly suitable for archival purposes, given a guaranteed life-span in excess of 30 years, and over one million read/write passes. As with 8mm drives, high-end autoloader mechanisms exist for DLT to provide terabytes of tape storage in a single enclosure.

Given the high transfer rate, storage capacity and longevity provided by this technology, many manufacturers are looking at DLT as the basis for high-end auto-loader storage subsystems.

Ultrium is optimised for high capacity storage, providing native 10, 30, 50 and 100GB cartridges and a native transfer rate of 20MB per second in the first generation. It offers 384 data tracks compared with Accelis' 256, thus increasing capacity significantly, and fourth-generation cartridges in premium drives are currently projected to hold 800GB of data (1600GB compressed) with transfer rates of 160MB per second (320MB per second compressed). The error correction code (ECC) used in the Ultrium products is powerful enough to ensure reliable recovery of data even with the loss of a track or head.

The Accelis and Ultrium formats use the same head, media, track layout,  channel and servo technology, as well as sharing many common electronic building blocks and code blocks. Where they differ is that the Accelis format uses a dual-reel tape cartridge and a midpoint load feature to achieve fast access time to data. In contrast, the Ultrium format uses a single-reel tape cartridge and a wider tape width to provide high capacity.

LTO technology is an �open format� technology, which means that users will have multiple sources of product and media. The �open� nature of LTO technology also provides a means of enabling compatibility between different vendors' offerings.

Slightly more expensive and limited to the same maximum capacities is CD-RW (CD-Rewriteable). The only advantage of CD-RW is that data can be rewritten to the same disk over and over, which may prove slightly more cost-effective in the longer term. Today's MultiRead CD drives will read CD-RW media and CD-RW drives will read CD-ROM, CD-R and CD-RW media from any industry-standard source. In addition, the multifunction drives will write both CD-R and CD-RW media.

Like CD-R, WORM (Write Once Read Many) drives can only be written to once, but media capacities are higher. WORM drives adopt a variety of methods to write data to the disk, but they all result in a change of the reflective properties of the disk surface. Like CD-ROMs, data is read by detecting such changes. This medium is therefore suitable for backing up and archival purposes. After each backup, a "history marker" can be placed on the disk. The disk can later be temporarily returned to the state it was in at any particular history marker, thus allowing access to the files at that point. 

Obviously the disk will eventually fill up and is then unusable except to read the data stored on it. For this reason, people will use it for archival rather than backup. 

For instance, many businesses might like to keep several years worth of sales statistics available, but limited disk storage doesn't allow them to be kept on-line. If they are transferred every few months onto tape before being deleted from the live system, they must be restored from tape before they can subsequently be analysed. If, however, they are written to a WORM disk, accessing the data at a later date is simply a matter of mounting the appropriate disk.

The permanence of write-once technology makes it ideal for archiving and security-intensive applications. For this reason, write-once solutions have been very popular with banks, legal firms, government agencies and others concerned with data integrity and long-term document storage.

A logical extension to the WORM drive is the Magneto-Optical (also called multifunction, rewritable optical or MO) drive. Whilst CD-ROM and WORM drives are purely optical, the magneto-optical, as its name suggests, fuses magnetic and optical technologies. The disk's recording layer is magnetic, and the drive uses a magnetic field (called a bias field) to change the polarity of areas of the disk. 

However, at normal operating temperatures the bias field is too weak to change the magnetic properties of the disk, so a laser beam is used to raise the temperature in highly localised areas to the point where the bias field is capable of affecting the polarity. The same laser beam, but at a much lower power, is used to read the data. The difference in polarity affects the way laser light is reflected and this is detected in a similar way to the change in reflectivity on a CD-ROM.

Early multifunction drives were forced to write the data in two passes. Later designs, however, use phase change technology to write data in one pass, thus making them much faster. It has to be remembered, though, that they are still slower than most hard drives, so you wouldn't want to use them as your main disk, even though the removable aspect is attractive.

MO media capacities range from 650MB to 2.6GB, but it is worth noting that writing to conventional MO media takes 50 percent longer than reading the same data because the drive must erase previously stored data, verify the media and then write the new data. To overcome this performance problem, a 2.6GB direct-overwrite solution has been introduced. Using direct-overwrite technology, the drives write at a full 4MB per second, making them more suitable for backup or disaster recovery applications.

A more common format at the time of writing is the Zip drive, of which 22 million units had been shipped by 1999. The latest iteration, a 250 MB Zip drive could painlessly replace the 100 MB drives in the future becoming the next �floppy drives�. In fact, one 250 MB Zip disk is approximately equivalent in capacity to 174 floppy disks. With its 29ms seek time, it is also much faster - up to 2.4 MB/sec compared to 0.06MB/sec for floppy.

Whereas Zip is little more than a glorified floppy drive, Iomega�s flagship removable drive product � Jaz � behaves much more like a traditional hard drive. It uses hard, thin-film magnetic disks which disks spin at respectable 5394 RPM. However, even if the Jaz drive's average seek time seems quite acceptable (10 ms read / 12 ms write), its average data transfer rate of 7.4 MB/sec is les than half that which you would expect from a typical hard drive. However, the maximum capacity of 2GB almost renders this a moot point, as well as consigning Jaz to the lowliest of backup tasks.

The ORB 2.2 GB drive from Castlewood Systems has a slightly larger storage capacity than it's main rival � the Iomega Jaz � as well as offering a wide range of interfaces, including both internal (EIDE, SCSI) and external (Parallel, SCSI, USB, IEEE 1394 Firewire). It is also available for both PC and Macintosh computers, and offers � along with the higher capacity - a 50 per cent increase in sustained transfer rate and lower media costs.

Finally, hard drive technology has reduced significantly in price to the point where it is now feasible to install a removable hard drive cradle in your server and backup your main hard disk directly to a second drive. With up to 80GB available on a single IDE drive at the time of writing, it is possible, with compression, to backup multiple fixed drives to a single removable one. 

Of course, the biggest downside with this approach is that it is necessary to bring the server down in order to swap removable hard drives. Although hot swap technology would get around this limitation, the cost of hot swap implementations is prohibitive at present, thus negating one of the main advantages of using removable hard disks as a backup media.

Even the smallest company can generate hundreds of megabytes or even gigabytes of data, the loss of which could seriously jeopardise the well being of the organisation. With such high stakes, investment in data protection hardware and software should be considered essential.