Storage networks

The massive growth in data and the development of storage systems, have created exciting new technologies that have added...

The massive growth in data and the development of storage systems, have created exciting new technologies that have added functionality to the most humdrum of applications. Where will it end?

The improvements in magnetic storage technology are almost unbelievable. I recall the days only 20 years ago when data processing systems had to manipulate data between a fixed five Mbyte and a removable five Mbyte cartridge.

The multiplatter CDC drives with 50Mbyte capacity were a wonder to behold; like tape drives, they were essential for movie makers, because you could see something happening through the perspex covers. Nowadays I sometimes wonder if I have enough capacity on my PC drive - is 6Gbytes enough? And not only has the capacity increased out of all proportion, but so has the speed and, even more, the size. These 2.5 cm drives they put in cameras hold a gigabyte.

The advances have made a significant impact on the whole IT industry. It is not just a matter of the same as before but cheaper, it is a whole new range of applications that have been enabled by the availability of storage. The obvious advance is in the ability to process graphics. The next trend is to include more moving imagery, whether we want it or not. The important thing to note is that it introduces new systems into the IT world, but doesn't replace the earlier requirements. We still have to run the payroll and invoicing systems!

The outcome of all this is that storage technology advances have affected all sectors of IT. But the development has not been planned - rather it has 'happened', Thus, since all these developments took place under different managers, and from different budgets, the physical storage is all over the place. This would not be such a problem if all these developments have stayed independent of each other, but they haven't. Text is stored on individual PCs - in many cases - most of which is valueless. But manuals, documents, groupware etc. all need communal storage. Data processing systems should have been easier to manage, but in practice this has proved difficult.

ERP talk
Today there has been a lot of talk about Enterprise Resource Planning (ERP), a key feature of which is to base various application sub-systems on a common database, but, historically, new application requirements were introduced as new systems and data were transferred between the applications as needed. Separate applications have created serious storage management problems, but in fact it is worse because many applications were introduced on new database management software, and on new and different hardware and operating systems.

In an ideal world all the existing systems could be replaced by one integrated system, with only one RDBMS and one physical storage pool, but it will never happen. Already, there are a lot of complaints that after all the effort of introducing an ERP system, there have been few gains - it is just too complex. Thus we will have to live with the problems of running and managing disparate systems, probably spending more on automating workflow with management tools.

It would be very nice to have a common database, which could then be run as a separate entity, to which a variety of applications could be interfaced from a wide range of clients, both local and remote. But even this is being too ambitious. And so attention has turned to the lowest common denominator - the physical storage.

The idea of a common file system has been extensively used in PC local area networks, Novell Netware being the most common, but it was also common with NFS on Unix servers. NFS and extensions has been used for large scale systems, but the shear size of the storage requirements, and the need to continually expand it has led to a compromise in which various computers are connected by a very high speed network to a storage sub-system.

Multiple sub-systems can be added as each one reaches its capacity, and there lies one of the key problems. If users are not to be tied into one supplier, then they need to be able to add storage sub-systems from mixed suppliers, and that means standards, networking protocols and software. It is not likely to happen in the IT industry, and we will still be tied to specific suppliers.

Read more on Software development tools