Contents Under Pressure - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Information Management

Contents Under Pressure

Terabytes of data, escalating workloads, and subsecond response to urgent requirements for business-critical information: Welcome to the realm of large database management. We analyze key trends in how organizations are meeting scalability challenges.

Scalability Challenges for Large Databases

When it comes to databases, popularity breeds challenges. Even as they amass more data, both OLTP and decision-support systems must get faster and more friendly to intense query activity. In a report from the only survey of its kind, we learn how key trends are shaping choices in technology.

It's a miracle the lights don't dim, the earth doesn't shake, and the sun isn't occasionally darkened by data clouds erupting from a system that just can't take it anymore. Across the globe, every minute of every day, enormous databases are running thousands, if not millions, of simultaneous operations to support online transaction processing (OLTP). Meanwhile, other databases take on massive data warehouses as they respond to the complex number-crunching needs of strategic business applications and their users.

Especially for the largest database-management systems, scalability is forever a work in progress. Boundaries are moving up and out constantly. For some, scalability is about the number of users, simultaneous queries or transactions, and query speed; for others, it is about the sheer amount of raw data the system must store and manage. And for many, it is about all of the above.

For the past 10 years, Winter Corp. has been tracking the world's largest and most heavily used databases. We have developed quantitative insights concerning the demographics, operating characteristics, and practices used to develop and manage large database. In this article, we discuss trends and directions in large databases, as revealed by our most recent survey — the 2003 TopTen Program, conducted from May to October 2003 - as well as other surveys and interactions with user organizations.

The TopTen Program received more than 300 survey results, which originated from 23 countries. Qualifying Windows-based databases had to contain more than 500 GB of data, while databases on all other platforms had to exceed 1 TB. Note that we define database size as the sum of user data, summaries, aggregates, and indexes — not the complement of attached disk. We required respondents to validate their entries by running queries developed by Winter Corp. and associated industry experts. We sorted the entry databases by usage and operating system, and assessed them according to four metrics: size, uncompressed data volume, number of rows/records, and workload. Based on these metrics, we developed category lists of the world's leading databases, along with data points that we are presenting in this article.

Steady Growth

The most compelling findings from the data are the growth in database size and workloads. This growth has several dimensions, the most visible and perhaps most daunting of which is size. Large databases may contain thousands of tables and support as many users, who are producing and demanding access to more data than ever before. These users are scattered across the enterprise in geographically diverse locations, and their numbers are rising steadily. Database management software is expected to process data volumes whose sizes dwarf what they handled previously.

Our research shows that database size is accelerating. Figure 1 discloses the range in size of the 10 largest decision-support and transaction-processing databases in the 2003 and 2001 Winter Corp. surveys. In two years since the 2001 program, the largest transaction-processing database almost doubled in size, from 10.5 to 18.3 TB. On the decision-support system (DSS) side, the 10th largest database is nearly the size of the biggest database in the 2001 program. And the largest decision-support database in 2003, at 29.2 TB, is almost triple the size of the 2001 leader.

FIGURE 1 In two years, the largest databases have increased in size two- and threefold.

Increases in row count provide another example of the growth in database scale. Figure 2 shows the growth in number of rows over the past two years. Row count on every platform for both decision support and transaction processing grew significantly since 2001. Unix DSS databases experienced the greatest growth. Their average number of rows grew six times, propelling them past transaction processing z/OS systems to become the new champions of row count.

FIGURE 2 The Unix DSS database average is now almost 40 billion rows.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 5
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

Remote Work Tops SF, NYC for Most High-Paying Job Openings
Jessica Davis, Senior Editor, Enterprise Apps,  7/20/2021
Blockchain Gets Real Across Industries
Lisa Morgan, Freelance Writer,  7/22/2021
Seeking a Competitive Edge vs. Chasing Savings in the Cloud
Joao-Pierre S. Ruth, Senior Writer,  7/19/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
Monitoring Critical Cloud Workloads Report
In this report, our experts will discuss how to advance your ability to monitor critical workloads as they move about the various cloud platforms in your company.
Flash Poll