HDF Group’s technology helps NASA, major companies juggle huge amounts of info

Copyright (c) 2010, The News-Gazette, Champaign-Urbana, Ill.

Distributed by McClatchy-Tribune Information Services.

(News-Gazette (Champaign-Urbana, IL) Via Acquire Media NewsEdge) Dec. 06–CHAMPAIGN, Ill. —

When companies and researchers have to juggle huge amounts of data, they sometimes turn to a technology developed in Champaign-Urbana to store and retrieve that information.

The technology, known as HDF (Hierarchical Data Format), was developed in the late 1980s at the National Center for Supercomputing Applications.

Today the technology continues to be developed and deployed by The HDF Group, a not-for-profit organization based in the University of Illinois Research Park.

Over the years, HDF technology has spread to new applications: NASA uses it to handle climate-change data collected by satellites.

Genetics researchers use it to compare data gained from DNA sequencing.

A major aerospace company uses it to handle flight-test data.

Computer chip manufacturers use it to evaluate chips on silicon wafers.

As uses have grown, The HDF Group has grown too. It now employs 34, and on Monday it will move to new quarters in the Gateway Building, the former home of Motorola.

“The amount of space is about the same, but the configuration is much better,” said Mike Folk, the group’s chief executive officer. “There’s room for growth in the Gateway Building, but there was no room in the old building.” The organization will take about 7,000 square feet on the second floor at 1800 S. Oak St., C. It was formerly on the second floor of the SAIC/Caterpillar building at 1901 S. First St., C.

When Steve Scopel, the group’s manager of human resources and business affairs, set out to look for new space for the organization last year, he was told he should plan for 20 percent growth over three to five years.

In the year since, the company has grown 10 percent, he said.

“The single reason for why we’re growing right now is, we’re experiencing resurgent use of HDF in supercomputing, now called the high-performance computing area,” Folk said.

HDF technology uses a file format designed to store and handle high-volume, complex data.

“Probably our biggest user is NASA’s Earth Observing System,” Folk said. That satellite system gathers data to help scientists understand climate change.

“That system consists of three satellites… Each satellite generates a terabyte of data per day,” he said.

NASA looked at 15 different file formats and selected HDF for archiving the data, partly because it’s flexible and scalable.

“As the amount of data grows, HDF can handle that growth,” Folk said.

HDF started out as a technology for handling simulation and visualization data on supercomputers.

As computing got faster, HDF kept up with the load. Around the year 2000, the big challenge was terascale computing the ability to perform a trillion operations per second.

Now researchers are doing petascale computing a thousand times faster, at a quadrillion operations per second.

The next generation of computing — exascale computing is expected to be available in six to eight years, and The HDF Group is already involved in projects focusing on that.

The HDF Group was formed at NCSA in 1987, and was spun out as a separate not-for-profit corporation in 2006.

Folk said that the group “struggled” with whether it should be not-for-profit or a for-profit company. He chose the former because he wanted the group to focus on its missions of long-term data preservation and user support and not get distracted by chasing whatever generates dollars.

Looking to the future, Folk said, one of the group’s goals is “to broaden our customer base. We now have 13 customers, and when we spun off, we had six.” But 60 percent of the income today comes from only two sources, and Folk said that’s risky for the long-term stability of the organization.

“We want to broaden our customer base, so we have less risk. We have to find a model that’s sustainable,” he said.

On the technical side, the biggest challenges are to develop staff capabilities to meet the needs of current and potential clients, and to develop the technology and the group’s methods for working with customers, he said.