NCSA releases HDF, Image, and DataScope
Scientific computing can be a messy business. “We were constantly encountering problems of different computing platforms having different data formats,” said Mike Folk, who has been with NCSA since 1988. “Scientists also wanted different types of data to all be mixed into a single file.” NCSA HDF—a general file format for scientific data management, as well as software for analyzing and converting data—answered the call.
Today, HDF and its progeny are used worldwide in many fields, including environmental science, high-energy physics, nuclear stockpile stewardship, and aerospace engineering.
It started, as many things did and still do at NCSA, with a research scientist and a student. Astrophysicist Michael Norman tasked one of his students to create an application that would create color raster images from his datasets. The result, ImageTool, “was very Sun [a popular Unix-based workstation] centric, but it really mapped the direction for tools of this time,” according to Tom Redman, a research programmer now in NCSA’s Automated Learning Group.
Redman then began working on expanding the tool, starting with a version for the Macintosh. “The Mac II was the first relatively inexpensive workstation that supported a graphics environment required of a tool like ImageTool. NCSA had a very good relationship with Apple, and Macs were already mainstream at NCSA. The graphics subsystem and windowing capabilities were very compelling,” said Redman. “Image [which ImageTool eventually became] was an application that allowed scientists to ‘see’ the huge volumes of data that they were producing on our supercomputers. For scientists of the day, this was an indispensable tool for data analysis.”
Over a short period, an entire suite of tools that allowed scientists to do visualization on their workstations was developed. Most notably, this collection included DataScope, a more powerful tool that worked with floating point numbers in the data, allowing data to be rescaled before they were converted to pixels.
The relationship with research scientists was crucial. They gave the programmers motivation. “A scientist was constantly coming up to you and saying ‘Can I…?'” said Mike Folk, who still works in software development at NCSA. But they also gave those programmers free rein. “You have to credit researchers like Norman for turning things over to us and letting us run,” said Folk. “That was key to our success. We were able to take what they were doing, listen to what was important, and harden it.”