HPC data analytics is changing how high-performance computing (HPC) environments are designed and used. Systems once focused primarily on running simulations and exporting their results are increasingly expected to support data-intensive analysis, machine learning and interactive work as well. The Blue Waters experience illustrates the infrastructure and software considerations involved in bringing these workloads together on a shared computing platform.
Why HPC data analytics is converging with simulation
Two developments are driving this convergence. First, observational research produces growing volumes of data, increasing the need for computationally demanding methods that can extract useful information. Deep neural networks, for example, require substantial computing and storage resources.
Second, large simulations generate outputs that can be difficult to store and analyze on local systems. Transferring those outputs may also delay decisions about what to run next. An alternative is to analyze results where they are produced: either by reading data from high-performance storage or by processing it in memory while a simulation is running. This approach can make more complex analysis possible and reduce dependence on moving large datasets to separate workstations.
The relationship works in both directions. Data analysis can reveal patterns in simulation output, while simulation can complement analysis of observational data. Combining these methods can also involve machine learning, multi-scale modeling and multi-physics simulations.
Designing storage for different data workloads
Traditional scientific computing commonly writes simulation output to a parallel filesystem, then transfers files elsewhere for visualization and analysis. Data-processing systems built around instruments often follow a different pattern: ingest streams of data, process them through independent or readily parallelized tasks, and retain selected results.
Modern data science adds another requirement: keeping both incoming and transformed data available for repeated queries as new information and hypotheses emerge. This interactive, iterative style of work places different demands on storage and software than a single simulation run followed by a one-time export.
Blue Waters used the Lustre parallel filesystem, which provides shared access to files across compute nodes through dedicated storage servers. Files can be distributed across servers, and large files can be striped across them to increase bandwidth. The report contrasts this architecture with the Hadoop Distributed File System, which operates across local disks in a Hadoop cluster. Using Hadoop on an HPC system presented challenges, including adapting it to shared parallel storage and integrating it with the batch scheduler. Hadoop Adapter for Lustre was later available as an alternative to emulating local disks.
Apache Spark offers another model by keeping as much working data in memory as possible, reducing storage access. A large-memory, many-node system can therefore support substantial Spark workloads, with the potential to size a cluster for a particular analysis and release resources when the work is complete.
Software environments and interactive access
Making data-science tools usable at scale
Python is widely used in data science, but its many library files can create startup overhead on a parallel filesystem, particularly when programs run across many cores. Blue Waters addressed this issue with a custom Python environment, bwpy, presented as a virtual read-only filesystem. This allowed library access to be managed separately by each node while the shared filesystem handled a single large file.
Interactive tools such as Jupyter notebooks can run in different ways depending on workload: on a login node for light use, on a dedicated compute node for heavier work, or across multiple nodes for demanding analysis. However, access to large compute resources is governed by a batch scheduler. Jobs may wait in a queue, so unpredictable start times can conflict with continuous interactive work. Reservations for specific nodes at a scheduled time can support interactive development, debugging and exploration at scale.
Managing software consistency and workflows
Data-analysis applications often rely on multiple external packages. Using package versions different from those used to develop and validate an application can cause incompatibilities or errors. Containers can package an application’s software environment, but running Docker directly requires administrator privileges—an unsuitable security arrangement for a shared system. Blue Waters instead supported Shifter and Singularity to run Docker images on compute nodes without that requirement.
Workflows also need to coordinate program execution and data movement. Scientific workflow tools such as Swift/T can schedule large numbers of serial tasks. Globus Online can manage and monitor transfers to and from the filesystem and near-line tape archive. These capabilities help connect computation, storage and transfer into repeatable workflows.
GPUs and machine learning workloads
Graphics processing units (GPUs) can accelerate both traditional simulations and deep neural network applications. On Blue Waters, GPUs supported training a single network and exploring different network hyperparameters. TensorFlow was among the available tools. More broadly, these workloads show why a system intended for multiple kinds of computation must account for more than processor performance: memory, storage, networking and software all affect how effectively applications can scale.
Practical implications for complex computing environments
The Blue Waters example is a research computing environment, not a specification for an industrial automation system. Its broader lesson is that combining workloads requires attention to the complete computing workflow, not just the processing engine. Teams assessing infrastructure for demanding analytics should consider:
- Data location: Whether analysis can run close to large datasets or simulation output.
- Storage behavior: How shared files, parallel access, in-memory processing and long-term archives fit the workload.
- Software consistency: How package versions and application dependencies are managed on shared resources.
- Resource access: Whether interactive development and scheduled production work require different access models.
- Workflow coordination: How task execution and data transfers are automated and monitored.
For industrial automation engineers and technical decision-makers, these are useful questions when evaluating data-intensive computing requirements. The report does not prescribe a particular plant architecture; it demonstrates the importance of aligning storage, compute, software and scheduling with the work being performed.
Conclusion
HPC data analytics brings simulation, observational data processing and machine learning into a shared computing environment. Blue Waters supported this convergence through a combination of parallel storage, large-memory resources, GPUs, adapted software environments and workflow tools. The practical takeaway is that successful large-scale analysis depends on coordinating the full data path—from ingestion and execution to storage, transfer and repeatable workflow management—rather than treating compute capacity as the only requirement.