Swift/T: Dataflow Programming for Scientific Supercomputing Workflows

Organization: Argonne National Laboratory
Co-Developer(s): University of Chicago
Year: 2018

Swift/T: Dataflow Programming for Scientific Supercomputing Workflows is a programming language that makes it easy to express complex workflow features such as data dependencies, concurrent execution, and parallel tasks. At its core, it provides a practical implementation of the theory of dataflow processing, in which each execution happens when its input data is available (not subject to an instruction pointer). Thus, many executions may be ready to run at a given point in time. The Swift/T runtime, called Turbine, is a parallel system capable of handling trillions of data-dependent tasks. It contains a scalable data subscription/notification system that can handle extremely large task graphs. Its underlying task distributor is a masterworker system with multiple masters that carry out “work stealing” to rapidly distribute user tasks. Swift/T is being used by many projects within the new U.S. Department of Energy Exascale Computing Project, which brings together key exascale-ready applications and emerging software technologies. Swift/T has recently made major contributions to enable cancer research on supercomputers, both via deep learning and conventional simulation.

For more info: www.anl.gov