# DMTN-025: A survey of workflow management systems

• Mikolaj Kowalik,
• Hsin-Fang Chiang,
• Greg Daues and
• Rob Kooper

Latest Revision: 2018-06-11

# Scope of document¶

This document summarizes the results of the survey regarding viable candidates for a workflow management system (WMS) to be used by LSST Batch Production Services [CONOPS]. A WMS helps automate orchestration and execution of large batches of tasks on available, possibly distributed, computational resources.

# Introduction¶

At the time of writing this report (Aug 2016), it is hard to point to any clearly defined “winners” or, at least dominating, standards in the field. Different projects focus on different aspects of an orchestration which are usually dictated by direct needs of their authors and/or were designed to work in certain “ecosystems”. Such issues cast some doubts on the possibility of their application in other domains or interoperability with technologies different than intended. On top of that, the maturity of those projects, their support from developers and community, and available funding varies significantly, making a selection of a WMS satisfying needs of the LSST project regarding Batch Production Services far from being straightforward. In order to being able to make, at least, educated guess which WMS would work best for the LSST project, we decided to take a closer look at 8 of the most “serious”-looking WMS’s and survey them all in more detail against a set of selected criteria we deem important considering project’s needs.

Finally, it is worth emphasising that this current survey has a preliminary character. Its goal is not to select a future workflow management system for the LSST project but to select 3 or 4 close contenders for further, more elaborate, testing with the LSST stack in HPC environment.

# Methodology of the survey¶

Our goal is to pick a WMS that will work in the batch system of LSST. To narrow down the list of potential candidates to a smaller subset we have looked at past and upcoming XSEDE conferences and workshops [XSEDE]. The main reason to look at the XSEDE workshops is that all of the workflows discussed at these workshops are capable of running on the XSEDE resources and thus are ready for large scale executions (Makeflow, Pegasus, RADICAL-Pilot, and Swift). Besides the workflows discussed there we considered some of the more recent workflows systems that have been open sourced by different large companies (Airflow, CloudSlang, and Pinball). Finally we looked at other fields such as high energy physics and picked one of the prominent workflows from there (Panda). Each of the candidates was then reviewed against the set of criteria which may be divided into three main categories described in more detail below.

## Technical merits¶

Before we compared the different workflow systems we created a set of criteria that were used to evaluate the different systems. These criteria focused on making sure that the workflow system that was picked is established and well-proven, since it should last for the project duration, and we want to make sure we can get help from the community both with questions, help with solving problems and the ability to add back to the workflow system chosen.

### Flexibility in workflow generation¶

Workflow management systems use wide variety of methods to create workflows; some uses dedicated GUIs, others follow “configuration as a code” approach and the workflow description is a script written in Python. Those methods offer different degrees of freedom and flexibility for the authors of the workflow and therefore we consider this aspect an important characteristic of a workflow management system.

### Interoperability¶

As we already mentioned in the Introduction some of the existing workflow management systems were designed to work in certain “ecosystems”. As the LSST project’s execution environments may differ from the one a WMS was designed for, it is essential to know what technologies a given system can cooperate with currently or how easy it can be extended to support others.

### Scalability¶

As workflows varies in size and complexity, it is important that a workflow management system is able to handle both small and large workflows (in terms of number of tasks) equally well, e.g., that memory requirements grows sufficiently slow with increasing number of tasks in a workflow.

### Performance¶

Though scheduling tasks and tracking their provenance is not a particularly CPU-intensive process, we wanted to be sure that under a large load a workflow management system will not become a bottleneck and will utilize available resources efficiently.

### Reliability¶

With a large number of tasks running on distributed computational resources some failures are inevitable. Thus any decent workflow management system should support automatic retries, i.e., rerun the failing task, to be able to deal with transient errors (e.g. a temporary network downtime). It should also track the provenance of tasks to allow for a workflow restart in case of more permanent execution errors (e.g. runtime errors due to flawed input).

### Monitoring tools¶

Workflow management systems provide logs which can be used to monitor tasks execution. However, with numerous large and complex workflows, going through logs is not a particularly useful method to monitor workflow execution in a real time. That is why in our survey we decided to take into account existence of any extra monitoring tools such as dashboards.

## Available support¶

Considering the scale, complexity, and longevity of the LSST project, we are convinced that purely technical merits cannot be the only ones to be taken into account during the evaluation process. After all, every piece of software has bugs which need to be fixed, configuration issues which may not be covered in documentation, and so on. Thus, we also decided to consider things like:

• communication channels allowing tracking of a current project’s development and reporting of issues;
• the number of active developer and their responsiveness in addressing issues reported by users;
• the number of use cases, size of the project’s community and its activity on available project’s channels;
• and last but not least, its funding.

## Overall “user experience”¶

Finally, we decided to estimate the overall overhead associated with getting started with a given workflow management system by installing (almost) each of the surveyed frameworks on our workstations. That step allowed us to address following questions:

• How thorough and complete is project’s documentation? Is a user able to get all necessary information from it or does he or she have to use the “Use the Force, Read the Source!” principle on regular basis?
• How complex is framework’s installation and configuration process? Does it have a relatively small number of dependencies or, on contrary, does it require a multitude of external libraries to have all of its advertised features enabled?
• What is the learning curve associated with creating and executing an example workflow?

# Results¶

We evaluated each project listed earlier based on criteria described in the previous section using available online resources, i.e., white papers, documentation, described use cases, and users’ opinions. Our findings is best summarized in Table 1 (entries marked with a question mark denote pieces of information we were not able to determine at the current level of scrutiny). For sake of completeness, we also provide farther below more in-depth reviews concerning each investigated workflow management system for an interested reader.

Table 1 Comparison of surveyed workflow management systems.
Airflow CloudSlang Makeflow Pinball PanDA Pegasus RADICAL Pilot Swift
Flexibility in workflow generation high high low not tested medium medium high high
Interoperability
• Hive,
• Mesos
• CoreOS,
• Docker,
• Heroku
• Condo
• SLURM,
• PBS,
• SGE,
• Torque
? Condor
• Condor,
• XSEDE,
• SGE,
• EC2,
• Condor,
• SLURM,
• PBS
• SGE
• Condor,
• SLURM
• PBS
• SGE
Performance high high high high high high high high
• Retries
• Restarts
• Replications
• yes
• yes
• ?
• no
• yes
• no
• yes
• yes
• no
• yes
• ?
• ?
• ?
• ?
• ?
• yes
• yes
• yes
• yes
• no
• no
• yes
• yes
• yes
Logging yes yes yes yes yes yes yes yes
Monitoring tools GUI no GUI GUI GUI GUI no no
Scalability good good good good good good good good
Load balancing no ? yes ? yes yes ? yes
Prioritization yes ? ? ? yes yes yes yes
Provenance tracking yes ? ? ? yes yes yes yes
“Pilot job” functionality yes ? yes yes yes yes yes yes
Data transfer yes [1] ? yes ? yes yes yes yes
Data caching no ? yes ? ? ? ? no
Language of implementation Python Java C Python Python, JavaScript Python, Perl, shell Java Python Java
Dependencies
• MySQL,
• Celery,
• message broker
None None MySQL Condor
• Condor,
• Ant,
• OpenSSL
MongoDB Globus
OS requirements Linux / OSX / Win Linux / OSX / Win Linux / OSX / Win Linux / OSX / Win Linux Linux Linux / OSX / Win Linux OSX / Win
Licence type Apache v2 Apache v2 GNU GPL v2 Apache v2 Apache v2 Apache v2 MIT Apache v2
“DOB” 2015 2014 2009 2015 2005 2001 2014 2007
Active development yes yes yes yes no yes yes yes
Latest release May, 2016 July, 2016 May, 2016 ? 2015 Apr., 2016 May, 2016 Aug., 2015
Documentation quality good very good good poor good good good very good
Installation complexity medium easy easy not tested not tested easy easy easy
Learning difficulty low low low not tested high medium low high
Popularity high high low low low [2] low [3] low medium

Footnotes

 [1] Staging data in and out must be included explicitly in the DAG representing a workflow with help of so called Transfer operators.
 [2] Number of reported use cases is small but some of those projects are huge!
 [3] Same as above.

# Summary¶

LSST Batch Production Services [CONOPS] are about executing “[...] campaigns on computing resources to produce the desired LSST data products [...]” where campaigns are defined as sets of pipelines (ordered ensembles of computational steps), inputs they are being run against, and methods of handling their outputs. As campaigns can vary in size and complexity, we will face a non-trivial problem of orchestrating their execution on potentially distributed computational resources while satisfying the data-dependencies. As this pattern is quite common in many scientific and business applications, there exist many frameworks called “workflow management systems”, whose sole purpose is to automate such processes.

To select an optimal workflow management system for LSST Batch Services we made a detailed, multi-aspect survey of a few available workflow management systems to estimate their usefulness in implementing objectives of LSST batch operations [CONOPS]. Based on our findings, we have selected three of them: Airflow, Pegasus, and PanDA for further tests in which the chosen candidates will be used to orchestrate execution of a few specific LSST pipelines (yet to be determined) in NCSA’s HPC environment.

Currently a large process is underway to change how the middleware operates. This new version is the Gen3 Middleware and consists of a rewrite of Butler as well as the addition of SuperTasks and pre-flight. The pre-flight will use the Gen3 Butler, as well as the SuperTask to create a quantum graph (workflow like graph) that can be executed to run the actual pipelines. This work is in the implementation phase, depending on the results and changes of this Gen3 Middleware we will re-evaluate some of the workflow systems to see which one fits best with this new middleware and the project requirements.

During this time we have briefly looked at the Parsl software as another option of a workflow system as well as successfully prototyped using the DESDM framework in LSST operations additionally porting it to Python 3 to be ready as another solution.

## DESDM Framework¶

In the past year we have looked at the DESDM framework as one of the options to execute the pipelines in an operations environment. At NCSA we have experience running this as part of the DES project running in production for the past 5 years for nightly processing as well as producing data releases. NCSA has trained operators to run the software, analyze the results and retry select sections of the workflow. The system is ready to use and can execute tasks in parallel on a HTCondor cluster. Code is available for glide-ins, allowing an existing cluster (such as lsst-dev) to be configured to allow for execution of tasks from the DESDM framework. The framework will collect statistics, enabling us to do performance testing, as well as provenance tracking allowing us to find out exactly for each data file generated how it was generated, with what inputs and by whom. In the past months we have spend time on converting the existing code base from Python 2 to Python 3 to fit into the LSST software stack. We also wrote LSST-specific plugins to enable running of the Gen2 LSST pipelines. We have done tests using both the Python 2 version running the LSST software stack and RC dataset as well as using the python 3 version and the same dataset to test whether the DESDM framework is ready to run the existing LSST pipelines.

## Parsl¶

Parsl (Parallel Scripting Library) is a scripting library written in Python. The current version is 0.5.1. Parsl has been under development for the past 2 years and is developed as a collaboration between The University of Chicago, Argonne National Laboratory and the University of Illinois. The code is released under the Apache 2.0 License and is available at GitHub. Parsl is a workflow system that can manage the data movement and execution of tasks on different hardware such as cloud, clusters and other resources. Tasks can be executed in parallel and will only be executed when required data is ready. The tasks that are executed are either bash scripts or Python functions. The functions will take arguments that are supplied during the definition of the workflow, such as parameters or outputs from previously executed tasks. The workflow system uses futures to setup the tasks to be executed, and will only execute those tasks that are needed for the final result. If a task requires the results from another task it will wait for the previous task to finish. Files in Parsl are abstracted allowing for the movement and placement of the files anywhere in the filesystem. Parsl will move the file to the required compute resource and the task will use the special Parsl file object to read and write the file from the disk, alleviating it from knowing where the file is actually stored. Tasks written in Python have some limitations and require all modules to be explicitly imported in the code, and global variables do not exist. Workflows in Parsl are programmatically defined and are coded using the Python language, using Parsl specific functions. Tasks that are to be executed in this workflow system are annotated with the @Apps annotation. Parsl is written from scratch and is influenced by the SWIFT language which we have looked at in more detail. We have not looked into Parsl in detail yet to see how it would fit in the LSST project.

# Appendix A: Airflow¶

## Overview¶

Airflow is a platform to programmatically author, schedule and monitor workflows developed and used internally by Airbnb. It is written in Python and released under Apache License, v2.0. The project was started in the fall of 2014 by Maxime Beauchemin at Airbnb and officially brought under the Airbnb Github and announced in the spring of 2015. The project joined the Apache Software Foundation’s incubation program in the winter of 2016.

## Features¶

### Flexibility¶

Airflow’s workflows are specified as a Python code which offers a great deal of flexibility when designing workflows. On top of that, Airflow supports Jinja templating. Thus workflows can be generated programmatically. Additionally, it has a simple plugin manager that can integrate external features into its core if required.

### Monitoring¶

Airflow comes with the web dashboard allowing a user to visualize, manage and monitor execution of available workflows.

### Performance¶

We were not able to find any solid performance metrics of Airflow.

### Interoperability¶

Airflow comes fully loaded with ways to interact with commonly used systems like Hive, Presto, MySQL, HDFS, Postgres and S3. Sources I came across do not mentioned more HPC-oriented technologies/protocols though, in theory, it may be easily extensible due to its modular architecture.

### Reliability¶

Airflow supports retrying failed tasks. However, it seems that workflow restarts are not directly supported although this blogpost describes how to use of subdags to emulate them.

### Scalability¶

Project’s website claims that due to its modularity Airflow can orchestrate arbitrary number of tasks and workers. In their own words:

Airflow is ready to scale to infinity.

Sadly enough, they do not provide any solid data to support this bold claim. One example we were able to find, mentions running 20’000 tasks per day on a 32-core machine which is not particularly impressive from a HPC perspective.

## Support¶

Currently the projects is being developed and maintained by a group of 9 programmers.

There are two places where an Airflow user can seek assistance: maling list and Gitter channel.

Community gather around the mailing list seems to be rather vibrant. From the moment it was established (Apr, 2016) there were approx. 700 posts. Similarly with Gitter channel, there are quite a few Airflow users who may be asked for help if needed.

Project’s GitHub page lists 39 companies which declared that they are using Airflow as their workflow management system.

Currently the project undergoes incubation at the Apache Software Foundation. Though it means it is not yet fully endorsed by ASF, it suggests it may have a sustainable source of funding in the near future.

## User experience¶

### Documentation¶

Airflow comes with a rather well written tutorial explaining how to create, validate and execute an example workflow. However, the user guide and API reference are somewhat lacking. Information is provided without a clear order and a level of details one would expect from a user guide. Some more advanced aspects (e.g. writing hooks) are not covered at all. Probably the worst is the chapter regarding, rather feature-rich, Airflow’s web UI. It is basically a catalog of screenshot focused more on displaying some its features rather than explaining how to work with it in some organized manner.

### Installation and configuration¶

Besides a working Python installation, the project does not list any external dependencies. Though there are other extra packages available (their complete list can be found here) and they may required some additional software to work.

Airflow installation is pretty straightforward as it can be installed via PyPI using pip. Though initially, recommended installation procedure


span.prompt1:before {

## User experience¶

### Documentation¶

Besides project’s README file, Pinball’s documentation is practically non-existent.

### Installation¶

Pinball requires:

• Graphviz: a graph visualization software,
• libmysqlclient: MySQL client libraries.

Pinball uses MySQL database as persistent storage so an access to a running MySQL instance is also required.

Installation is very easy; Pinball package is available via PyPI and can be installed by pip.

### Hands-on experience¶

Taking into account overall impression (lack of decent documentation and support), we did not even try to setup an example workflow with Pinball.

## Summary¶

Pinballs looks like a rather immature project with uncertain future. Practically, it has no documentation, does not seem to be actively developed, and support from developers looks rather poor. Others seems to come to similar conclusions though to be fair at least installation issue was fixed (see the link above). As a result, it is not a viable candidate for use with LSST Batch Operation Services.

## Overview¶

RADICAL Pilot (RP) is a pilot job system (a library) written in Python and released on MIT License. It allows a user to run large numbers of computational tasks concurrently on a multitude of different distributed resources, like HPC clusters and Clouds. RP is being developed by RADICAL, a group from Rutgers University. Its latest stable version, 0.40.3, was released in May, 2016.

## Features¶

### Flexibility¶

As RP is a library, its workflow specifications are Python scripts. Thus RP can be used to execute (theoretically) arbitrarily complex workflows and generate them dynamically. Jobs can be single or multi-threaded; MPI applications are also supported.

### Monitoring¶

Each compute unit has a metadata associated with it which can be inspected and used to monitor a unit execution. However, RP do not provide any monitoring tools besides logging.

### Performance¶

We were not able to find any metrics regarding rate of dispatching jobs or sustainable CPU utilization levels.

### Interoperability¶

RP is built on top of The Simple API for Grid Applications (SAGA, also being developed by RADICAL), a high-level API for accessing distributed resources, meaning that RP should work on a variety of backends such as PBS, Condor and Condor-G, SGE, SLURM, LSF or Amazon EC2. Data transfer is supported via local copying, SFTP, and GSISFTP protocols.

### Reliability¶

Each compute unit progress through a certain state model and failed units can be easily detected. However, RP documentation does not provide examples of any recovery mechanisms beyond simple retries. As far as we can tell workflow restarts or task replications are not possible.

### Scalability¶

According to this source, RP can marshal O(10k) distributed cores and schedule O(10k) tasks in a single application context across multiple distributed HPC resources.

## Support¶

The group developing RP consists of 4 researchers and developers and 10 students.

There are two Google groups for dealing with project’s issues:

1. radical-pilot-users (public) to report problems and get assistance from the team and community,
2. radical-pilot-devel (requires membership) for contributors and people interested in discussing implementation issues.

The traffic on a public forum is very light (42 threads over 2 years) with one or two threads per month on average. All reported issues were addressed by a RP developer rather quickly.

It seems that the community is very small, e.g. few users. On top of that, not a single showcase is mentioned on project’s website and we were unable to find any external project claiming using RP internally.

RP is a part of RADICAL Cybertools, an abstraction-based suite supporting science on a range of high-performance and distributed computing systems. The whole project is supported by two NSF grants till 2018 with a total budget approx \$1.5M.

## User experience¶

### Documentation¶

Documentation includes a tutorial, with ready to run example scripts, introducing various aspects of working with RP, e.g., setting up a session, monitoring jobs, staging input and output data. However, one have to at least scheme the user guide anyway to be already familiar with core RP concepts to understand what is going on. Also, rather annoying thing is that documentation is not entirely in sync with the code base. We encountered few cases when following documentation verbatim led to execution errors and applying “Use the Force, Read the Source” principle was necessary.

### Installation¶

RP requires the following packages:

• Python >= 2.7,

• Virtualenv >= 1.11,

• Pip == 1.4.1

Warning

They really mean it! Newer versions of pip fail to install RP properly without emitting any errors.

Using RP on remote machines requires also setting up a passwordless login to the particular machine. Besides that, RP needs to access to a MongoDB server. The server is used to store and retrieve operational data during the execution of an application using RP. The MongoDB server must be reachable from both, the host that runs the RP application and the target resource which runs the pilots.

Installation is very easy; RP package is available via PyPI and can be installed by pip. The MongoDB location is communicated to RP via an environment variable.

### Hands-on experience¶

Overall, writing an example workflow and executing it on a local host is relatively simple.

## Summary¶

Though RADICAL-Pilot comes with relatively decent documentation, offers a great deal of flexibility in writing workflows, and should integrate well with typical HPC environment. Despite those advantages it comes with two serious limitations. Firstly, it only supports data on file abstraction level, so data == files at this moment which may not integrate well with data butler used by LSST Stack. Secondly, though the developers claim to fix it in a near future, RP is not yet able to function in environment using Anaconda based Python distribution. Thus it has to be ruled out as a viable candidate for LSST Batch Production Services.

# Appendix H: Swift¶

## Overview¶

Swift is a simple, implicitly parallel functional language with C-like syntax, written in Java, and released on Apache Licence, ver. 2. Its latest stable version, 0.96.2, was released in August, 2015. From its white paper:

Swift is a scripting language designed for composing application programs into parallel applications that can be executed on multicore processors, clusters, grids, clouds, and supercomputers. Unlike most other scripting languages, Swift focuses on the issues that arise from the concurrent execution, composition, and coordination of many independent (and, typically, distributed) computational tasks. Swift scripts express the execution of programs that consume and produce file-resident datasets. Swift uses a C-like syntax consisting of function definitions and expressions, with dataflow-driven semantics and implicit parallelism.

## Features¶

### Flexibility¶

As a scripting language Swift offers a great deal of flexibility in designing workflows. Support for iterations and conditional executions allows for their dynamic generation.

### Monitoring¶

Swift offers no monitoring tools besides logs.

### Performance¶

According to its white paper, Swift is able to dispatch thousands (approx. 3000) of jobs per second sustaining high CPU utilization (80–95%).

### Portability¶

Remote execution and data transfer are provided by abstract interfaces called providers thus Swift execution model can be extended to new computational environments by implementing new data and/or execution providers. Currently implemented data providers supports protocols including direct local copying, GridFTP, HTTP, WebDAV, SCP, and FTP. Execution providers enable job execution by using POSIX fork, GlobusGRAM, Condor and Condor-G, PBS, SGE, SLURM, LSF, and SSH services. Swift workflows can be used on many remote computational resources including Beagle (UChicago), Blues (LCRC), Edison (NERSC), EC2 (Amazon), Open Science Grid, Stampede (XSEDE/TACC), Swan (Cray MPN). Complete list of supported sites can be found in this location.

### Reliability¶

There are three reliability mechanisms implemented by Swift runtime environment: retries, restarts, and replications. It means that Swift will automatically rerun a failing job completely reattempting site selection, stage-in, execution, and stage-out. It also keeps a restart log that encapsulates which function invocations have been successfully completed and may be skipped in subsequent Swift runs. If a job has been queued on a site for too long (based on a configurable threshold), a replica job is submitted. If any of those jobs begin executing, other replicas will be cancelled.

### Scalability¶

Swift easily scales in terms of size of the workflow and the resources. It can effectively run few tasks on a local machine as well as hundreds of thousands of jobs on thousand-core clusters.

## Support¶

There are also two mailing lists for dealing with project’s issues:

1. swift-user to report problems and receive assistance from the team and community,
2. swift-devel for contributors and people interested in discussing implementation issues.

Though in the latest project report from October 5, 2015, the authors admitted that many Swift users still communicate with the group through direct private conversations.

Note

Due to migration of services at UC Computation Institute they are offline at the time we were writing this review and may still remain inaccessible.

In the same report they stated that at least 20 scientific teams is using Swift in designing and executing their workflows. Those teams form a very broad spectrum of scientific activities including physical, biological, social, and computer sciences. Based on unique IPs accessing project’s site, the estimated user base is approximately 2400.

According to Dan Katz, at the moment of writing this review the project is not funded. However, the NSF grant proposal for its further development has just been recommended for funding.

## User experience¶

### Documentation¶

Swift comes with a really extensive documentation and numerous, more in-depth scientific papers presenting its design and various performance metrics. Though its tutorial is rather limited. It does not provide much introductory information but shows examples of Swift scripts for various basic workflow. Granted, with C-like syntax getting the general idea is rather easy but some Swift specific constructs are presented without any explanation. Ironically, we found reading the user guide which explains Swift’s syntax and data types much more approachable.

### Installation¶

Swift is written in Java and depends on Java 1.7 or greater. Though not strictly required, the authors recommend Oracle Java.

• an MPI implementation (MPICH, OpenMPI, etc.),
• Tcl 8.6,
• Software development tools: gcc, make, SWIG,
• optionally, ZSH for the build script.

Installation is very simple. Swift can be installed either by downloading precompiled binary distribution or compiling its source though for that Apache Ant has to be installed in addition to JDK.

Currently there is no binary distribution for Swift/T. It has to be compiled from the source.

### Hands-on experience¶

There is a significant overhead with running even simple custom workflows as a user must firstly learn Swift syntax and understand it language model. But once those aspects are more or less understood, executing a workflow on distributed computer resources is rather simple.

## Summary¶

Swift seems to be a feature-rich, thoughtfully designed piece of software with well written, extensive documentation, and quite thoroughly tested in scientific settings. It major drawback comes from the fact that there is rather steep learning curve associated with getting acquainted with its language model. There is also a minor issue of lacking any high-level monitoring tools. However, it still seems to be a viable candidate for a workflow management system to be used in LSST Batch Operation Services.

This review focuses mainly on Swift/K version which runs on the Karajan grid workflow engine and is designed to execute a workflow of program executions across wide area resources, exploiting diverse schedulers (PBS, Condor, etc.) and data transfer technologies. There exists also a newer implementation of the Swift language for high-performance computing called Swift/T. In this implementation, the Swift script is translated into an MPI program that uses the Turbine and ADLB runtime libraries for highly scalable dataflow processing over MPI on a single large system. Comparing to Swift/K it offers:

• Enhanced performance: 1.5 billion tasks/s;
• Ability to call native code functions (C, C++, Fortran);
• Ability to execute scripts in embedded interpreters (Python, R, Tcl, Julia, etc.);
• Enhanced built-in libraries (string, math, system, etc.).

Its latest stable version, 0.8.0, was released in April, 2015.

According to Daniel Katz, once ability to execute tasks at different computational sites is implemented in Swift/T, Swift/K version will be deprecated. At the same time, Swift scripts written for Swift/K should require little to no modifications to be able to run with Swift/T.

Swift/K uses Globus as middleware to talk to various resources.

# References¶

 [CONOPS] (1, 2, 3) Concept of Operations for the LSST Batch Production Services (http://ls.st/cr4).
 [XSEDE] Workflow Systems on XSEDE, tutorial presented at XSEDE15 and XSEDE16 (https://sites.google.com/site/xsedeworkflows/).