Flexibility in workflow generation¶

Workflow management systems use wide variety of methods to create workflows; some uses dedicated GUIs, others follow “configuration as a code” approach and a workflow description is a script written in Python. Those methods offer different degrees of freedom and flexibility for workflow’s authors and therefore we consider this aspect an important characteristic of a workflow management system.

Interoperability¶

As we already mentioned in Introduction some of the existing workflow management systems were designed to work in certain “ecosystems”. As the LSST project’s execution environments may differ from the one a WMS was designed for, it is essential to know what technologies a given system can cooperate with currently or how easy it can be extended to support others.

Scalability¶

As workflows varies in size and complexity, it is important that a workflow management systems is able to handle both small and large workflows (in terms of number of tasks) equally well, e.g., that memory requirements grows sufficiently slow with increasing number of tasks in a workflow.

Performance¶

Though scheduling tasks and tracking their provenance is not a particularly CPU-intensive process, we wanted to be sure that under large load a workflow management system will not become a bottleneck and will utilize available resources efficiently.

Reliability¶

With large number of tasks running on distributed computational resources their failures are inevitable. Thus any decent workflow management system should support automatic retries, i.e., rerun the failing task, to be able to deal with transient errors (e.g. a temporary network downtime). It should also track tasks’ provenance to allow for a workflow restart in case of more permanent execution errors (e.g. runtime errors due to flawed input).

Monitoring tools¶

Workflow management systems provide logs which can be used to monitor tasks execution. However, with numerous large and complex workflows, going through logs is not a particularly useful method to monitor workflow execution in a real time. That is why in our survey we decided to take into account existence of any extra monitoring tools such as dashboards.

Flexibility¶

Airflow’s workflows are specified as a Python code which offers a great deal of flexibility when designing workflows. On top of that, Airflow supports Jinja templating. Thus workflows can be generated programmatically. Additionally, it has a simple plugin manager that can integrate external features into its core if required.

Monitoring¶

Airflow comes with the web dashboard allowing a user to visualize, manage and monitor execution of available workflows.

Performance¶

We were not able to find any solid performance metrics of Airflow.

Interoperability¶

Airflow comes fully loaded with ways to interact with commonly used systems like Hive, Presto, MySQL, HDFS, Postgres and S3. Sources I came across do not mentioned more HPC-oriented technologies/protocols though, in theory, it may be easily extensible due to its modular architecture.

Reliability¶

Airflow supports retrying failed tasks. However, it seems that workflow restarts are not directly supported although this blogpost describes how to use of subdags to emulate them.

Scalability¶

Project’s website claims that due to its modularity Airflow can orchestrate arbitrary number of tasks and workers. In their own words:

Airflow is ready to scale to infinity.

Sadly enough, they do not provide any solid data to support this bold claim. One example we were able to find, mentions running 20’000 tasks per day on a 32-core machine which is not particularly impressive from a HPC perspective.

Documentation¶

Airflow comes with a rather well written tutorial explaining how to create, validate and execute an example workflow. However, the user guide and API reference are somewhat lacking. Information is provided without a clear order and a level of details one would expect from a user guide. Some more advanced aspects (e.g. writing hooks) are not covered at all. Probably the worst is the chapter regarding, rather feature-rich, Airflow’s web UI. It is basically a catalog of screenshot focused more on displaying some its features rather than explaining how to work with it in some organized manner.

Installation and configuration¶

Besides a working Python installation, the project does not list any external dependencies. Though there are other extra packages available (their complete list can be found here) and they may required some additional software to work.

Airflow installation is pretty straightforward as it can be installed via PyPI using pip. Though initially, recommended installation procedure

pip install airflow

did not allow us to start the scheduler. We had to install Hive operators with

pip install "airflow[hive]"

to make it work.

Hands-on experience¶

The learning curve does not seem to be particularly steep and simple workflows can be set up with minimal overhead. Though, out of the box, Airflow uses very simplistic, sequential task scheduler. Using more advanced ones (allowing to scale up number of workers) requires setting up a real database backend (e.g. MySQL or Postgres) separately. If workers are to be executed on remote sites, setting up a message broker (e.g. Rabbitmq or Redis) and Celery queue on top of that is required.

Flexibility¶

Specifying workflows with the help of a YAML-like language offers a great deal of flexibility in their creation. Support for loops and conditional executions allows for their dynamic generation too.

Monitoring¶

An execution log file stores all events, and users can subscribe to events, but CloudSlang does not appear to provide monitoring tools or GUI to visualize the workflow execution.

Performance¶

Available sources do not provide any data regarding CloudSlang performance metrics.

Interoperability¶

Apparently, the entire model of the CloudSlang engine assumes that it is being run on a single machine with shared filesystem so in-memory objects like variables can be transferred between actions and flows but beyond that it does not support any forms of data transfer. Additionally, actions and flows are executed exclusively by the engine so their execution cannot be delegated to Condor, PBS, SLURM, etc.

Reliability¶

The recovery mechanism guarantees that each step of an execution plan will be run, by monitoring status of available workers. The non-responsive workers’ records in the queue get reassigned to other workers that pick up from the last known step that was executed. However, workflow restarts are not implemented.

Scalability¶

Project’s documentation claims that CloudSlang workflows optimized for high throughput and are horizontally scalable but they do not provide any information how big workflows it can really handle.

Documentation¶

CloudSlang comes with an well written documentation which includes an extensive tutorial. The tutorial provided a nice guidance how to create a simple action and a flow and gently introduced more advanced aspects of YAML allowing to create arbitrary complex workflows.

Installation¶

CloudSlang is written in Java and depends on Java 7 or greater. Additionally, to compile the project from its source Apache Maven >= 3.0.3 should be available.

Installation is simple, CloudSlang can be installed either by downloading precompiled binary or compiling from the source. Docker images with CloudSlang are also available.

Hands-on experience¶

Due to CloudSlang limitations (see Summary for details) our testing did not go beyond examples from tutorial. Though, it seems, even complex workflows can be created with minimal overhead.

Flexibility¶

The syntax of MakeFlow’s workflow specification is similar to the traditional UNIX Make program, but more restrictive because only plain rules are allowed, for example no pattern rules, wildcards, or special variables. A Makeflow script consists of a set of rules, each specifying a set of target files to create, a set of source files needed to create them, and a command to run. Makeflow relies on the file (or directories) dependencies explicitly stated in the rules. There seems to be no support for dynamic workflows.

Monitoring¶

MakeFlow does not offer any monitoring tools besides logs. However when installed with Work Queue, a dashboard for monitoring tasks currently being executed is available.

Performance¶

Some benchmark performance tests were done in Albrecht et al. (2012), with several workflow patterns with small and large files, in multiple environments including Work Queue, Condor, Hadoop, SGE, and local. For most tests, Makeflow used with Work Queue achieved better performance than other environments.

Interoperability¶

Makeflow can be deployed to a variety of different systems without modification, including local execution on a single multicore machine as well as batch systems like HTCondor, Work Queue, Sun Grid Engine (SGE) or its derivatives, Torque, Slurm, Moab, Chirp, or custom-defined batch submission commands. For some batch systems, a distributed file system shared across all nodes of the cluster is assumed. Though MakeFlow handles data transfer and caching in supported systems if used with Work Queue.

Reliability¶

By maintaining a transaction log, Makeflow can resume a workflow and continue from where it left off.

Scalability¶

Makeflow can handle \(O(10^{6})\) jobs, with \(O(1000)\) of jobs concurrently, according to their 2012 paper. By using Makeflow with Work Queue, one can distribute the load across a collection of machines.

Documentation¶

Makeflow comes with a user manual and tutorial that introduce the Makeflow language and basic usages of the tools. These documentations make starting easy; however the documentation on more advanced features seems to be scarce.

Installation¶

Installation of cctools is straightforward, using make. There is no need to download any external packages.

Hands-on experience¶

Overall we found Makeflow lightweight and easy to start. It is easy to create a simple workflow with everything hard-coded. One uses executables and files like in shell scripts. However, Makeflow does not support workflows to be made programmatically.

Flexibility¶

It might be emphasized that PanDA is foremost a workload management system concerned with the large scale management of jobs to a possibly diverse collection of computing sites. PanDA does support workflow with the combination of JEDI and DEFT, though we might note that these are relatively recent additions. They utilize GraphML format to represent a DAG corresponding to a given workflow.

Monitoring¶

PanDA comes with a web based monitoring and browsing system that provides an interface to PanDA for users and operators.

Performance¶

PanDa proved itself as a scalable and reliable system capable of handling very large workflow. In 2014, PanDA processed about a million jobs per day, with about 150,000 jobs running at any given time.

Interoperability¶

The BigPanDA project was established in 2012 as an extension of the PanDA project. BigPanDA strives to generalize PanDA for use by other experiments, and to extend its scope to computing clouds and HPC platforms.

Reliability¶

Scalability¶

PanDA is capable of handling very large workflows.

Documentation¶

PanDA comes with an exhaustive documentation describing architecture of all its numerous components. However, it does not provide any tutorial or a user guide. Thus, taking into account project’s scale, there is a significant overhead for newcomers to overcome just to get started with it.

Installation¶

At this point, we did not installed PanDA locally. Though taking into account that it is a complex, multicomponent systems its installation process may not be trivial.

Hands-on experience¶

See above.

Flexibility¶

Pegasus provides a good deal of flexibility in the generation of workflows. They can be created programmatically with help of provided Java, Perl, and Python API which allows to easily write DAX files in XML format. Users can embed a sub workflow inside a workflow, by either specifying a DAX Job in the DAX, or specifying a DAG Job in the DAX.

Interoperability¶

Pegasus execution environments can be Condor pools, Grid infrastructures such as Open Science Grid and XSEDE, Amazon EC2, Google Cloud, and many campus HPC clusters.

Monitoring¶

Pegasus has a runtime monitoring daemon that monitors the running workflow, parse the workflow job, and task logs and populates them into a workflow database. The database stores both performance and provenance information. It provides tools for monitoring status, debugging and diagnosing failures, showing summary and statistics, with web dashboard and plots.

Performance¶

The Pegasus mapper can reorder, group, and prioritize tasks in order to increase overall workflow performance. Most optimization is done at compile time workflow restructuring. Runtime optimization relies on hierarchical workflow, as the sub-workflows are mapped and planned just-in-time, so mapping and execution interleaves, and allow some dynamic workflows. Full features of dynamic planning may be added in the future.

Reliability¶

Jobs and data transfers are automatically retried in case of failures. Debugging tools such as pegasus-analyzer help the user to debug the workflow in case of non-recoverable failures.

When errors occur, Pegasus tries to recover when possible by retrying tasks, by retrying the entire workflow, by providing workflow-level checkpointing, by re-mapping portions of the workflow, by trying alternative data sources for staging data, and, when all else fails, by providing a rescue workflow containing a description of only the work that remains to be done.

Scalability¶

Pegasus can scale both the size of the workflow, and the resources that the workflow is distributed over. Pegasus runs workflows ranging from just a few computational tasks up to* 1 million* (using sub-DAG). The number of resources involved in executing a workflow can scale as needed without any impediments to performance.

Documentation¶

The beginner tutorial is nice, but documentation on features beyond the tutorial level vary and some seem difficult to find. Nonetheless one can learn from the available examples.

Installation¶

Pegasus lists Condor and OpenSSL as its dependencies. There are binary distribution available or it can be build from scratch using ant.

Building it from the source using ant is straightforward. Though we failed to build it on Mac due to an error related to OpenSSL and we didn’t attempt to solve the problem, compiling it on Centos went fine.

Hands-on experience¶

For trivial jobs, the overhead seems high. We made a much simplified mock-up of the first stage of LSST DRP processing. Pegasus’ own tools cannot visualize a workflow unless it is executed successfully; this makes debugging DAX construction difficult for beginners. Besides linking the input/output, the job dependency needs to be specified explicitly.

Execution environment¶

The Pegasus planner needs 4 components to map out the plan and make a concrete workflow over the resources:

• DAX: the abstract workflow description in XML, generated by Pegasus API
• Site catalog: describes the execution environment
• Replica catalog (files): specifies locations of input data
• Transformation catalog (executables): specifies locations of software used by the workflow

Pegasus relies on Condor DAGMan as the workflow execution engine.

MPI support¶

There is a tool pegasus-mpi-cluster (PMC) to run DAG on some HPC systems. PMC is designed to work either as a standalone tool or as a complement to Pegasus. In the mode that the entire workflow is managed by PMC (PMC-only mode), DAGMan and Condor are not required. Or, PMC can be used as the wrapper for executing clustered jobs in Pegasus. In this mode Pegasus groups several tasks together and submits them as a single clustered job to a remote system.

Documentation¶

Besides project’s README file, Pinball’s documentation is practically non-existent.

Installation¶

Pinball requires:

• Graphviz: a graph visualization software,
• libmysqlclient: MySQL client libraries.

Pinball uses MySQL database as persistent storage so an access to a running MySQL instance is also required.

Installation is very easy; Pinball package is available via PyPI and can be installed by pip.

Hands-on experience¶

Taking into account overall impression (lack of decent documentation and support), we did not even try to setup an example workflow with Pinball.

Flexibility¶

As RP is a library, its workflow specifications are Python scripts. Thus RP can be used to execute (theoretically) arbitrarily complex workflows and generate them dynamically. Jobs can be single or multi-threaded; MPI applications are also supported.

Monitoring¶

Each compute unit has a metadata associated with it which can be inspected and used to monitor a unit execution. However, RP do not provide any monitoring tools besides logging.

Performance¶

We were not able to find any metrics regarding rate of dispatching jobs or sustainable CPU utilization levels.

Interoperability¶

RP is built on top of The Simple API for Grid Applications (SAGA, also being developed by RADICAL), a high-level API for accessing distributed resources, meaning that RP should work on a variety of backends such as PBS, Condor and Condor-G, SGE, SLURM, LSF or Amazon EC2. Data transfer is supported via local copying, SFTP, and GSISFTP protocols.

Reliability¶

Each compute unit progress through a certain state model and failed units can be easily detected. However, RP documentation does not provide examples of any recovery mechanisms beyond simple retries. As far as we can tell workflow restarts or task replications are not possible.

Scalability¶

According to this source, RP can marshal O(10k) distributed cores and schedule O(10k) tasks in a single application context across multiple distributed HPC resources.

Documentation¶

Documentation includes a tutorial, with ready to run example scripts, introducing various aspects of working with RP, e.g., setting up a session, monitoring jobs, staging input and output data. However, one have to at least scheme the user guide anyway to be already familiar with core RP concepts to understand what is going on. Also, rather annoying thing is that documentation is not entirely in sync with the code base. We encountered few cases when following documentation verbatim led to execution errors and applying “Use the Force, Read the Source” principle was necessary.

Installation¶

RP requires the following packages:

• Python >= 2.7,

• Virtualenv >= 1.11,

• Pip == 1.4.1

  Warning

  They really mean it! Newer versions of pip fail to install RP properly without emitting any errors.

Using RP on remote machines requires also setting up a passwordless login to the particular machine. Besides that, RP needs to access to a MongoDB server. The server is used to store and retrieve operational data during the execution of an application using RP. The MongoDB server must be reachable from both, the host that runs the RP application and the target resource which runs the pilots.

Installation is very easy; RP package is available via PyPI and can be installed by pip. The MongoDB location is communicated to RP via an environment variable.

Hands-on experience¶

Overall, writing an example workflow and executing it on a local host is relatively simple.

Flexibility¶

As a scripting language Swift offers a great deal of flexibility in designing workflows. Support for iterations and conditional executions allows for their dynamic generation.

Monitoring¶

Swift offers no monitoring tools besides logs.

Performance¶

According to its white paper, Swift is able to dispatch thousands (approx. 3000) of jobs per second sustaining high CPU utilization (80–95%).

Portability¶

Remote execution and data transfer are provided by abstract interfaces called providers thus Swift execution model can be extended to new computational environments by implementing new data and/or execution providers. Currently implemented data providers supports protocols including direct local copying, GridFTP, HTTP, WebDAV, SCP, and FTP. Execution providers enable job execution by using POSIX fork, GlobusGRAM, Condor and Condor-G, PBS, SGE, SLURM, LSF, and SSH services. Swift workflows can be used on many remote computational resources including Beagle (UChicago), Blues (LCRC), Edison (NERSC), EC2 (Amazon), Open Science Grid, Stampede (XSEDE/TACC), Swan (Cray MPN). Complete list of supported sites can be found in this location.

Reliability¶

There are three reliability mechanisms implemented by Swift runtime environment: retries, restarts, and replications. It means that Swift will automatically rerun a failing job completely reattempting site selection, stage-in, execution, and stage-out. It also keeps a restart log that encapsulates which function invocations have been successfully completed and may be skipped in subsequent Swift runs. If a job has been queued on a site for too long (based on a configurable threshold), a replica job is submitted. If any of those jobs begin executing, other replicas will be cancelled.

Scalability¶

Swift easily scales in terms of size of the workflow and the resources. It can effectively run few tasks on a local machine as well as hundreds of thousands of jobs on thousand-core clusters.

Documentation¶

Swift comes with a really extensive documentation and numerous, more in-depth scientific papers presenting its design and various performance metrics. Though its tutorial is rather limited. It does not provide much introductory information but shows examples of Swift scripts for various basic workflow. Granted, with C-like syntax getting the general idea is rather easy but some Swift specific constructs are presented without any explanation. Ironically, we found reading the user guide which explains Swift’s syntax and data types much more approachable.

Installation¶

Swift is written in Java and depends on Java 1.7 or greater. Though not strictly required, the authors recommend Oracle Java.

Swift/T (see Comments section for more details) additionally will require:

• an MPI implementation (MPICH, OpenMPI, etc.),
• Tcl 8.6,
• Software development tools: gcc, make, SWIG,
• optionally, ZSH for the build script.

Installation is very simple. Swift can be installed either by downloading precompiled binary distribution or compiling its source though for that Apache Ant has to be installed in addition to JDK.

Currently there is no binary distribution for Swift/T. It has to be compiled from the source.

Hands-on experience¶

There is a significant overhead with running even simple custom workflows as a user must firstly learn Swift syntax and understand it language model. But once those aspects are more or less understood, executing a workflow on distributed computer resources is rather simple.