With ephemeral EMR we have to bootstrap our cluster with our application requirements before every single run of a PySpark application. This should be a simple pip install of our PySpark application package. But the reality couldn’t be further from simple.
- We need to install and update packages on EMR because the default installations lack behind many versions.
- We should create a virtual Python environment for our application as global Python packages can interfere in weird ways. We all have been there.
- We need to install our PySpark application and requirements with pip
- We need to copy assets from S3, e.g. the main.py script for spark-submit
- Finally, we call spark-submit with our application-specific Spark configurations to finally run our workload
This approach to EMR has a few pain points:
- The bootstrap process is a lot of paid for cluster-time while we do not actually run any real workload. This will add up in particular for big clusters!
- We have to maintain and download assets which are not part of our Python application package, e.g. the spark-submit main.py script
- We have to maintain application-specific Spark configurations for our jobs independently of our Python application package which we pass as arguments to the spark-submit script. These usually end up scattered across Jenkins pipelines or Airflow DAGs adding unnecessary complexity to maintaining PySpark applications.
- Packaging and installing a Python application with complex dependencies is leading you straight to the notorious dependency hell.
Considering how popular Python is, the Python toolchain is far from ideal and installing packages with complex dependency chains with pip ends often in the notorious dependency hell. Unfortunately, pip lacks a powerful dependency resolver.
In commercial data science teams there is usually a second layer of complexity with private package indexes for proprietary code. And I bet that your private package index is slow. Therefore, a common EMR bootstrap script for your PySpark application might look like this:
Wouldn’t it be great if running a PySpark application was as simple as just calling an executable? No pip install of requirements. No main.py. No spark-submit with Spark memory configurations cluttering up Jenkins or Airflow.
PEX (Python EXecutable) is a file format and associated tools to create a general purpose Python environment virtualization solution similar to virtualenv. PEX was originally developed at Twitter in 2011 to deploy Python applications to production. PEX files are self-contained executable Python virtual environments. The emphasis is on self-contained and executable which makes PEX files ideal for application deployment to production environments. With PEX files the only step required to deploy an application is to copy the file. No pip install and no PATH modifications required.
With the help of PEX running a PySpark application on EMR no longer requires any bootstrapping!
- PEX greatly simplifies running PySpark applications and
- saves money by utilising the cluster to run our actual application workload much earlier without the need of any cluster bootstrapping.
To create a PEX archive you use the pex utilities. You can simply install it with
pip install pex
When you do this within a new Python virtual environment, you can use pex to package itself. The following command creates a pex file containing pex and requests, using the console script named “pex”. Save the created executable to ~/bin/pex and you can use pex in or outside of any virtualenv like any other executable on your PATH.
pex pex requests -c pex -o ~/bin/pex
There is one complication with PEX: the executable contains a self contained Python virtual environment but not the Python interpreter itself: the PEX executable is platform dependent. Currently, EMR runs Python 3.6.10 on Linux and you might develop on a Mac. Therefore, it’s generally best practise to use Docker to create reproducible results.
Build a docker image compatible with EMR to create your PEX archives within a docker container:
FROM python:3.6-slim-busterRUN apt-get update && apt-get install -y --no-install-recommends
&& rm -rf /var/lib/apt/lists/*RUN pip3 install pexRUN mkdir /app
WORKDIR /appENV PATH=/root/.local/bin:$PATH
If your organisation uses a private package index, e.g. Artifactory, then PEX shows another weakness: at the time of writing it does not expose parameters for the requests library via the CLI which means we cannot set a custom network timeout for pip when using PEX directly to resolve package dependencies. A workaround is to use a wheelhouse. The following script can be used to build a pex archive using a wheelhouse and a private package index with a custom timeout:
#!/usr/bin/env bashpip3 download -r requirements.txt
--timeout 120pex . -r requirements.txt
Our aim is to package a fully self-contained PySpark application and run it without the need of spark-submit. Therefore, our Python main function has to create a SparkSession:
if __name__ == "__main__":pex_file = os.path.basename([path for path in sys.path if path.endswith(".pex")])
os.environ["PYSPARK_PYTHON"] = "./" + pex_filespark = (
You can pass any option from your usual spark-submit to the SparkSession builder.
This allows you to execute your PySpark application inside a PEX executable for example like this:
./my_application.pex -m my_application.main
The final step is to execute our pex application as an EMR step. We will use the script-runner and a generic thin wrapper to execute a PEX as a step.
The script-runner calls our thin wrapper which pulls a PEX file from S3 and executes it with all environment variables and command line arguments we might need. The following script is a thin wrapper called pex-executor.sh you can use. Simply put it on S3 to make it available to your EMR cluster:
# Author: Jan Teichmann
# Version: 2020-02-10
# Wrapper to execute a PEX archive via an EMR Step
# Step type: Custom JAR
# JAR location: s3://eu-west-1.elasticmapreduce/libs/script-runner/script-runner.jar
# ./some-etl-job.pex -m package.module -fromdate=2020-04-20 -todate=2020-04-22aws s3 cp $1 .
chmod +x $(basename -- $1);shift;
Now you can submit an EMR step, e.g. via Airflow’s EmrAddStepsOperator:
"Name": "Run my_application Step",
As you can see above, we first pass two s3 paths. The first to our thin wrapper called pex-executor.sh which will be executed by AWS script-runner.jar. The pex-executor script in turn will download the application pex executable.
We also define two environment variables HADOOP_HOME and SPARK_HOME which are needed for EMR. You could add any additional environment variables you might want.
We then pass the name of the executable and pass any CLI parameters for our Python application.
PEX allows us to run PySpark applications as fully self-contained executables just like a Spark application with an uber-JAR or fat-JAR would allow were we to use the Scala API.
This greatly simplifies the use of ephemeral EMR clusters with PySpark and saves time and saves money as we do not have to bootstrap the cluster.
Wrap the creation of your pex executables into a Jenkins pipeline and you have a powerful DevOps pattern to build packages and upload them to S3 for deployment.
You can then schedule your PySpark applications for example with Airflow. Because your PEX applications are fully self-contained you will be able to create extremely generic DAGs in Airflow without that application any application logic and configurations end up scattered across multiple platforms and repositories.
Simplicity is the ultimate sophistication