Build and Deploy

The statement has been made many times that a paradigm should be easy to start, easy to extend, and still be scalable to great levels. To accomplish this the deployment of code also needs to be taken into account.

The Go programming language has one great double edged sword. A feature that makes Go, but also a feature that limits Go. This is the descision to make all go projects statically compile. Being able to freeze software into a simple, portable binary is a deeply powerful thing. But it is seen by many as a limitation.

I continually insist on having the best of both worlds. The pop library and paradigm have been built on Python. This choice came for many reasons. One reason is because Python is so easy, extensible and popular; another is because the dynamic nature of loading code in Python lends itself to building powerful plugin interfaces.

But Python lacks a reliable way to deploy code like Go does. Even after 19 years Python code is still difficult to deploy to users.

But not anymore.

Pop Build

One of the core pop projects is a tool called pop-build. This tools allows you to create a single binary from your pop project that can be deployed to other similar platforms.

This means that it is easy to create, for the first time, deployable Python projects that can run on multiple systems. Pop Build bundles Python and all deps into the single binary, meaning that you don’t need your application to target multiple versions of Python. It also means that you don’t need to worry about version dependencies either. Pop Build delivers your deps with your binary, just like Go does.

This also means that pop projects DON’T need to deploy with Pop Build. They can still deploy as Python projects.

The best of both worlds. Static compiled deliverables, but still the ability to execute dynamic code and develop quickly!

Using Pop Build

The main idea behind Pop-Build is to make building python projects as a single frozen binary easy. Therefore using pop-build just takes a few steps.

We can start with an existing python application. We will assume that your python project has a standard setup.py and requirements.txt.

The only other thing that most projects will need is a run.py.

The run.py

Since Pop-Build creates a single binary, it needs a single entry point. This entry point is defined in the file run.py. This is just a simple python script that is used to start your application. The contents of the run.py is typically the same code that is used in your setuptools entry point, or the startup script used in distutils. There is nothing special about the run.py, pop-build just needs an entry point that is clean python.

A typical run.py will look like this:

import myapp

def main():
    myapp.start()

main()

Just some good old python! If you are building a pop project then pop-seed will create the run.py for you.

Run Pop Build

Thats right! Except for a run.py, your python project likely already has everything needed! Pop Build uses the setup.py and requirements.txt files to build the environment used.

So assuming you have a standard python project defined, all you need to do is cd to that directory and run pop-build, in this example we will assume the application is called foo:

pop-build -n foo

This will kick off the process and the resulting binary will be placed in dist/foo

Now that the binary is available it can be directly called.

What Happened?

Pop-Build starts with the version of python that you exceuted pop-build with. This python is the python that will be embeded in your binary. Next it creates a venv for your application. With the venv in hand, Pop-Build populates it. The venv is populated with all of the deps that are defined as requirements for the main application, including the application itself.

Now that the venv has been set up, we tell PyInstaller to create a binary from the run.py. But PyInstaller is all about building a binary from all of the imports that come from the run.py. This is done to build a small binary and include only the most required code. But this is not the case for many applications. It is typical that things are late imported and the application assumes that a larger python environment is available. It is also typical that extra files are needed by the application and are typically added via the setup.py.

Instead of following the imports, Pop-Build bundles the venv into the binary created by PyInstaller. This means that you have a strong assurance that the full, needed environment is available. This does make a larger binary, but it allows for a much easier and reliable build. Also, the binary is typically not much bigger.

Using the Build Addon

Many python projects require C libraries. How is it then, that the dynamic libs can be added to the final binary? Pop-Build has an answer to this.

When running pop-build we can use a configuration file. This file allows for any option that would be passed on the cli to be defined, but also to define the routines for external builds.

A Pop-Build config that adds the library libsodium looks like this:

build:
  libsodium:
    sources:
        - https://download.libsodium.org/libsodium/releases/LATEST.tar.gz
    make:
        - tar xvf LATEST.tar.gz
        - cd libsodium-stable && ./configure && make
    src: libsodium-stable/src/libsodium/.libs/libsodium.so
    dest: lib/

This example shows how we can define a library to download and build, then the src which is relative to the root of the build and the dest which is relative to the root of the venv.

The src can be a directory or a list of files. The dest is just a single directory to store the files.