Build and Deploy

The statement has been made many times that a paradigm should be easy to start, easy to extend, and still be scalable to great levels. To accomplish this, the deployment of code also needs to be taken into account.

The Go programming language has one great double edged sword. A feature that makes Go, but also a feature that limits Go. This is the decision to make all go projects statically compile. Being able to freeze software into a simple, portable binary is a deeply powerful thing. But it is seen by many as a limitation.

I continually insist on having the best of both worlds. The pop library and paradigm have been built on Python. This choice came for many reasons. One reason is because Python is so easy, extensible and popular. Another is because the dynamic nature of loading code in Python lends itself to building powerful plugin interfaces.

But Python lacks a reliable way to deploy code like Go does. Even after 19 years Python code is still difficult to deploy to users.

But not anymore.

Tiamat

One of the core pop projects is a tool called tiamat. This tool allows you to create a single binary from your pop project that can be deployed to other similar platforms.

This means that, for the first time, it is easy to create deployable Python projects that can run on multiple systems. Tiamat bundles Python and all dependencies into a single binary, meaning that you don’t need your application to target multiple versions of Python. It also means that you don’t need to worry about version dependencies either. Tiamat delivers your dependencies with your binary, just like Go does.

This also means that pop projects DON’T need to deploy with Tiamat. They can still deploy as Python projects.

The best of both worlds. Static compiled deliverables, but still the ability to execute dynamic code and develop quickly!

Using Tiamat

The main idea behind Tiamat is to make building Python projects as a single frozen binary easy. Therefore using Tiamat just takes a few steps.

We can start with an existing Python application. We will assume that your Python project has a standard setup.py and requirements.txt.

The only other thing that most projects will need is a run.py.

The run.py

Since Tiamat creates a single binary, it needs a single entry point. This entry point is defined in the file run.py. This is just a simple Python script that is used to start your application. The contents of the run.py is typically the same code that is used in your setuptools entry point, or the startup script used in distutils. There is nothing special about the run.py, tiamat just needs an entry point that is clean Python.

A typical run.py will look like this:

import myapp

def main():
    myapp.start()

main()

Just some good old Python! If you are building a pop project, pop-create will create the run.py for you.

Run Tiamat

That’s right! Except for a run.py, your Python project likely already has everything needed! Tiamat uses the setup.py and requirements.txt files to build the environment used.

So, assuming you have a standard Python project defined, all you need to do is cd to that directory and run tiamat. In this example, we will assume the application is called foo:

tiamat -n foo

This will kick off the process and the resulting binary will be placed in dist/foo

Now that the binary is available, it can be directly called.

What Happened?

Tiamat starts with the version of Python that you executed tiamat with. This Python is the Python that will be embedded in your binary. Next it creates a venv for your application. With the venv in hand, Tiamat populates it. The venv is populated with all of the dependencies that are defined as requirements for the main application, including the application itself.

Now that the venv has been set up, we tell PyInstaller to create a binary from the run.py. But PyInstaller is all about building a binary from all of the imports that come from the run.py. This is done to build a small binary and include only the most required code. But this is not the case for many applications. It is typical that things are late imported and the application assumes that a larger Python environment is available. It is also typical that extra files are needed by the application and are typically added via the setup.py.

Instead of following the imports, Tiamat bundles the venv into the binary created by PyInstaller. This means that you have a strong assurance that the full, needed environment is available. This does make a larger binary, but it allows for a much easier and reliable build. Also, the binary is typically not much bigger.

Using the Build Addon

Many Python projects require C libraries. How is it, then, that the dynamic libs can be added to the final binary? Tiamat has an answer to this.

When running tiamat, we can use a configuration file. This file allows for any option that would be passed on the cli to be defined, but also to define the routines for external builds.

A Tiamat config that adds the library libsodium looks like this:

build:
  libsodium:
    sources:
        - https://download.libsodium.org/libsodium/releases/LATEST.tar.gz
    make:
        - tar xvf LATEST.tar.gz
        - cd libsodium-stable && ./configure && make
    src: libsodium-stable/src/libsodium/.libs/libsodium.so
    dest: lib/

This example shows how we can define a library to download and build, then the src which is relative to the root of the build and the dest which is relative to the root of the venv.

The src can be a directory or a list of files. The dest is just a single directory to store the files.