Patterns¶

Before we start into patterns, it is necessary to own that there are many new concepts and terms that are being introduced in Plugin Oriented Programming. It is, after all, a new programming paradigm!

One of the goals of Plugin Oriented Programming is to express the paradigm in as few concepts as possible. If you recall some of the rules from earlier on, they stated over and over how critical it is that code can be transferred to new people easily.

Similarly the paradigm needs to be easy to understand. So before you start to say “Hubs, Subs, and Patterns! Oh My!”, let me assure you that there are only a few more concepts to learn, and that once you use them just a couple of times they quickly start to make sense.

Patterns are critical to Plugin Oriented Programming. When making pluggable software the ways that plugins are applied needs to be consistent and understandable. This makes the software pluggable not just because it exists in a plugin framework, but truly pluggable by design.

Patterns Are Extensible¶

If software is not written in an extensible and pluggable way then even if it is built inside of a plugin framework, that does not make the software pluggable.

When making a Sub consider how the interface can be extended, and created, via the additions of plugins. Different established patterns do this in different ways.

Let’s take a look at a few of the patterns that exist in the wild to see how these problems are commonly dealt with.

Collection Pattern¶

The Collection Pattern is used to collect data. It is useful when downloading and combining data from multiple sources, or scanning hardware for specific properties. A great example for this pattern is a little application called “Grains” that is used to collect system data.

The grains system creates a common and simple pattern. It runs all of the exposed functions found in the grains Sub and those functions in turn extend a data structure found on the hub.

Let’s take a look at the init.py file for grains:

# Import python libs
import asyncio


def __init__(hub):
    # Set up the central location to collect all of the grain data points
    hub.pop.sub.load_subdirs(hub.grains, recurse=True)
    hub.grains.GRAINS = {}


async def collect(hub):
    """
    Collect the grains that are presented by all of the app-merge projects that
    present grains.
    """
    await hub.grains.init.run_sub(hub.grains)
    # Load up the subs with specific grains
    await hub.grains.init.process_subs()


async def run_sub(hub, sub):
    """
    Execute the contents of a specific sub, all modules in a sub are executed
    in parallel if they are coroutines
    """
    coros = []
    for mod in sub:
        if mod.__sub_name__ == "init":
            continue
        for func in mod:
            ret = func()
            if asyncio.iscoroutine(ret):
                coros.append(ret)
    for fut in asyncio.as_completed(coros):
        await fut


async def process_subs(hub):
    """
    Process all of the nested subs found in hub.grains
    Each discovered sub is hit in lexicographical order and all plugins and functions
    exposed therein are executed in parallel if they are coroutines or as they
    are found if they are natural functions
    """
    for sub in hub.pop.sub.iter_subs(hub.grains, recurse=True):
        await hub.grains.init.run_sub(sub)

The init Function¶

The __init__ function is simple, as it should be. It recursively loads nested Subs that have been created and sets up a dict on the hub inside of the grains namespace. Since the GRAINS dict is all caps we know it is not a function or a Sub, it is a variable. The GRAINS dict is also located under hub.grains. This means that all plugins created on the Sub are intended to have write access to the hub.grains.GRAINS variable.

The run_sub Function¶

Next we have the meat of the pattern. The run_sub function is simple. It just iterates over all of the plugins exposed in a Sub and calls them! If they are a coroutine then they get awaited in a batch. But the end result is simple, any plugin that is added will be found, and all functions will be called when the pattern is executed.

A Grains Plugin¶

Now that we have the pattern down, let’s say that we add another file to the Sub called test.py:

async def test(hub):
    hub.grains.GRAINS["test"] = True

An Extensible Pattern¶

We now have a simple pattern! The test function will be found and executed inside of the run_sub function, and that function in turn obeys the interface and adds the desired data onto the hub.grains.GRAINS dict.

Spine Pattern¶

The spine pattern defines the startup spine of an application. This is a pattern where your application loads up configuration data, starts worker processes and loads the bulk of the subsystems to be used.

While many projects will have a simple cli startup sequence, that does not constitute a spine pattern. Most projects should not have what would be considered a Spine pattern. This pattern should be thought of as a pattern that itself is used to app-merge many other projects.

This is also a reason why loading configuration data or starting an async loop should not happen in the __init__ function for any Sub.

This is yet another opportunity to cover the fact that all projects should be small and brought together in tight, efficient ways. Here are a few things that are commonly done inside of a Spine pattern:

Set up the core data structures used by the application
Load up conf and read in the application configuration
Load up additional subsystems
Start up an asyncio loop
Start the main coroutines or functions
Start the patterns in the Subs’ init.py files

Beacon Pattern¶

The beacon pattern is used to gather events. In this example we will make a simple crypto-currency tracker. The Sub in this example will be called beacons. It uses an asyncio queue to collect and store data. This would be a simple init.py:

# Import python libs
import asyncio


def __init__(hub):
    """
    Set up the local data stores
    """
    hub.beacons.QUE = asyncio.Queue()


async def start(hub):
    """
    Start the beacon listening process
    """
    gens = []
    for mod in hub.beacons:
        if not hasattr(mod, "listen"):
            continue
        func = getattr(mod, "listen")
        gens.append(func())
    async for ret in hub.pop.loop.as_yielded(gens):
        await hub.beacons.QUE.put(ret)

This example shows iterating over the modules found in the beacons Sub. The plugins are defined as needing to implement an async generator function. We call the async generator function which returns an async generator that gets appended to a list. That list is then passed to the as_yielded function that yields as the next async generator yields. The yielded data is then added to a QUE that can be ingested elsewhere.

Following this pattern a plugin that emits a beacon could subsequently look like this:

import asyncio
import aiohttp

async def listen(hub):
    while True:
        async with aiohttp.ClientSession() as session:
            async with session.get("https://api.cryptonator.com/api/full/btc-usd") as resp:
                yield(resp.json())
        await asyncio.sleep(5)

Now we have a bitcoin ticker. More modules could act as means to gather data about other crypto-currencies. The pattern expressed here makes tracking more coins easy and allows for a separate Sub to process the collected data.

This pattern shows the concept of exposing reusable interfaces in a great way. This example does not care what is pulling the data off the queue, it just has a single function, to place said data on the queue. Now many different Subs could be used to read the output data in multiple ways.

Flow Pattern¶

The flow pattern is used for flow based interfaces. This follows an async pattern where data is queued and passed into and/or out of the subsystem. This is an excellent pattern for applications that do data processing. Data can be loaded into the pattern, processed, and sent forward to the next interface for processing. This pattern is used to link together multiple flow subsystems or to take data from a beacon or collection pattern and process it.

In the init.py file start a coroutine that waits on an async queue that is fed by another subsystem.

import asyncio

def __init__(hub):
    hub.flows.QUE = asyncio.Queue()


async def start(hub, mod):
    while True:
        data = await hub.beacons.QUE.get()
        ret = await getattr(f"flows.{mod}.process"){data}
        await hub.flows.QUE.put(ret)

Using a flow pattern makes pipe-lining concurrent data fast and efficient. For a more elegant example take a look at the internals of the umbra project.

Router Pattern¶

The router pattern is used to take input data and route it to the correct function and route it back. This is typically used with network interfaces. A typical init.py will look something like this:

import aiohttp

def start(hub):
    app = asyncio.web.Application()
    app.add_routes([asyncio.web.get("/", hub._.router)])
    aiohttp.web.run_app(app)

async def router(hub, request):
    data = request.json()
    if "ref" in data:
        return web.json_response(getattr(hub.server, data["ref"])(**data.get("kwargs")))

This example assumes that the sender is sending a json payload with 2 keys, one called “ref” to reference the function on the hub and another called “kwargs” so that any arguments can also be sent. This can be a great and simple way to expose part of the hub to the network.

Now the plugin subsystem can be populated with modules that expose request functions.

Just Examples¶

There are many ways to create patterns. These examples are not intended to be a document of all available patterns, but are meant to get you thinking about what kinds of patterns you can make inside of a Plugin Oriented Programming system.

Remember to make your Subs follow patterns to expose a re-usable interface! That will make every aspect of your code pluggable!

Plugins Allow for App Merging!¶

We have alluded to App Merging a few times. Well, the excitement is finally here! App Merging is one of the most powerful aspects of Plugin Oriented Programming, and it is in the next chapter.

Now that you know about the hub, Subs, plugins, and patterns; there is only one more high level concept to get familiar with; App Merging. Plugin Oriented Programming apps are not just made from plugins, they are plugins! Read on to learn how!

Patterns¶

Patterns Are Extensible¶

Collection Pattern¶

The init Function¶

The run_sub Function¶

A Grains Plugin¶

An Extensible Pattern¶

Spine Pattern¶

Beacon Pattern¶

Flow Pattern¶

Router Pattern¶

Just Examples¶

Plugins Allow for App Merging!¶

Intro to Plugin Oriented Programming

Navigation

Related Topics

Patterns¶

Patterns Are Extensible¶

Collection Pattern¶

The __init__ Function¶

The run_sub Function¶

A Grains Plugin¶

An Extensible Pattern¶

Spine Pattern¶

Beacon Pattern¶

Flow Pattern¶

Router Pattern¶

Just Examples¶

Plugins Allow for App Merging!¶

The init Function¶