Plugin Oriented Configuration

Applications need to be able to be configured. This presents a number of unique challenges to Plugin Oriented Programming. For the concepts around app merging to work, the loading up of configuration data at the startup of an application must be taken into full account. The process of loading up that data must also be pluggable.

Addressing these needs creates a difficult point of intersection. As these points get evaluated it becomes necessary to have systems in place that can make configuration of the applications not only easy, but accessible and extensible.

The configuration loading system used by pop is called pop-config. Extensive documentation on how to use pop-config is presented by the pop-config project.

Configuration Merging

Since apps can merge, configuration can merge. When defining configuration that can be merged between multiple applications it becomes critical that the loaded configuration is consistent across applications even after they have been merged.

To do this, the conf.py files of each project are read, parsed, namespaced, and allowed to be loaded via several mediums. The options for a specific application are stored on the hub in a predictable path that is made available to the application, whether loaded standalone or merged into another application.

Configuration Priority

When loading a robust configuration system for an application, it needs to load configuration options in a specific order. Configuration of an application, as applied during the startup of the application, can be loaded from several sources.

There are four main sources where configuration loading can occur: defaults, environment variables, configuration files, and command-line flags. These then need to be applied in the correct priority.

  • Command-line flags overwrite environment variables
  • Environment variables overwrite configuration file options
  • Configuration file options overwrite defaults

The pop-config system allows you to simply place the configuration values, defaults, accepted sources, and documentation, into a single location. This makes it easy to add configuration options to a project and automatically makes those configuration options app-mergeable!

The conf.py Dictionaries

Plugin Oriented Programming presents a single file to specify the configuration of a given project. This file is used to set up all configuration options and to load said options onto the hub.

We have already introduced this file for the use of the DYNE dictionary. But there are 3 additional dictionaries found therein. The CLI_CONFIG dictionary, the CONFIG dictionary, and the SUBCOMMANDS dictionary.

Each dictionary holds data to present configuration options to the end user.

CONFIG Dictionary

The CONFIG dictionary is a very powerful system for applying configuration values and where the root of configuration should be applied. The CONFIG dictionary is primarily used for non command-line flags, those should be reserved for the CLI_CONFIG dictionary.

Basic Settings

Nearly every config setting needs to have 2 basic options, default and help. These are very self explanatory. default sets the default value of the option if no option is passed and help presents, not only the command-line help, but is also the single source of documentation for the option.

Here is a simple example:

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

This establishes the basic data for the setting and is all that is needed for settings in the CONFIG dictionary.

Destination

When the argument is named “test” it will appear on the option namespace as “test”. This may not always be desirable. If the name of the option and where it needs to be stored differs, then use the dest option:

CONFIG = {
    "test": {
        "default": "Red",
        "dest": "cheese",
        "help": "What color to test",
    },
}

In this example the option will be stored under the name “cheese”.

Location

Once the config system has been run, all configuration data will appear in the hub.OPT namespace. This means that in our first example, if the system in question is part of an app named myapp, then the option data will be present at hub.OPT[“myapp”][“test”]. In the second example, the option data will be present at hub.OPT[“myapp”][“cheese”].

CLI_CONFIG

The heaviest configuration options will be in the CLI_CONFIG dictionary. All options that appear on the CLI need to be activated in the CLI_CONFIG, but the basic configuration needs to be in the CONFIG dictionary.

Pop-config uses Python’s venerable argparse under the hood to present and process the arguments. Pop-config will also transparently pass options from the dictionary into argparse. This makes Pop-config transparently compatible with new argparse options that are made available.

This document is intended, therefore, to present the most commonly used options. Please see the argparse doc for more in depth data on available options.

If the cli option is very simple it can be as simple as just doing this:

CLI_CONFIG = {
    "test": {},
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

This way the options presented in the cli are all explicitly set. But additional options about how the cli is presented to the end user is easy to do.

This document will cover all available options for the CLI_CONFIG. Remember that the default and help values should always be in the CONFIG section to facilitate easy app-merging of configuration data and settings.

Source

By default, the CLI_CONFIG references the local CONFIG setting. The source option allows you to reference a documented configuration from a separate project configuration. This powerful option allows you to manage the arguments and flags in a namespace of an app that is being merged into this app. The benefit here is that the CONFIG values do not need to be rewritten and you maintain a single authoritative source of documentation.

When using source in the CLI_CONFIG the namespace that defined the option in the CONFIG dictionary will own the option. This makes it easy for an application that uses its own config namespace to be app merged into another application that can then transparently manage the configuration of the merged app.

Dyne

A powerful option in the CLI_CONFIG is dyne. This uses vertical app merging to modify another application’s cli options. This allows a vertical app merge repo to define cli arguments that will be made available when the plugins are installed to extend an external app.

Options

By default the options presented on the command-line are identical to the name of the value. So for the above example the presented option would be –test. If alternative options are desired, they can be easily added:

CLI_CONFIG = {
    "test": {
        "options": ["-t", "--testy-mc-tester", "-Q"],
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

Pop-config automatically determines between short options and long options.

Positional Arguments

Positional arguments are very common and can create a much more user friendly experience for users. Adding positional arguments is easy. Just use the positional argument:

CLI_CONFIG = {
    "test": {
        "positional": True,
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

When working with multiple positional arguments the display_priority flag can be used:

CLI_CONFIG = {
    "test": {
        "positional": True,
        "display_priority": 2,
    },
    "run": {
        "positional": True,
        "display_priority": 1,
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
    "run": {
        "default": "green",
        "help": "What color to run",
    },
}

In the above example the first argument will be run and the second will be test.

Accepting Environment Variables

Operating systems allow for configuration options to be passed in via specific means. In Unix based systems like Linux and MacOS, environment variables can be used. In Windows based systems the registry can be used. To allow for an os variable to be used just add the os option:

CLI_CONFIG = {
    "test": {
        "os": "MYAPP_TEST",
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

Now the flag can be set by setting the environment variable MYAPP_TEST to the desired configuration value.

Actions

Actions allow a command-line argument to perform an action, or flip a switch.

The action option passes through to argparse. If the examples in this document do not make sense you can also check the argsparse section on action.

A number of actions are supported by argparse. Arguably the most frequently used actions are store_true and store_false:

CLI_CONFIG = {
    "test": {
        "action": "store_true",
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

A few other useful actions are append and count. If append is used then every time the argument is used the option passed to the argument is appended to the final list. The count option allows for the number of times that the argument is passed to be counted up. This is useful for situations where you want to specify what the verbosity of the output should be, so that you can pass -vvv in a similar fashion to ssh.

Number of Arguments

The number of arguments that should be expected can also be set using the nargs option. This allows for a specific or fluid number of options to be passed into a single cli option.

The nargs option passes through to argparse. If the examples in this document do not make sense you can also check the argsparse section on nargs.

Integer (1)

Specifying an integer defines the explicit number of options to require:

CLI_CONFIG = {
    "test": {
        "nargs": 3,
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

The above example will require that exactly 3 options are passed to –test.

Question Mark (?)

One argument will be consumed from the command-line, if possible, and produced as a single item. If no command-line argument is present, the value from default will be produced.

Asterisk (*)

All command-line arguments present are gathered into a list.

CLI_CONFIG = {
    "test": {
        "nargs": "*",
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

Plus (+)

Just like ‘*’, all command-line args present are gathered into a list. Additionally, an error message will be generated if there wasn’t at least one command-line argument present.

Type

The value type can be enforced with the type option. A type can be passed in that will be enforced, such as int or str.

CLI_CONFIG = {
    "test": {
        "type": int,
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

Render

Sometimes it is desirable to load up complex data structures from the command-line. This can be done with the render option. The render option allows you to specify that the argument passed will be rendered using a data serialization medium such as json or yaml.

CLI_CONFIG = {
    "test": {
        "render": "yaml",
    },
}

CONFIG = {
    "test": {
        "default": "Red",
        "help": "What color to test",
    },
}

This cli could then look like this:

myapp --test "Food: true"

Then the resulting value would be: {“Food”: True}

Subcommands

Sometimes it is desirable to have subcommands. Subcommands allow your CLI to work in a way similar to the git cli, where you have multiple routines that all can be called from a single command.

This example shows how multiple subcommands can be defined and utilized.

CLI_CONFIG = {
    "name": {
        "subcommands": ["test", "apply"],
    },
    "weight": {},
    "power": {
        "subcommands": ["apply"],
    },
}
CONFIG = {
    "name": {
        "default": "frank",
        "help": "Enter the name to use",
    },
    "weight": {
        "default": "150",
        "help": "Enter how heavy it should be",
    },
    "power": {
        "default": "100",
        "help": "Enter how powerful it should be",
    },
}

SUBCOMMANDS = {
    "test": {
        "help": "Used to test",
        "desc": "When running in test mode, things will be tested",
    },
    "apply": {
        "help": "Used to apply",
        "desc": "When running in apply mode, things will be applied",
    },
}

In this example we see that the option name will be available under the subcommands test and apply. The option power will be available only under the subcommand apply and the option weight is globally available.

Conclusion

In the end, pop-config gives you everything that you need to create not only beautiful command-line applications, but also deeply robust configuration loading systems that facilitate app merging.