Tip of the Day: Combining Abseil and Typer Flags

tl;dr: Launch Typer apps using absl.app.run and known_only flags parsing to get absl.flags as well as Typer argument processing:

from absl import app as absl_app, flags
import typer

app = typer.Typer()
FLAGS = flags.FLAGS


def main() -> None:
    def run_typer(args: list[str]) -> None:
        app(args[1:])

    def parse_flags(argv: list[str]) -> list[str]:
        return FLAGS(argv, known_only=True)

    absl_app.run(run_typer, flags_parser=parse_flags)


if __name__ == "__main__":
    main()

This particular recipe didn’t seem to be very easy to find online, so I’m documenting it here. Why anyone would want to use two very different and conflicting flag/argument processing systems at the same time is an interesting question.

The summary is that FLAGS are for library code (and avoiding hard-coded values which cannot be changed at runtime) and Typer options/arguments are for the CLI “front-end”.

Flags are not just for patriots

#

I’m a fan of abseil. Or more accurately, I got used to using the internal version of it when I was coding Python at Google. While it may be marketed as a C++ library, the python implementation has some useful features for flag handling.

Google uses commandline flags rather than environment variables for configuring apps. This is very much in the spirit of Rule 2 in the Zen of Python: Explicit is better than implicit.

And also against the 12 Factor App guidelines, which conveniently fail to mention commandline flags as a mechanism for configuring an app.

One of the many things I prefer about using flags is that they are always visible, so you don’t get silently different behaviour in different contexts. And if you (or your tooling) can manage a set of environment variables, you can just as easily manage an array of arguments. So there’s no tooling advantage to using env vars.

However, it’s also unwieldy to type more than a handful of flags on the commandline.

We find ourselves needing to balance

• arguments to the command (input, output)
• configuration / options (to control the processing / output)
• overrides for defaults (maybe to change backend parameters, or dev/prod environment)

There are definitely some cases which work better with flags than environment variables:

# Initial command:
gen_diagrams --verbose --output=/some/path
# With env override
DIAGRAM_TYPE=svg MAX_WIDTH=200 gen_diagrams --verbose --output=/some/path
# With flags
gen_diagrams --verbose --output=/some/path --diagram-type=svg --max-width=200

In your shell history (especially when using the default up-line-or-beginning-search binding for the Up arrow in ohymyzsh), it helps to have the command at the start of the line. This makes it easier to search in history, and if I start typing gen_dia and hit Up, the last invocation of gen_diagrams will be helpfully found and offered in my command buffer, ready for modification and execution. Conversely, it’s awkward to remember whether I set DIAGRAM_TYPE or MAX_WIDTH first the last time I ran the command, and then search for that instead.

Whose flag is this? Library config versus runtime arguments

#

When you build a commandline tool using library functions, you have to think a lot about the interface of the tool. How will it be called? What arguments are provided by the user, and what parameters are needed by the library functions?

Typer makes it easy to define commandline commands and specify their arguments and options. But this only provides arguments to that outer-level function.

With the gen_diagrams example, we could be using something like PlantUML as a backend with a helper library to run the plantuml command in a consistent way. What if we want to change the default arguments to plantuml or choose a different JDK for running it?

I don’t want to have to annotate every command in the CLI to accept a bunch of common backend settings. With Typer, this can be done with a callback function which consumes common arguments before invoking the intended function, eg supporting a verbose flag:

import typer
app = typer.Typer()
VERBOSE = false

@app.callback()
def common_options(verbose: bool|None=None):
    global VERBOSE
    if verbose:
        VERBOSE = verbose

@app.command()
def do_something(arg: str):
    if VERBOSE:
        print("About to do_something")
    ...

But this only really helps for settings which are owned by the top-level program / script.

In the current project I’m working on, I have a data class which is backed by some kind of object store (could be a database, could be a flat file). I want to be able to support settings like DB_PATH or DB_HOST without having to add them as parameters to every command. “Aha!” say the environment variable fans- this is what environment variables are for! Which is true, but also (for the reasons mentioned before) not the whole picture.

With absl.flags, my module can declare its own settings and defaults which can be changed at runtime via a flag, without having to annotate every command to support every backend setting.

So I can have things.py:

from absl import flags

flags.DEFINE_string("thing_path", default="")
flags.DEFINE_string("thing_host", default="db.example.com")

FLAGS = flags.FLAGS

class ThingStore():
    @classmethod
    def connect(cls):
        if FLAGS.objstore_path:
            return cls.read_objs(FLAGS.thing_path)
        else:
            return cls.db_connect(FLAGS.thing_host)

And a cli:

import typer
from things import Thing

app = typer.Typer()

@app.command()
def query_all():
    for i in objectstore.connect():
        print(i)

And I can call it as

dbcli query-all
dbcli query-all --thing_host=testdb.example.com