Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Making SPy installable without its repository

Today, SPy can only be installed as an editable install of a git checkout, followed by a manual make -C spy/libspy. The goal is that users run uv tool install spylang, and that developers run uv sync and nothing else. Everything compiled is built by uv sync (developers) or when the package is built (users). Anything still missing is built automatically the first time it is needed.

What has to be built

SPy needs a small C runtime library, called libspy. It has about 6000 lines of C and no external dependency.

libspy exists in several variants, each described by:

The standard variants are the ten that make -C spy/libspy builds today, all without --march: native and native-static (release and debug, exe), and wasi and emscripten (release and debug exe, debug testlib). emscripten needs emcc, and native-static does not exist on macOS: they are skipped when unavailable.

Who uses which variant

Each variant is a directory of spy/libspy/build/ today (a cache entry later).

VariantUsed by
wasi/debug-testlibThe interpreter: every VM loads its libspy.wasm (vm.ll), so every spy command uses it, spy build included (for the interpreted part: imports and redshift). Out-of-tree modules: its libspy.a is linked with their archives into one wasm. Tests: the [interp], [doppler] and [C] backends. The [C] backend also compiles the generated C to a wasm module linked with this libspy.a.
native/debugspy build (the default is --target native, debug) for the compiled part; the [native] and [py-cffi] test backends.
native/releasespy build --release.
native-static/{debug,release}spy build --static, with or without --release.
wasi/{debug,release}spy build --target wasi, with or without --release.
emscripten/{debug,release}spy build --target emscripten; the [emscripten] test backend and the playground (debug).
emscripten/debug-testlibSPy running on Pyodide under node: the VM loads its libspy.mjs.

So pytest with the default backends needs only wasi/debug-testlib, which is also the only variant shipped in the wheel. Everything else is needed by spy build or by the other backends, and is built from the sources, on first use if it was not already built.

The garbage collector libgc is not part of libspy. It is only linked into native executables, and it is found as a system or conda library. This does not change.

Two kinds of variants

The wheel contains:

Finding libspy at run time

One function, ensure_libspy, replaces the fixed spy/libspy/build/... paths. It returns the directory containing libspy.a for a variant, building it first if it is not in the cache.

The key hashes the libspy sources and headers, the variant, the compiler flags and the compiler version. Hashing the whole of libspy takes about 1 ms. The build directory is stored in the user cache directory (~/.cache/spy/libspy/<target>/<build type>-march-<march>-<key> on Linux), apart from the directories made by make. SPY_CACHE_DIR overrides the location of the cache; the tests use it.

The module spy.libspy.builder also defines the helpers variant_name, get_cflags_for_variant, get_ninja and _gen_build_ninja, described below. CompilerConfig._ensure_libspy_built is the only caller. Without --march it returns the directory built by make -C spy/libspy; with --march it calls ensure_libspy.

import hashlib
import os
import re
import shutil
import subprocess
import tempfile
from typing import Optional

import platformdirs
import py.path

import spy.libspy
from spy.build.build_info import BuildType, OutputKind
from spy.build.flags import get_build_dirname, get_cc
from spy.errors import WIP


def cache_key(
    target: str, build_type: BuildType, kind: OutputKind, march: Optional[str]
) -> str:
    """
    Name of a build, computed from everything which influences it: if one of
    the inputs changes, the key changes, so that a stale libspy.a can never be
    reused by mistake.

    The inputs are the libspy sources and headers, the variant (target, build
    type, kind, march), the compiler version and the flags.
    """
    libspy_dir = spy.libspy.BUILD.dirpath()
    h = hashlib.sha256()
    for subdir in ("src", "include", "vendored"):
        for p in sorted(libspy_dir.join(subdir).visit(rec=True)):
            if p.check(file=True) and p.ext in (".c", ".h"):
                h.update(p.relto(libspy_dir).encode())
                h.update(p.read_binary())
    h.update(variant_name(target, build_type, kind, march).encode())
    h.update(" ".join(get_cflags_for_variant(target, build_type, kind, march)).encode())
    cc = get_cc(target).split()
    res = subprocess.run([*cc, "--version"], capture_output=True)
    h.update(res.stdout + res.stderr)
    return h.hexdigest()[:16]


# how many builds are kept for each variant, see prune_cache()
KEEP_PER_VARIANT = 3

# <build type>-march-<march>-<key>, see ensure_libspy()
_BUILD_DIRNAME = re.compile(r"(?P<variant>.+)-[0-9a-f]{16}")


def cache_root() -> py.path.local:
    """
    The user cache directory of SPy: $SPY_CACHE_DIR if it is set, else the
    standard one of the platform (~/.cache/spy on Linux, honoring
    $XDG_CACHE_HOME).
    """
    env = os.environ.get("SPY_CACHE_DIR")
    if env:
        return py.path.local(env)
    return py.path.local(platformdirs.user_cache_dir("spy"))


def cache_dir() -> py.path.local:
    """
    Where the variants built by ensure_libspy() are stored, as
    <cache_dir>/<target>/<name>-<key>. The libspy built by `make` is not here.
    """
    return cache_root().join("libspy")


def _mtime(path: py.path.local) -> float:
    try:
        return path.mtime()
    except OSError:  # removed by another process
        return 0.0


def prune_cache(keep: int = KEEP_PER_VARIANT) -> int:
    """
    For each variant (same target, build type, kind and march, whatever the
    key), keep only the `keep` most recently used builds and delete the others.
    Return how many were deleted.

    Each edit of the libspy sources gives a new key, so without this the cache
    would grow for ever. Only the directories named like a build are
    considered, so a temporary directory of a build in progress is never
    deleted. The date of last use is the mtime of the directory, which
    ensure_libspy() updates at each use.
    """
    removed = 0
    root = cache_dir()
    if not root.check(dir=True):
        return 0
    for target in root.listdir():
        variants: dict[str, list[py.path.local]] = {}
        for d in target.listdir():
            m = _BUILD_DIRNAME.fullmatch(d.basename)
            if m and d.check(dir=True):
                variants.setdefault(m["variant"], []).append(d)
        for builds in variants.values():
            builds.sort(key=_mtime, reverse=True)
            for old in builds[keep:]:
                shutil.rmtree(old, ignore_errors=True)
                removed += 1
    return removed


def _touch(path: py.path.local) -> None:
    """Record that this build has just been used."""
    try:
        os.utime(path)
    except OSError:
        pass  # e.g. read-only cache: not a reason to fail


def ensure_libspy(
    target: str, build_type: BuildType, kind: OutputKind, march: Optional[str]
) -> py.path.local:
    """
    Return the directory containing libspy.a for this variant, building it
    first if it is not in the cache.
    """
    assert march is not None, "only the --march variants are built here for now"
    key = cache_key(target, build_type, kind, march)
    name = f"{get_build_dirname(build_type, kind)}-march-{march}-{key}"
    final = cache_dir().join(target, name)
    if final.join("libspy.a").check(file=True):
        _touch(final)
        return final

    # Build in a temporary directory, then rename it: a rename is atomic, so
    # nobody ever sees a half-built library, and several processes (e.g.
    # pytest-xdist workers) can start together without corrupting the cache.
    final.dirpath().ensure(dir=True)
    tmp = py.path.local(tempfile.mkdtemp(dir=str(final.dirpath())))
    try:
        cflags = get_cflags_for_variant(target, build_type, kind, march)
        tmp.join("build.ninja").write(_gen_build_ninja(target, cflags))
        subprocess.run([*get_ninja(), "-C", str(tmp)], check=True)
        if not tmp.join("libspy.a").check(file=True):
            raise WIP(f"ninja did not produce {final}/libspy.a")
        try:
            tmp.rename(final)
        except OSError:
            pass  # another process finished first
    finally:
        shutil.rmtree(tmp, ignore_errors=True)
    prune_cache()
    return final

Explanations:

A later step extends ensure_libspy to every variant: it first looks for a prebuilt variant shipped in the wheel (spy/libspy/prebuilt/).

spy.libspy currently loads the wasm at import time. It becomes lazy, so that importing spy never triggers a build by surprise, and so that the build hook can import spy.libspy.builder without wasmtime.

Building libspy

spy.libspy.builder contains the list of C files, the Ryu flags and a small build.ninja generator with two rules: cc and ar (which creates the .a archive). All the other flags come from spy.build.flags. It does not use NinjaWriter, because NinjaWriter needs a CompilerConfig, which itself calls ensure_libspy. Both share fmt_flags.

This module also handles jsffi.c (emscripten only) and the final wasm link, and builds the standard variants:

STANDARD_VARIANTS = [
    ("native", "release", "exe"),
    ("native", "debug", "exe"),
    ("native-static", "release", "exe"),
    ("native-static", "debug", "exe"),
    ("wasi", "release", "exe"),
    ("wasi", "debug", "exe"),
    ("wasi", "debug", "testlib"),
    ("emscripten", "release", "exe"),
    ("emscripten", "debug", "exe"),
    ("emscripten", "debug", "testlib"),
]


def build_standard_variants() -> None:
    for target, build_type, kind in STANDARD_VARIANTS:
        if target == "emscripten" and shutil.which("emcc") is None:
            continue
        if target == "native-static" and sys.platform == "darwin":
            continue
        ensure_libspy(target, build_type, kind, march=None)


if __name__ == "__main__":
    build_standard_variants()

The Makefile is kept as a wrapper, so that make -C spy/libspy still rebuilds everything, into the cache:

all:
	python -m spy.libspy.builder

Then spy/libspy/build/ is no longer used, and python -m spy.build.flags --libdir returns the directory given by ensure_libspy.

spy.libspy.builder imports spy.build.ninja, which imports spy.build.config, which imports spy.libspy, which loads wasmtime. Today config.py imports spy.libspy.builder lazily, inside _ensure_libspy_built. This chain is cut, so that the build step can use the writer without needing the runtime.

Wheel and sdist

The build backend becomes hatchling, with a build hook. For a wheel it produces the prebuilt variant. For an editable install it fills the cache.

pyproject.toml:

[build-system]
requires = ["hatchling", "ziglang==0.13.0", "ninja==1.13.2", "py==1.11.0", "platformdirs"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["spy"]
# hatchling skips files matched by .gitignore, which lists libspy.a and
# libspy.wasm: the built files must be listed explicitly.
artifacts = ["spy/libspy/prebuilt/**"]

[tool.hatch.build.hooks.custom]
path = "hatch_build.py"

[tool.uv]
# uv rebuilds an editable install, hence reruns the hook, when one of these changes
cache-keys = [
    { file = "pyproject.toml" },
    { file = "spy/libspy/**/*.c" },
    { file = "spy/libspy/**/*.h" },
]

hatch_build.py, at the root of the repository:

from pathlib import Path

from hatchling.builders.hooks.plugin.interface import BuildHookInterface


class CustomBuildHook(BuildHookInterface):
    def initialize(self, version, build_data):
        from spy.libspy.builder import build_prebuilt, build_standard_variants

        if version == "editable":
            # nothing is written in the package: the variants go to the cache
            build_standard_variants()
        else:
            build_prebuilt(Path(self.root) / "spy" / "libspy" / "prebuilt")

build_prebuilt builds the wasi/debug-testlib variant and links libspy.wasm. The requires list contains ziglang (the compiler), ninja, platformdirs and py (imported by spy/__init__.py).

For an editable install, the hook fills the cache with all the standard variants and writes nothing in the package. A developer who runs uv sync, then pytest or spy, finds everything already built: nothing is compiled by these commands. Two points:

If the C sources change without uv sync rerunning the hook, ensure_libspy builds the changed variants the first time they are needed: the hook only fills the cache in advance.

Zig and ninja

ziglang stays a normal runtime dependency (a wheel of about 81 MB, against about 9 MB for wasmtime and less than 1 MB for everything else). Zig is needed to compile to wasi, for --static builds, for any variant built from sources into the cache, and to link out-of-tree wasm modules into the interpreter.

Both tools are currently started as python -m ziglang or as a bare ninja. Under uv tool install, python is not the interpreter of the tool’s environment, and the executables of dependencies are not on PATH. Both are started through the interpreter that runs SPy:

import sys


def get_zig() -> list[str]:
    try:
        import ziglang  # noqa: F401
    except ImportError:
        raise RuntimeError(
            "zig is required for this operation: pip install ziglang"
        ) from None
    return [sys.executable, "-m", "ziglang"]


def get_ninja() -> list[str]:
    return [sys.executable, "-m", "ninja"]

Every call site goes through these two functions (get_ninja is first defined in spy.libspy.builder, and moves next to get_zig). Making ziglang an optional extra later only means editing pyproject.toml: the error message is already in place.

Out-of-tree modules

An out-of-tree module is a Python package, outside the SPy source tree, that wraps a C library. Its build_info() function returns the include directories, the archives and the linker flags to use. Today its archives are built beforehand by a per-module Makefile that calls python -m spy.build.flags.

Steps

Each step is a separate pull request.

Future
Metaclasses written in SPy: a design sketch
Future
Calling prebuilt stdlib modules from the interpreter