About SPy package distribution

Definitions

Wheel: the file format (.whl) in which Python packages are distributed. Its name carries tags (Python version, ABI, platform) that tell pip whether it can be installed on a given machine. Symbol: the name under which a compiled function or global variable is known to the linker, for instance spy_depb$bump. Linker: the tool that merges object files and archives into one program, and connects each use of a symbol to its single definition. Static linking: merging at build time, here into one WASM module. Dynamic linking: connecting separately built modules when the program is loaded. SONAME: the version name (such as libfoo.so.1) embedded in a Linux shared library and recorded by every program linked against it.

The terms archive, manifest, WASM instance, linear memory, redshift and ABI are defined in Calling prebuilt stdlib modules from the interpreter.

What a package contains¶

A package would be a wheel with three kinds of file:

.spy sources. The compiler needs them to type-check and redshift calls into the package, for the same reason C libraries ship header files.
A wasm32 archive (.a), produced from the sources by the C backend.
A manifest, which lists what the archive implements, the packages it requires, and the libspy version and build configuration it was built for.

An archive holds WebAssembly object code, which does not depend on the operating system or on the CPU. Sources, archive and manifest are therefore platform independent, and the wheel can be a py3-none-any wheel: no platform tag is needed.

A linked .wasm file is not part of the package. It would be a complete program with its own copy of libspy, its own memory and its own allocator, which cannot be combined with another one (next section).

Using a package from the interpreter¶

One instance, not two¶

Loading the .wasm of a package as a second WASM instance would give two linear memories. A pointer is an index into one memory, so a str created by one instance means nothing to the other. Making both instances import the same memory fails for three reasons:

each program places its global variables and string literals at the same low addresses, so they overwrite each other, unless both are built as position-independent code, which WebAssembly supports poorly;
each carries its own copy of the allocator and of the garbage collector of libspy; they would corrupt each other’s bookkeeping, and neither collector sees the references held by the other;
each needs its own stack region.

So everything is linked into one instance: libspy and the archives of all the packages are merged into the vm.ll of the interpreter. There is one memory and one allocator, and a pointer means the same thing for interpreted and compiled code.

A user module and its dependency¶

The machinery of the prebuilt stdlib already accepts modules which are not in the stdlib. Here depa plays the role of a package which depends on depb:

# depb.spy
def bump(x: i32) -> i32:
    return x + 1

# depa.spy
import depb

def twice(x: i32) -> i32:
    return depb.bump(depb.bump(x))

The C backend lowers depa.twice to (line directives removed):

int32_t spy_depa$twice(int32_t x$0) {
    int32_t $v0;
    $v0 = spy_depb$bump(x$0);
    return spy_depb$bump($v0);
}

The call is compiled as a C call to the symbol spy_depb$bump, so the archive of depa must contain the code of depb too. It does: it holds depa.o, depb.o and native_shim.o, while the manifest lists only depa::twice, the function of the requested module. The interpreter then uses it like this:

nsl = get_native_stdlib(("depa",), ("/tmp/deptest",))  # builds, then caches
vm = SPyVM(native_stdlib=nsl)
vm.path.append("/tmp/deptest")
vm.import_("depa")
w_twice = vm.lookup_global(FQN("depa::twice"))   # a W_NativeFunc
vm.unwrap(vm.fast_call(w_twice, [W_I32(40)]))    # 42

The second argument of get_native_stdlib is the list of directories where to look for modules, in addition to the stdlib. Called directly from the interpreter, depb::bump stays interpreted: only the functions of the requested modules are replaced.

Each symbol needs one owner¶

An archive contains the compiled code of every module of its import tree. Linking the archives of depa and depb into one instance therefore fails:

wasm-ld: error: duplicate symbol: spy_depb$bump
>>> defined in .../libspy_stdlib.a(depb.o)
>>> defined in .../libspy_stdlib.a(depb.o)

For a function this is only a link error. For a module variable, two copies would mean two independent states. Two ways to give every symbol exactly one definition:

Ownership by package. An archive contains only the modules of its own package, and the manifest says which other packages are required. The bundle is libspy plus all the archives. A generic instantiation used by several packages, such as array[u32], has no single owner: it would be emitted with weak linkage, which lets the linker keep one copy.
One archive per bundle. The wheel ships only sources. The archive of the whole set of modules is built in a single C backend run on first use, and cached like the native stdlib (step 5 of its plan). Definitions are unique by construction, and the archive is always built against the installed libspy headers, so no version check is needed. The price is a compilation on first use.

The first option is needed to ship prebuilt archives. The second needs nothing new in the package format.

The modules must be known before the VM starts¶

vm.ll is created by the constructor of SPyVM, so the bundle must be linked before. In SPy, import is a static declaration: ImportAnalyzer collects every import statement, including those inside functions, starting from the root module. The full set of modules is therefore known before anything runs:

walk the imports from the root module;
find, for each module, the archive of the package which provides it;
link and cache the bundle (wasm_bundle.py);
create the VM on this bundle.

Today ImportAnalyzer receives a VM, mostly for vm.path and robust_import_caching. Step 1 needs a variant which works without a VM.

This does not fit an incremental session such as IPython: the imports of a cell are known only when the cell runs, and relinking would move the live heap. The rule of the prebuilt stdlib covers this case: a module which is not in the bundle stays interpreted, slower but identical. A session can also declare its modules at startup.

Using a package from `spy build`¶

Recompile from .spy sources. The compiler sees everything, so it can inline and specialise across module boundaries. This is the only option for a native binary: wasm-ld produces WebAssembly, so a wasm32 archive can be linked only into a WASM target (wasi, emscripten).
Link the wasm32 archive. Faster builds, no cross-module optimisation, WASM targets only.

A native archive per platform, as C libraries do, would be a third option. It would require platform-specific wheels.

Versioning libspy¶

On Linux, a shared library such as libfoo.so.1.2.3 embeds a SONAME, libfoo.so.1. Every program records the SONAME it was linked against, and the loader looks for exactly that name. The major number changes only when the ABI breaks (a struct layout, a removed function or a changed signature), so several majors can be installed side by side.

libspy is linked statically: there is no loader, and the check happens when the bundle is linked. What can break an archive:

the layout of the structs of the libspy headers (spy_StrObject, gc_ptr) and the signatures of its functions;
the build configuration: gc_ptr has a length field only in debug builds, so the archive must match the build type, and probably --march.

The plan, in two stages:

while libspy changes often, the manifest records the exact libspy version and the build configuration, and the bundle step refuses any mismatch;
once libspy is stable, an ABI number which is bumped only for incompatible changes. libspy defines a symbol such as spy_abi_1, referenced by every archive, so that a mismatch fails at link time.

Open questions¶

Cross-module redshifting. If mypackage.add(x, y) can be specialised for i32 arguments, does that specialisation happen at package-compile time (several variants in the archive) or at application-compile time (which requires the source)? This is a trade-off between convenience of distribution and performance.

Python bindings. There is no mechanism to generate a Python extension from SPy code. It would need something like a CFFI or pybind11 equivalent for SPy, and it would be the only part which needs platform-specific wheels.

PyPI and WebAssembly. There is no agreed platform tag for wasm32 wheels on PyPI. It matters only for Python bindings, since the wheels above are py3-none-any.

Plan¶

Each step is a separate, testable change.

☐ 1. Import pre-pass. Compute the set of modules from the root module, without a VM.

☐ 2. Several archives in one bundle. Link libspy and the archives of all the required packages, with the ownership rule chosen above.

☐ 3. Package layout. One py3-none-any wheel with sources, archive and manifest; the interpreter finds them through the installed packages.

☐ 4. Exact version check. The manifest records the libspy version and the build configuration; a mismatch is an error.

☐ 5. ABI number. Once libspy is stable.