Metaclasses written in SPy: a design sketch

1. The claim¶

In most languages, dataclasses, interfaces or sum types are hard-wired in the compiler. Here the compiler would know only about structs. Every other kind of class (heap-allocated classes, classes with inheritance, protocols, variants) would be a metaclass written in SPy, in the standard library. A user could read it, change it, or write a new one with the tools used for any other SPy code.

2. What a class statement does today¶

Today only @struct class works. A class statement is executed in three steps:

1. Declaration. When a module starts, before any import is executed, the compiler creates an empty type for every class of the module and registers it under its name.

2. Body. The body is executed, which collects a ClassBody. The methods are typed at this moment, and their self gets the declared type.

3. Definition. The ClassBody is used to define the declared type (layout, accessors, __make__, __eq__...). Today this is done by a Python method, define_from_classbody, which SPy code cannot see.

The plan is to replace step 3 by metaclasses written in SPy. Three facts constrain them:

• Declaration happens before the imports. It cannot depend on the metaclass nor on the base classes, which are expressions that can come from any import.

• The methods are already typed with the declared type. A metaclass that returned a new type, like a Python class decorator, would leave self and every annotation other: Point pointing at the declared one.

• A type is defined once, with its complete content.

So a metaclass must define the declared type, not return another one.

3. The metaclass and the base are data of the type¶

Today the metaclass of a struct type is the Python class that implements it inside the compiler. That forces the compiler to know it at declaration, which is impossible. The solution is to turn the relations between types into fields of the type object (w_ marks a reference to an SPy object, as in the compiler’s code), filled when the type is defined. Every class is first declared as the same generic empty type, whatever its metaclass.

• w_meta is the metaclass, the type of the type. It is set once, at definition, just before define_from_classbody is called.

• w_base is the base type (StructType in class Record(StructType)). It is set once, at definition, from the base class, which is evaluated after the imports. It is used by super() and by the lookup of the methods of a metaclass.

The metaclass of class C(Base, metaclass=M) is M if given, otherwise the metaclass of Base, otherwise StructType, the default metaclass (today’s @struct is how it is spelled). As in Python, an error is raised if M is not a subtype of the metaclasses of the bases.

With this, isinstance works at the SPy level:

class Point(metaclass=Record):
    x: i32
    y: i32

assert isinstance(Point, Record)       # Point.w_meta is Record
assert isinstance(Point, StructType)   # Record.w_base is StructType
assert isinstance(Point, type)         # StructType.w_base is type

A metaclass is written as a class that inherits from another metaclass, like Record above, which inherits from StructType. A class whose base is a metaclass is itself a metaclass, and it has no layout, since its instances are types, which the compiler holds at compile time. So inheriting means two things only: w_base is stored, and methods are looked up by name along the chain of bases, at compile time. No vtable is needed. The rule has restrictions:

• the base must be a metaclass, and there is a single base (inheritance between ordinary classes is provided by ClassMetaClass, section 6);

• a metaclass has no fields and no instances at run time;

• super() inside a method of a metaclass is a compile-time lookup in w_base.

The tower is not circular: the compiler’s rule defines the metaclasses without layout, StructType is the first of them, and Record and ClassMetaClass are written on top of it.

4. A metaclass in SPy: Record¶

Here is a simplified Record, a heap-allocated class: a Point is a pointer to data allocated on the heap, so copying a Point copies the pointer, as in Python. (The real one, in stdlib/metaclasses/record.spy, does more, see the end of this section.)

from operator import OpSpec
from unsafe import gc_ptr, gc_alloc


class Record(StructType):
    def define_from_classbody(self, body):
        cls = self                          # the type being defined: the wrapper
        fields = body.fields                # {"x": i32, "y": i32}

        # 1. Declare the data struct. For now it only has a name.
        Data = StructType.declare(self.fqn.join("Data"))

        # 2. Build the body of the wrapper, then define it.
        wrapper = ClassBody()
        wrapper.add_field("__ll__", gc_ptr[Data])
        wrapper.add_methods(body.methods)   # the user's methods, unchanged

        @blue.metafunc
        def __getattribute__(m_self, m_name):
            name = m_name.blueval
            if name not in fields:
                return OpSpec.NULL          # not a field: default lookup
            T = fields[name]

            def get(self: cls, attr: str) -> T:
                return getattr(self.__ll__, name)

            return OpSpec(get)

        @blue.metafunc
        def __setattr__(m_self, m_name, m_v):
            name = m_name.blueval
            if name not in fields:
                return OpSpec.NULL
            T = fields[name]

            def set(self: cls, attr: str, v: T) -> None:
                setattr(self.__ll__, name, v)

            return OpSpec(set)

        wrapper.add_method("__getattribute__", __getattribute__)
        wrapper.add_method("__setattr__", __setattr__)

        # "The class wins": a __new__ written by the user is kept.
        if "__new__" not in body.methods:

            @blue.metafunc
            def __new__(*args_m):
                arg_types = args_m.types

                def new(*args: *arg_types) -> cls:
                    self = cls.__make__(gc_alloc[Data](1))
                    self.__init__(*args)
                    return self

                return OpSpec(new)

            wrapper.add_method("__new__", __new__)

        super().define_from_classbody(wrapper)

        # 3. Define the data struct, now that the wrapper is defined.
        Data.define_from_classbody(ClassBody(fields=fields))

• class Record(StructType) is a metaclass (section 3). Its method define_from_classbody is called by the compiler, with self being the type to define: the declared Point, which is an instance of Record, since w_meta was set just before.

• body.fields is the ordered mapping of the lines x: i32 of the user’s class, and body.methods the one of its methods. The user’s methods are kept as they are: their self is already typed with the declared type, which is the wrapper.

• StructType.declare(fqn) creates a declared type, here for the data struct, whose w_meta is StructType. Like the compiler does for every class statement, it makes the name unique and registers the type as a global, which the C backend needs to write it. self.fqn.join("Data") builds its name from the one of the class.

• ClassBody() creates an empty body, and add_field, add_method and add_methods fill it (section 5). __ll__ is the only field of the wrapper: a pointer to the data.

• __getattribute__ and __setattr__ are metafunctions: the compiler runs them for each p.x and p.x = v, with the name known at compile time, and the OpSpec they return says which function to call. OpSpec.NULL means “not handled here”.

• super().define_from_classbody(wrapper) calls the method of StructType, which defines self as a plain struct from the body that we built.

• The data struct is defined last. A struct’s layout is computed when it is defined, from the size of each field. The wrapper contains only a pointer to Data, whose size does not depend on Data, so the wrapper can be defined while Data is only declared. Defining Data afterwards allows it to have fields of the type of the wrapper (a node of a linked list has a field next of its own class).

For this user code:

class Point(metaclass=Record):
    x: i32
    y: i32

    def __init__(self, x: i32, y: i32) -> None:
        self.x = x
        self.y = y

the compiler ends up with the following code, written here as if the calls selected by the OpSpecs were inlined:

@struct
class Point:
    __ll__: gc_ptr[Point_data]    # the only field: copying a `Point` copies the pointer

    def __init__(self: Point, x: i32, y: i32) -> None:
        self.__ll__.x = x         # `self.x = x`, once `__setattr__` has been resolved
        self.__ll__.y = y

    def __new__(x: i32, y: i32) -> Point:
        self = Point.__make__(gc_alloc[Point_data](1))   # allocate a Point_data
        self.__init__(x, y)                              # run the user's __init__
        return self

@struct
class Point_data:             # defined after Point; its real name is built by
    x: i32                    # `self.fqn.join("Data")`
    y: i32

What the real Record adds¶

The one of stdlib/metaclasses/record.spy also provides, each time by calling a helper function (section 6):

• __init__, __eq__, __ne__ and __repr__, generated from the fields, as Python’s @dataclass does. A method is generated only if the user’s class does not have it already (“the class wins”), so a written __init__ is kept.

• __is__, so that a is b works. It compares the two __ll__ pointers: two variables are the same object when they hold the same address. Without it, is is rejected on a struct.

• __erase__ and __unerase__, which turn an object into an untyped pointer and back. A protocol (section 6) can only receive the objects of metaclasses that provide them.

• The keyword implements, for the explicit conformance to protocols (section 6).

5. An API to add methods to a ClassBody¶

The compiler gives define_from_classbody a ClassBody, and a metaclass builds another one to pass to super(). This API is invented.

A body is mutable until it is passed to define_from_classbody. Defining a type from it (and from it only) is what gives the type its layout and its methods.

The signature is what a vtable needs to give a type to each of its slots (a FnPtr[[gc_ptr[Root], *argtypes], restype]). The class attributes (attrs, add_attr) are how metaclasses keep information for each other: a variant reads its cases in body.attrs, ClassMetaClass stores a dictionary __info__ that its subclasses read (section 6), and Protocol stores __protocol_methods__.

Types also get one read-only attribute, T.__fields__: the ordered {name: type} of the logical fields of any type. For a Record these are x and y, not __ll__, which is the physical field of the wrapper. A protocol uses it to check that a class has the attributes that it requires.

6. Which metaclasses in the standard library¶

The metaclasses are orthogonal: each one is a complete definition, and a class has exactly one metaclass. There are no stacked metaclasses and no decorators that patch a class. A metaclass differs from another by three independent choices:

• storage: a value (the fields side by side) or a pointer to heap-allocated data;

• dispatch: none, or single inheritance with a vtable (a table of function pointers, one per virtual method, used to find the right method at run time);

• generated methods: none, or __init__, __eq__, __ne__ and __repr__ written by the metaclass from the fields (this is what Python’s @dataclass does).

A metaclass inherits from another only for a real “is-a” relation, and a class that has a base must use a metaclass that is a subtype of the metaclass of its base (as in Python). Code is shared with helper functions, called inside define_from_classbody.

The names below are inventions. The file of each one is stdlib/metaclasses/<name>.spy (record.spy, class_metaclass.spy, dataclass.spy, protocol.spy, variant.spy).

• StructType is the default metaclass: a plain struct, with nothing generated. There is no metaclass for value structs with generated methods: __init__ cannot be generated (a value struct is immutable, so there is no object to initialize in place, and its constructor __new__ already exists), and __eq__ and __ne__ are already generated by StructType.

• Record is the heap-allocated class of section 4, without inheritance. It is the recommended replacement of Python’s dataclasses in new code.

• ClassMetaClass adds a pointer to a vtable and single inheritance to a heap class. The user writes __init__ and the other methods.

• DataClass is ClassMetaClass plus generated methods, to port Python code that uses @dataclass with inheritance (Antonio thinks that dataclasses should not support inheritance). The generated methods must include the fields of the base classes, which it reads from the class attribute __info__ that ClassMetaClass sets on every class.

• Protocol and Variant are as in the page on inheritance. A class conforms to a protocol without inheriting from it. In a Variant, Shape.Circle is the constructor (a static method of the variant), and the case type is Shape.__cases__["Circle"]. For match, the metaclass provides Shape.__tag__(self) (the tag) and Shape.__as_Circle__(self) (the payload, as the case struct), so that case Shape.Circle(r) becomes a test of the tag followed by a read of r in the payload.

The helper functions¶

The metaclasses share five blue functions, in _metaclass_helpers.spy. A new metaclass that wants the same features calls them too, with only the API of section 5.

The last four are used by Record and ClassMetaClass, and add_generated_methods and add_protocol_defaults by DataClass too.

Explicit conformance to a protocol: implements¶

A protocol can carry default methods (a method with a body, which a class gets for free), and a class receives them only if it declares that it conforms to the protocol (“explicit conformance”). It is spelled with a keyword, not class Circle(Shape): by the rule of section 3, the metaclass of class Circle(Shape) would be the one of Shape, Protocol, so Circle would be a protocol itself.

class Shape(metaclass=Protocol):
    def area(self) -> f64: ...           # required method: no body

    def show(self) -> None:              # default method: has a body
        print(self.area())


class Circle(metaclass=Record, implements=[Shape]):
    r: f64

    def __init__(self, r: f64) -> None:
        self.r = r

    def area(self) -> f64:
        return 3.14159 * self.r * self.r
    # no `show` here: it comes from Shape

The compiler passes the keyword to define_from_classbody(self, body, implements=()), which calls add_protocol_defaults. For Shape, it checks that Circle has area (otherwise a TypeError is raised when the class is defined) and adds to the body a copy of show bound to Circle: it is as if the user had written def show(self: Circle), so that self.area() is a direct call. A class that does not write implements conforms to the protocol only implicitly: it receives no default method.

A required attribute of a protocol (density: f64) is checked in the same way: the class must declare a field of exactly this type, which is read in __fields__ (section 5).

The keyword final¶

ClassMetaClass and DataClass accept the keyword argument final=True. As in Python, the compiler passes the keywords of a class statement to the metaclass, here as define_from_classbody(self, body, final=False). A metaclass that does not accept a keyword refuses it.

class Shape(metaclass=ClassMetaClass):
    def area(self) -> f64:
        return 0.0

class Circle(Shape, final=True):
    r: f64

    def area(self) -> f64:
        return 3.14159 * self.r * self.r

def total(c: Circle, s: Shape) -> f64:
    # `c.area()` is a direct call (`Circle` is final), `s.area()` goes through the vtable
    return c.area() + s.area()

A final class gives three guarantees:

1. No subclass. class Sub(Circle) is an error, raised when Sub is defined.

2. Static calls. If the static type of an object is a final class, the exact method is known at compile time, so the call is direct and the C compiler can inline it. For a non-final class, the object may be of a subclass, so the call goes through the vtable. This is the main interest of final: performance.

3. Possibly no vtable pointer. If a final class has no base, nothing can ever need the pointer. If the base has a vtable, the pointer stays, since an object of the class can be converted to the type of the base.

All three are done by ClassMetaClass, and the compiler does not know about final. For each method, the metaclass writes the function that the callers use: a direct call for a final class, a read of the vtable for the others. It also stores what the subclasses need in the class attribute __info__ (a dictionary, set with add_attr): in particular __info__["final"], which a subclass reads when it is defined to refuse the inheritance. In effect, Record is a final class without a base whose methods are generated.

final is a keyword and not a metaclass, because it combines with every metaclass that has inheritance (otherwise each would need a variant: FinalClassMetaClass, FinalDataClass). A decorator @final would not work either, since a decorator runs after the definition, and the layout (with or without a vtable pointer) is decided during it.

7. What SPy needs¶

The work is cut into small features, each one meant to be one pull request (PR) that can be tested alone. They are grouped in four phases:

• Preliminary: features that are useful by themselves and have nothing to do with metaclasses.

• Milestone 1: what Record, the first metaclass written in SPy, needs. No vtable is required.

• Milestone 2: what the metaclasses with inheritance and protocols need.

• Milestone 3: what Variant and the match statement need.

Inside a phase, the features are independent unless a “Depends on” says otherwise. Each has a stable identifier (P1, M1.3, ...).

Each milestone has two parts, and part B does not start before part A is done:

• Part A: the compiler supports the lowered code written by hand. The lowered code of a class is the code that its metaclass produces: ordinary SPy (@struct classes and functions) with no metaclass, as in the last listing of section 4. Here it is written by hand, in a .spy file, with a check file (functions that use the class) and a Python test. Part A needs no metaclass, no ClassBody and no w_meta.

• Part B: the metaprogramming. The metaclasses of stdlib/metaclasses/ generate the code that part A wrote by hand. The check functions are the same, and the metaclass is correct when it passes them. If they fail, the compiler is not the cause, since part A showed that it can compile the generated code.

Preliminary¶

☐ P1. Starred expressions in blue code. PR: spylang/spy#710 (in progress). (*x,), f(*args) and [a, *b], where all the values are known at compile time.

☐ P2. .keys(), .values() and .items() on interp_dict, the dictionary that blue code uses. PR: (to open). The metaclasses walk over body.fields and body.methods with them.

☐ P3. *args: *types in red code. PR: (to open). A red function can take a variable number of arguments whose types are given by a blue tuple types. Inside the function, args is a tuple with these types (see __new__ in section 4).

☐ P4. Unrolled for loops in red code. PR: (to open). A for loop whose iterable is known at compile time is replaced by one copy of its body per element. It includes unroll_range, for a range of integers, which the metaclasses use to loop over the fields when they generate __init__, __eq__, __ne__ and __repr__ (Antonio is working on it, September-October 2026).

☐ P5. Call with *args in red code. PR: (to open). f(*args) where args is a red tuple. The number of elements is part of the type of args, so the compiler can unroll a loop over the positions and produce (args[0], args[1], ...). Depends on P4.

☐ P6. isinstance(obj, T) in SPy code. PR: (to open). It makes isinstance(Point, StructType) true (section 3).

☐ P7. Importing a module from a directory of the library. PR: (to open). The files of stdlib/metaclasses/ import each other and are imported by user code. Today a module is a single file {name}.spy found in a directory of vm.path: there are no packages. Either packages are supported (from metaclasses.record import Record), or the directory is added to vm.path and the files are imported by their bare names (record, protocol, ...), which become top-level names.

Milestone 1: Record¶

Part A: the lowered code of Record, by hand¶

Check of part A. The first reference file is for a class with only fields, which is the simplest one that has all the generated methods of section 4. The user code, which the metaclass will receive in part B, is:

class Point(metaclass=Record):
    x: i32
    y: i32

Its lowered code, written by hand, is examples/exp_record/point_by_hand.spy. Part A is done when this file compiles and runs on the three backends (interp, doppler and C) and passes its checks: Point(1, 2) == Point(1, 2), Point(1, 2) != Point(1, 3), repr(Point(1, 2)) and p is q. The checks are in a separate file, so that part B runs the same ones on the Point generated by the metaclass.

Two features are needed for this check:

☐ M1.10. OpSpec.NULL from __getattribute__ and __setattr__ falls back to the default lookup. PR: (to open; the change is already in the branch, in spy/vm/modules/operator/attrop.py). Without it, reading self.__ll__ inside the hook calls the hook, which reads self.__ll__, forever.

☐ M1.11. The __is__ hook. PR: (to open). a is b calls __is__ of the type of a, as a == b calls __eq__. Today is is rejected on value types, and a struct is a value type, so a heap object (a struct that holds a pointer) can only be compared by comparing the pointers by hand.

Part B: the metaprogramming of Record¶

☐ M1.1. w_meta, w_base and a minimal StructType. PR: (to open). Every type gets the two fields of section 3, and StructType exists as an object that SPy code can name, with the minimum it needs to be the metaclass of the existing structs. After it, isinstance(Point, StructType) is true for an existing @struct class Point. The value of w_base for a class without base is object. It changes no behavior for the user. Depends on P6.

☐ M1.2. One generic declaration, and the backend asks the type for its layout. PR: (to open). Every class is declared as the same empty generic type, whatever its metaclass. The C backend, the pointer types and sizeof ask the type whether it has a layout, instead of checking for the Python class of structs (a possible criterion: w_meta is a subtype of StructType). Depends on M1.1.

☐ M1.3. Base class and metaclass= in the parser and the AST. PR: (to open). class C(Base, metaclass=M) is parsed and kept in the AST, instead of one kind that can only be @struct. It touches the parser and the AST only, so it can be done in parallel with M1.1.

☐ M1.4. A class whose base is a metaclass is a metaclass, and super() in its methods. PR: (to open). The rule of section 3. Depends on M1.1 and M1.3.

☐ M1.5. StructType and its API visible from SPy. PR: (to open). StructType.declare(fqn), define_from_classbody(self, body) on any declared type, FQN.join, and the attribute T.__fields__. Today they exist only in Python. Depends on M1.1 and M1.2.

☐ M1.6. ClassBody as an SPy object. PR: (to open). ClassBody with the API of section 5, including the Method objects. Depends on M1.5 and P2.

☐ M1.7. Default values in fields. PR: (to open). x: f64 = 0.0 in a class body (body.defaults). The first Record does not need it.

☐ M1.8. stdlib/metaclasses/record.spy, with the helper functions of section 6. It is done when the Point generated by Record passes the checks of part A. Depends on part A, P3, P4, P5, P7, M1.4 and M1.6.

Milestone 2: inheritance and protocols¶

These features need a vtable or fat pointers.

Part A: the lowered code, by hand¶

The two reference files below are the by-hand versions of the lowerings of the page on inheritance (Inheritance, variants and protocols: a design sketch, sections 3.3 and 5.3). They use only @struct classes, pointers and functions. As for Record, each one has a check file and a test.

☐ M2.1. Low-level pointers: FnPtr, static_ptr, untyped pointers, ptr_cast and addr_of. PR: (to open). What vtables and fat pointers (a data pointer and a vtable pointer) are built with. It is probably several PRs, one per name. A struct value built at compile time (a vtable) must become a constant in static memory when addr_of is applied, and a function must be accepted where a FnPtr is expected.

☐ M2.2. Layout of structs with a zero-size field. PR: (to open). A zero-size struct (a struct without fields) is, for example, the vtable of a root class without virtual methods, which the vtable of its subclass has as first field, __parent__. Today calc_layout aligns each field with (offset + (size - 1)) & ~(size - 1): for a field of size 0 this is 0, so the offset goes back to the start of the struct and the next fields overlap the previous ones. The rule must be fixed (a zero-size struct could have a size of 1, as in C++, since C does not allow an empty struct).

☐ M2.3. Single inheritance with a vtable, by hand: examples/exp_class/shape_by_hand.spy. The lowering of section 3.3 of the page on inheritance, also with the version where Circle is final. Depends on M2.1, M2.2 and M1.11.

☐ M2.4. Protocols with a fat pointer, by hand: examples/exp_protocol/shape_by_hand.spy. The lowering of sections 5.3 and 5.8 of the page on inheritance, including the copy of a default method written in the class Circle. Depends on M2.1.

Part B: the metaprogramming¶

☐ M2.5. Keyword arguments in a class statement. PR: (to open). class Circle(Shape, final=True) and class Circle(metaclass=Record, implements=[Shape]). The keywords are passed to define_from_classbody. Depends on M1.3 and M1.5.

☐ M2.6. Method.bind(T). PR: (to open). Needed for the default methods of a protocol (section 5). Depends on M1.6.

The metaclass files that these features make possible:

☐ M2.7. class_metaclass.spy (ClassMetaClass). Depends on M2.3, M2.5 and M1.6.

☐ M2.8. protocol.spy (Protocol, and the keyword implements of Record and ClassMetaClass). Depends on M2.4, M2.5 and M2.6.

☐ M2.9. dataclass.spy (DataClass, optional). Depends on M2.7.

Milestone 3: variants and match¶

A variant is a closed set of cases, and match is how user code takes it apart.

Part A: the lowered code, by hand¶

The reference file below is the by-hand version of the lowering of section 4.3 of the page on inheritance. It uses only @struct classes, pointers and functions, and has a check file and a test.

☐ M3.1. ll_union. PR: (to open). A union of the structs of the cases, with a named member per case (payload.Circle).

☐ M3.2. Variants, by hand: examples/exp_variant/shape_by_hand.spy. The lowering of section 4.3 of the page on inheritance: a tag and an ll_union, with one constructor per case. Depends on M2.1 and M3.1.

Part B: the metaprogramming¶

☐ M3.3. Nested class statements in a class body. PR: (to open). Needed to write the cases of a variant inside its class. The nested types appear in body.attrs.

☐ M3.4. match on a type that provides __tag__ and __as_<Case>__. PR: (to open). case Shape.Circle(r) becomes a test of Shape.__tag__(self) followed by a read of r in Shape.__as_Circle__(self) (section 6). The compiler only calls these two methods of the type of the subject, so it knows nothing about variants. It can be tested with methods written by hand on a struct.

☐ M3.5. variant.spy (Variant). It is done when the match statements of the check file pass on the variant generated by the metaclass. Depends on M3.2, M3.3 and M3.4.

8. Open questions and limits¶

Questions that the plan does not answer yet:

1. Typing a method again. Is Method.bind (typing the function again from its AST) the right mechanism for the default methods of a protocol, or should a protocol write them as blue generics of the class (def show(Self), which returns the function), so that no new typing is needed?

2. Blue functions in a body. How does a body carry blue metafunctions such as __getattribute__? The implementations assume that add_method accepts them.

3. Static type of a declared name. The local name Point is created at declaration with the static type of a plain type. To check: the uses of a blue name rely on the type of the value (w_meta, once set), not on this static type.

4. Order of the definitions. The rule “define the wrapper first, then what it points to” is the one of all the metaclasses. Two classes whose data contain a field of the other’s type would probably fail in either order (not tested), and sizeof of a declared but undefined struct fails.

Not handled by the tentative implementations:

• default values in the generated __init__ of a heap class (the *args: *types form has no place for them);

• super().method() and static or class methods in a class with a vtable (every method whose name does not start with __ is virtual);

• the conversion of a Protocol to a smaller protocol;

• a variant stored by value (the page on inheritance, section 4.6).