Skip to content

[WIP] Interoperability with other Python binding frameworks#1140

Open
oremanj wants to merge 9 commits intowjakob:masterfrom
oremanj:interop
Open

[WIP] Interoperability with other Python binding frameworks#1140
oremanj wants to merge 9 commits intowjakob:masterfrom
oremanj:interop

Conversation

@oremanj
Copy link
Contributor

@oremanj oremanj commented Aug 17, 2025

See also pybind/pybind11#5800, the same feature for pybind11.

pymetabind is a proposed standard for Python <-> C-ish binding frameworks to be able to find and work with each other's types. For example, assuming versions of both nanobind and pybind11 that have adopted this standard, it would allow a nanobind-bound function to accept a parameter whose type is bound using pybind11, or to return such a type, or vice versa. Interoperability between different ABI versions or different domains of the same framework is supported under the same terms as interoperability between different frameworks. Compared to pybind11's _pybind11_conduit_v1_ API, this one also supports implicit conversions and to-Python conversions, and should have significantly less overhead.

The essence of this technique has been in use in production by my employer for a couple of years now to enable a large amount of pybind11 binding code to be ported to nanobind one compilation unit at a time. Almost everything that works natively works across framework boundaries too, at only a minor performance cost. Inheritance relationships and relinquishment (from-Python conversion of unique_ptr<T>) don't work cross-framework, but those are the only limitations I'm aware of.

This PR adds nanobind support for exposing nanobind types to pymetabind for other frameworks to use ("exporting") and using other frameworks' types that they have exposed to pymetabind ("importing"). Types bound by a different framework than the extension module's own nanobind domain are called "foreign". There are some internals changes to allow foreign types to be represented in the same type maps as native nanobind types; this also includes an updated version of the per-thread fast c2p map that allows safe removal of types (since we can make our own types immortal but we can't force everyone else to make their types immortal). It is possible to compile nanobind without the code to support interop, using the new cmake option NO_INTEROP.

Current status: nominally code complete and existing tests pass, but I haven't added interop-specific tests or public-facing docs yet.

Performance: I have not yet measured the performance impact of this change, but I expect it to be quite low in situations where the foreign bindings don't need to be used. The new type_c2p_fast caches negative lookups, and we note whether any foreign bindings exist for a C++ type at the same time as we look up the nanobind type for it. If any foreign bindings have been imported, we do need to look up in type_c2p_fast before failing in some cases where we previously could avoid a lookup completely. When the foreign bindings do need to be used to perform a cast, they require a second c2p_fast lookup and some likely-modest indirection overhead.

Memory cost: Exporting a type allocates a 56-byte structure, a capsule object to wrap it, and adds that capsule object to the type's dictionary. Importing a type adds a new entry to the type_c2p_slow map.

Code size: With NO_INTEROP, size libnanobind.a adds up to 8533 bytes smaller than baseline on my machine (an arm64 mac), probably due to reusing nb_ptr_map for the type_c2p_fast map. Without NO_INTEROP libnanobind.a is 8983 bytes larger than baseline.

Things that need to happen before this can be released:
[x] add user-facing documentation
[x] add unit tests
[ ] test correctness of nanobind/pybind11 interop
[ ] test performance
[ ] solicit feedback from maintainers of other binding libraries
[ ] release pymetabind v1.0, incorporating said feedback

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants