Reuse the C ABI bindings to generate Qt bindings for other non-Go languages

[mappu]

No description provided.

[arnetheduck]

Played around a bit with this idea in arnetheduck#1 - works fine - miqt codebase makes it very easy to adapt/copypaste around, great stuff ;)

Here's a laundry list of questions and random thoughts that came to my mind while doing the POC, in no particular order:

• Instead of converting miqt_string/map/array in the C wrapper, it maybe makes sense to export these as-is from qt by wrapping QString et al in some shape or form - generics make this a bit tricky and perhaps a middle ground would have to be found here - however, for strings and bytes at least, the Nim wrappings at least would be better off exposing QString to the "user" of the library (much like QT itself allows a choice between std::string and QString with converters
• the clang->il conversion could probably stay a little bit closer to C as well (naming, etc)
• some of the cgo:isms aren't really needed, like _cgo_export.h - all the information is already there in the clang AST
• isSubClass looks redundant - since destructors are virtual (are they, all?), they should not need the extra dynamic_cast - potentially, an ownership tracker could be introduced here
• virtual inheritance looks unnecessary - use QAbstractButtton as example, nothing will ever inherit from MiqtQAbstractButton so it doesn't look like it needs to virtually inherit from QAbstractButton - MitqQPushButton for example never touches MiqtQAbstractButton.

Regarding non-go languages, one candidate is c itself ;) Looking at the generated header files, they actually look very easy to use from a native C project - seen as such:

• the clang step could trivially be done using a Makefile
• the c++ -> c step could potentially be done using Qt/C++/CMake itself to reduce the language tax for any one target language - c++ is needed anyway - though this tax is somewhat mitigated by this being a "generator"-only concern - users of wrappers don't need the extra foreign language.
• the upstream shiboken6 project works more or less the same way - either its generator or the xml files it outputs could maybe serve as a predigested source of binding information - there are both benefits and risks here.

As a side note, the above is more or less how the existing Nim QML bindings are structured as well, via DOtherSide where a single C binding is used to support multiple languages - it has fallen out of maintenance however, so here we are.

[rcalixte]

• virtual inheritance looks unnecessary - use QAbstractButtton as example, nothing will ever inherit from MiqtQAbstractButton so it doesn't look like it needs to virtually inherit from QAbstractButton - MitqQPushButton for example never touches MiqtQAbstractButton.

This is necessary for Qt's ecosystem, namely event handling. Without the virtual methods, customization is closed off. I don't know if this is true for QML since I'm not a huge fan of it.

Regarding non-go languages, one candidate is c itself ;) Looking at the generated header files, they actually look very easy to use from a native C project - seen as such:

I haven't really publicized it anywhere too large but I've been working on releasing bindings for C and Zig. I have both in a working state for Qt 6.4 but I need to get some things sorted for Qt 6.7 along with more examples and some other things before I'm comfortable saying more publicly or releasing them to the masses.

• the c++ -> c step could potentially be done using Qt/C++/CMake itself to reduce the language tax for any one target language - c++ is needed anyway - though this tax is somewhat mitigated by this being a "generator"-only concern - users of wrappers don't need the extra foreign language.

On my end, I was comfortable leaving things in Go so as to try to keep with "upstream" (being Go bindings) so that future improvements would be simpler to implement. Unfortunately, I've made quite a few structural changes so it's not as easy as rebasing but I highly value the idea of something as close to a shared codebase for such a large and complex undertaking where the goal is consumption rather than maintenance. (I'm also hoping to be able to backport some of the changes/decisions to benefit the Go bindings as well.) Still, I've mostly ended up with two hard forks. Of course, you are free to do what you feel is best. I look forward to seeing what you come up with!

[arnetheduck]

• virtual inheritance looks unnecessary - use QAbstractButtton as example, nothing will ever inherit from MiqtQAbstractButton so it doesn't look like it needs to virtually inherit from QAbstractButton - MitqQPushButton for example never touches MiqtQAbstractButton.

This is necessary for Qt's ecosystem, namely event handling. Without the virtual methods, customization is closed off. I don't know if this is true for QML since I'm not a huge fan of it.

Ah sorry, I meant the inheritance itself, not the methods, ie the virtual after public in class MiqtVirtualQAbstractButton : public virtual QAbstractButton - I might very well be missing some detail though.

I have both

Nice! Looking forward to seeing them come out. Nim is somewhat closer to zig when it comes to wrapping things (vs cgo / go that brings more opinion to the table) so this might be a better starting point for nim as well.

highly value the idea of something as close to a shared codebase
Still, I've mostly ended up with two hard forks.

Indeed, and interesting. I'll see if I can extract some minor things from the nim poc as well - with luck, we'd be going in similar directions and then we can see how far off we end up.

One thing I wanted to reach for while messing with the poc was some basic template engine to make the generated code pop out more vs the go infrastructure of various quotes and plusses, but again not a critical concern for hopefully write-once stuff like this.

[rcalixte]

Nice! Looking forward to seeing them come out. Nim is somewhat closer to zig when it comes to wrapping things (vs cgo / go that brings more opinion to the table) so this might be a better starting point for nim as well.

I've thought about this quite a bit. Whether there will be a larger audience for consuming these bindings/wrappers or if there will be a larger audience for porting them. Languages like Nim, D, C3, etc. have crossed my mind for where interested folks might find value as long as they don't mind a little bit of Go.

One thing I wanted to reach for while messing with the poc was some basic template engine to make the generated code pop out more vs the go infrastructure of various quotes and plusses, but again not a critical concern for hopefully write-once stuff like this.

While trying to document some of this for some others, I've realized the large intersection of knowledge required to execute on this: C++, Qt, Go, C ABI, and then whatever target language. That is a complex set and Qt is a moving target with every passing version. This doesn't include the auxiliary tooling like Clang. Just thinking about it leads to burnout. 😅

But as for write-once, I don't know if that is possible. Given what has just changed with Clang (in the issue you recently opened) and upstream Qt versions able to change/introduce/deprecate significant portions of the toolkit, maintenance will be required. Still, as I've told @mappu, these bindings strike me as the best I've seen for Qt to hopefully minimize the long-term maintenance burden. The bindings for Qt 6.4 mostly work from the standpoint of being able to be dynamically bound on systems with Qt 6.7 but that is likely to remain true for the core components as long as they do not change too much. The less-traveled components may not be so stable. It remains to be seen over time how this will play out but I am optimistic. The main thing I've done to further try to simplify things is only focus on Qt 6 since Qt 5 will be deprecated by the middle of 2025 and these are novel bindings for the respective lanuages. I believe the Go bindings will maintain support for Qt 5.

[rcalixte]

Nice! Looking forward to seeing them come out. Nim is somewhat closer to zig when it comes to wrapping things (vs cgo / go that brings more opinion to the table) so this might be a better starting point for nim as well.

I've thought about this quite a bit. Whether there will be a larger audience for consuming these bindings/wrappers or if there will be a larger audience for porting them. Languages like Nim, D, C3, etc. have crossed my mind for where interested folks might find value as long as they don't mind a little bit of Go.

Pursuant to these, I think I'm going to alter my plans and get the Qt 6.4 bindings/wrappers ready for release instead. I don't like that it will have a short-term limitation of contributors to the library of requiring Qt 6.4 but the resulting bindings/wrappers will be mostly usable for Qt 6.4+ and will allow for unblocking other interested parties who want to use a base closer to C ABI. If it means more collaborators, I think it will be worth it.

[arnetheduck]

While trying to document some of this for some others, I've realized the large intersection of knowledge required to execute on this: C++, Qt, Go, C ABI, and then whatever target language.

Agree here - this is where my loose thought of having this in C/C++ came from, to reduce the surface area of "getting started" with contributions - targeting C in particular ensures that the core bindings are "functional enough" for all downstream languages without requiring knowledge in any one that you don't already have to know (assuming you know c++) - I maintain an llvm-based compiler for Nim, where the bindings are done more or less like this, ie a single C++->C binding set and each language (zig as well btw!) builds on top of that though the sheer scale of Qt makes it a different beast somewhat.

Anyway, this was mostly a food-for-thought comment that I thought might be relevant to bring up in an issue like this.

I believe the Go bindings will maintain support for Qt 5.

The Nim bindings would eventually be in this boat too I suspect as they would be used in part to make the qt5/6 transition for some projects, should the PoC move ahead - but there's a few ifs and buts to clear before that happens ;)

[mappu]

Miqt has grown some (small) handwritten packages; other language bindings will need to reimplement qt{,6}/mainthread.

To allow mainthread/ to be automatically generated, the bindings layer would have to support whatever is missing for QMetaObject::invokeMethod. I think this is the same issue as #122, but haven't looked into it yet.

[rcalixte]

Pursuant to these, I think I'm going to alter my plans and get the Qt 6.4 bindings/wrappers ready for release instead.

As much as I want to semi-rush this, I'm thinking of a mid-February release date to follow on the release of Zig 0.14.0 (edit: delayed until March). I don't have anything even pushed online yet but I do want to share with both of you if I have something ready beforehand since I think it might address some of the recently opened issues.

[arnetheduck]

the bindings layer would have to support whatever is missing for QMetaObject::invokeMethod.

Two things mainly:

• support for void**, I've added it to https://github.com/arnetheduck/nim-miqt-poc/pull/1/files#diff-1bfa2e7175ac24f662138857b0d45202710d2d20dd6a87556503a6ebb027814fR484 - with void**, qt_metacall becomes override-able which is what invokeMethod and pretty much every thing else in QMetaObject ends up calling
• a generator for the metaobject introspection data - ie what moc usually does - I wrote one here for nimqml: https://github.com/arnetheduck/nimqml-miqt-poc/pull/1/files#diff-7f8cf6a029f39b4f5a3072e98374fd30cb73691443de443ad41ca705b82824fdR447 - it's quite tedious but what it does is expose signals/slots/methods and then everything else "just works" - with that in place, half of the nimqml examples work, but eventually run into Compile-time detection for calling unsupported virtual method override #137 - ie the changes that make invokeMethod possible also enable QtQuick/QML (more or less, with a few more exclusions).

[arnetheduck]

release date

I've mostly been extracting small patches and issues from the nim poc that seemed like overall improvements no matter if the poc goes ahead or not - lmk if any of them run counter to your work - they're all entirely discardable from my point of view since it's just prototype code - I think the most interesting thing that would make things a lot easier is actually the override change described here: #133 (comment) - that would also simplify calling the base class method because the bindings can expose a "default" implementation with the base class implementations as the "default" pointers - code wanting call super then has as an easy option to do so but above all, I think this would simplify memory management / lifetimes of callback closures.

[rcalixte]

lmk if any of them run counter to your work

Oh, I passed the "difficult rebase" threshold a long time ago. 😅

At this point, I would need to hand-patch anything that can't be applied but that's on me. Similar to you, I started small and then...

I kept most of the Go name mangling and I really tried to keep the codebases as similar as possible so that code could hopefully be shared between all of the projects.

I think the most interesting thing that would make things a lot easier is actually the override change described here

I actually took an entirely different approach here. I'm using the object's pointer for all of the inherited methods since the vtable alignment works as long as it is upcasting (which is all the bindings/wrappers make available by default). Maybe I'm wrong here. 😅

I'd already removed isSubClass entirely from the bindings and wrappers. I've modified the included C library, etc. etc. The reason I initially got a bit antsy is because you're overlapping a lot of the same work that I had (except you've retained Qt 5). At the same time, we also diverge quite a bit so it looks like it might not be as easy as I'd initially wanted to have multiple languages using the same core codebase. As long as we don't venture off too much, I think we can still collaborate productively on maintaining the respective bindings. I still fully intend to be a consumer of the Go bindings once I'm done so it would be nice to see them consistently sharpened by the work from other languages.

I initially started with just implementing the signals/slots in C ABI instead of the Go callbacks, removing the Go headers (similar as you did), etc. etc. It spiraled a lot more than I wanted but it is also reasonably simple to grok the code and also maintain (🤞🏼). Assuming that I haven't categorically screwed this up, I'm hopeful that some of these changes are well-received. I will admit that it is a little bit of a relief to see that you've identified some of the same things that I have. Hopefully, I haven't completely wrecked this. 😅

At the moment, I'm primarily working on getting examples done, documentation, bug fixes, packaging, and keeping an eye on changes upstream. With this additional time before a release, I'm thinking I'll implement some form of the mainThread functionality that was recently added. I might also cherry pick from your branch if time permits and there's anything portable. I'm intrigued by the qt_metacall work you've done already since I left that out. 😅 (I found other ways to leverage Qt's meta-object system so I didn't think it was a glaring hole or concern for the bindings but it wouldn't hurt to have more options.)

TLDR: I started with a soft fork that eventually became a hard fork but that shouldn't stop us all from being able to collaborate well.

[arnetheduck]

Miqt has grown some (small) handwritten packages;

This is probably inevitable for any binding work, ie that there's a home for hand-written stuff to deal with the exceptions and corner cases so might as well make them "regular" in the generator somehow.

Go name mangling

As it happens "initial capital" is used for types instead of "public" in nim, so that cause a little bit of mess - easier to start from the Qt source names than "undo" what the go bindings need, also for reasons of grep friendliness when looking through the actual qt sources ;)

Nim can offer almost 1:1 naming with respect to upstream Qt because it supports overloading, so longer term any "conflict renaming" would also have to live in the "language-specific" section for things to work well. The point of the naming PR was more that this hopefully is "not unreasonable" change for the go bindings (it seems pretty clean code-wise at least) that at the same time has the potential to make future patchwork easier.

collaborate

In playgrounds like this, I try maintain my changes as a set of commits on top of the upstream so that I can upstream features individually and then rebase and reorder - this is still viable in the case of nim-miqt, but ymmv of course. There's a few more commits in that branch (which keeps getting rebased) that might be of general interest, now or eventually.

[rcalixte]

In playgrounds like this, I try maintain my changes as a set of commits on top of the upstream so that I can upstream features individually and then rebase and reorder - this is still viable in the case of nim-miqt, but ymmv of course.

I actually utilize mostly the same workflow... typically. I had a rough rebase in around November though so I broke things off at that point. Millions of lines of code at this point and all striving for mostly the same goals but in even slightly different ways. Even C and Zig are their own hard forks after I've split them up. The Venn diagram at least remains intact though.

[arnetheduck]

whatever is missing for QMetaObject::invokeMethod.

one more: access to staticMetaObject, for example via #16

[mappu]

a generator for the metaobject introspection data - ie what moc usually does

I looked into this as a possible option when implementing the qt/mainthread packages. However, the moc tool produces a header that is tied to a specific semver-minor version of Qt. The same moc output (allegedly) cannot be used for both Qt 6.4 and Qt 6.8. That makes it difficult to support multiple Qt 6 versions from one codebase.

[arnetheduck]

cannot be used for both Qt 6.4 and Qt 6.8.

It's abi-backwards-compatible within all minor releases - ie the data generated by moc 6.4 can be used with qt 6.8 but not the other way around. If you only generate 6.0-level data, you're fine for all 6.x releases - this is aided by a revision field in the metadata header, used like so

Also, the changes are decently small, even between the latest major releases - ie I looked at 5.15 vs 6.x and while they're different, it's mostly detail, the overall structure remains the same.

[arnetheduck]

First, a little success story: the miqt-based nim poc now has reached the quality level that it can replace (and extend) DOtherSide (the "manual" C binding), verifying this with a port of an actual app: status-im/status-desktop#17270

Separately, a second POC of the nim bindings is now available which further expand on treating the C bindings as a "stand-alone" binding and then importing those into a language projection.

There are two major / notable differences here to consider:

• the structure of the project changes to accommodate qt versions in a different way, following upstream conventions more closely - ie "versions" of bindings are now available from separate branches/repositories and the "modules" (core, widgets, gui etc) correspond 1:1 to upstream QT - this is important above all because qt itself is shipped as a collection of shared libraries - it's easier to link to the library correctly when each binding module corresponds to exactly one shared library.
• the binding generator itself becomes a CLI - the poc is not quite there yet but most changes in it have already been done so that instead of having a go file with a list of qt modules, one would pass in the name/pkgconfig of the module (ie QtCore/Qt5Core), and the generator takes it from there
  • the remaining blocker here is the recognition of classes from "other" qt modules than the one currently being processed, ie when processing widgets, core must be "loaded" in the internal miqt state for qt types to be recognised - this could be resolved by reading state from transitive #include directives instead of the current strategy of discarding them and relying on a particular processing order

[rcalixte]

Hi guys, they're finally public!

• https://github.com/rcalixte/libqt6c
• https://github.com/rcalixte/libqt6c-examples
• https://github.com/rcalixte/libqt6zig
• https://github.com/rcalixte/libqt6zig-examples

There's a lot to catch up on and a number of things to discuss. I'm not sure where to start but I have been reading through the issues. @arnetheduck, I think you'll see that the memory allocations for more manual cleanups match what you're working on and hopefully they can be instructive for you there (assuming I haven't screwed anything up). I don't think all the issues you've identified have been addressed.

I ended up separating out the virtual classes to separate files because I'd experimented with virtual inheritance among the virtual classes. I got it working but ended up removing it because it didn't appear to add any value (and I was worried about the long-term maintenance of likely future diamond inheritance problems). I saw the work being done on friend functions and while that approach is slightly less code, I'd already opted on a different approach for the protected methods so I mostly stuck with it. The same applies for the callbacks and overrides. I did end up using the protected methods implementations though since those were much better than what I'd clunked together. You'll notice that they're appended to the existing virtual methods and then iterated together. It appears to be working?

Overall, I got very greedy with the approach to the libraries, trying to get as much of the bindable surface areas as possible. This resulted in a lot of ugly hacks and brute force solutions. Apologies. The cleanup work is in flight to get the code back to a more readable state. If you happen to run the bindings generator, you'll also notice that the runtime is extended. It was originally six seconds when I started and right now, the C bindings take about a minute while Zig is about half of that. The timing will vary based on your hardware but the point is that these are taking much longer than the upstream. I'm also planning on some optimizations there but it wasn't urgent for the release so it's been left that way for now. I'll try to dig into the Clang changes again and hopefully have something closer to a resolution. I have some ideas I want to plug through based on the last look I did. Hopefully we can put off dealing with Clang 20 for a little while (or it's hopefully a less troublesome upgrade from 19.x).

My goal was to spend more time consuming than maintaining but right now, I am deep in the ledger in the wrong direction. While that won't likely change in the short-term, I can't wait to get to a point where I can really put all of these libraries through their paces.

The only other note is that I think Debian 13 will ship with Qt 6.8 if things go well. That will probably be the next version that I target and then most likely I'll wait for Debian 14, patching what might need to be patched along the way until then. Qt 6.9 is still in beta and will likely be released before Debian 13 but hopefully it'll be okay.

Up next is Nim!

[arnetheduck]

Funny coincidence - I pushed https://github.com/seaqt/seaqt-gen just yesterday (with links to C/Nim repos for Qt5/6), along with a few Nim projects and bindings - made it an org so that anyone with a qt binding itch to scratch can be part of it.

Happy to sift through your changes to see where we align, but I only see the generated code in those repos rather than the gen itself. Will the gen changes you've made be pushed as well?

Re Qt versions, one thing Qt does well is ABI stability - ie the only reason to use a later version is if you really need the new features but there are advantages to using the lowest possible version as well (6.2 for example, which is LTS) since binaries will then be compatible with all reasonable qt6 versions.

At some point I imagine the easiest thing to do might actually be to create a custom docker image with each minor qt version built from source on some random fixed distro so that it's not tied to any particular debian release schedule, though that's far down the prio list for me at least. That would fix clang at a specific version, much like currently.

The major "gap" right now is ownership tracking - ie when you call something like QToolbar::addWidget, the toolbar (this) becomes owner of the instance you pass in, but if you call QObject::setParent, this becomes the ownee - this is slightly annoying because in Nim, one can manage memory in a way similar to unique_ptr but this requires tracking the owner of the instance faithfully - not sure if there's a clear and obvious rule/pattern for that kind of stuff - QtJambi for example looks like it enumerates ownership quirks like this explicitly per function - perhaps there's a better way, but it's certainly something interesting to look into. Less of a problem for C, but certainly something go could encounter if the gc gets involved.

The other thing is exposing member data - this is necessary for meta-object support, ie staticMetaData along with a few more things - this would certainly be an "upstreamable" item that would benefit miqt and zig as well, but there's obviously the go/zig part to do then.

Anyway, congrats!

[rcalixte]

Happy to sift through your changes to see where we align, but I only see the generated code in those repos rather than the gen itself. Will the gen changes you've made be pushed as well?

Everything should be under the genbindings but I see now that they got clobbered by the .gitignore. Sigh. Fixed... Sigh.

[rcalixte]

Re Qt versions, one thing Qt does well is ABI stability - ie the only reason to use a later version is if you really need the new features

While they are certainly more stable, I have noticed breaking changes in the ABI for Qt. Enum values changing, function headers, etc. etc. I've seen some segfaults on distros with Qt 6.8 and I haven't dug deep yet, but I'm assuming it's something along those lines leading to the different outcomes. The stable/mature paths are good though. It at least meant that most of the examples compiled easily. 😅

At some point I imagine the easiest thing to do might actually be to create a custom docker image with each minor qt version built from source on some random fixed distro so that it's not tied to any particular debian release schedule, though that's far down the prio list for me at least. That would fix clang at a specific version, much like currently.

I was only hoping to release updates every few versions unless there was a drastic need. Debian's release schedule and stability line up pretty nicely for that. Users on faster-moving distros might feel differently. We'll see where the road takes us!

The other thing is exposing member data - this is necessary for meta-object support, ie staticMetaData along with a few more things - this would certainly be an "upstreamable" item that would benefit miqt and zig as well, but there's obviously the go/zig part to do then.

I saw this and wasn't sure what to make of it. I still have a lot to learn when it comes to Qt. The one thing I know would be helpful that I didn't get done (the hack is currently commented out) are the converter and mutable view functions for QMetaType. I think they'll be easier to bind in newer Qt but I could be wrong. I started down the path of function pointers in general so that it wouldn't need to be a hack but didn't get that done yet either.

There's so many fascinating things about working on these bindings. I've learned a lot and while I wish it was less painful, I'm still looking forward to doing more. I really want to get through the backlog of apps I want to write too.

[arnetheduck]

I saw this and wasn't sure what to make of it.

https://woboq.com/blog/how-qt-signals-slots-work.html is a great resource for metaobject internals! They're needed if you want the users of your bindings to declare their own signals and slots and have these accessible from QML interfaces for example.

[rcalixte]

I saw this and wasn't sure what to make of it.

https://woboq.com/blog/how-qt-signals-slots-work.html is a great resource for metaobject internals! They're needed if you want the users of your bindings to declare their own signals and slots and have these accessible from QML interfaces for example.

Ah. I saw your QML work. I don't know if that's a target yet.

[rcalixte]

Funny coincidence - I pushed https://github.com/seaqt/seaqt-gen just yesterday (with links to C/Nim repos for Qt5/6), along with a few Nim projects and bindings - made it an org so that anyone with a qt binding itch to scratch can be part of it.

I keep thinking if the vtable approach is where things should go despite the complications that it adds. I opted for a simpler approach because I want maintenance to be easy so that I have time to develop some applications but if there's core functionality that is nerfed because of that, it might not be worth it. We'll see where this takes us...

The directory structure is interesting. I can see the appeal.

[mappu]

@rcalixte @arnetheduck Congratulations to both of you, on the releases!

You've both been incredibly helpful and it's been a pleasure working together.

[arnetheduck]

the https://github.com/seaqt/seaqt-gen/tree/seaqt-c branch now contains a version of genbindings that produces only C code (ie no go/nim/etc) - on that branch, there is a bunch of commits that could be backported to other languages potentially, though they obviously require adaptations to the language projection as well.