Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build a C interface to taskchampion #332

Merged
merged 95 commits into from
Mar 4, 2022

Conversation

djmitche
Copy link
Collaborator

I thought I'd get a start on this, and so far it's going OK.

Among other things, this would make it possible to use TaskChampion as the "backend" to TaskWarrior. It's not clear that's the best plan yet, but even if we don't do so I think this will be a nice addition.

@djmitche djmitche requested a review from dbr as a code owner January 24, 2022 04:25
Copy link
Collaborator

@dbr dbr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good 🥳

Aside from the one error handling note, the C API looks about like I'd expect such a thing to look. I kept thinking "oh that API looks a bit accident prone, what if.." before remembering "C" 🙄

I guess the main test of this would be in actually trying to integrate it with TW (obviously with how it works, but also things like how it integrates with build-systems)

/// undone.
#[no_mangle]
pub extern "C" fn tc_replica_undo<'a>(rep: *mut TCReplica) -> i32 {
wrap(rep, |rep| Ok(if rep.undo()? { 1 } else { 0 }), -1)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could maybe return some kind of enum instead of numbers? So on C side it would be end up being roughly identical, but would allow doing tc_replica_undo(...) == TC_UNDO_SUCCESS

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

..although it probably makes sense to experiment with error handling on tc_replica_sync as that has more interesting set of errors to deal with (mainly "errors with with error messages")

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the enum idea. I don't know how many functions will have the equivalent of Result<bool>, so I'll need to think about how to generalize that. In any case, the error message for any of these is handled with tc_replica_error, so that should be OK.

@djmitche
Copy link
Collaborator Author

Thanks for the check!

How does the TCString support strike you? It's basically Cow but with CStr, CString, and String. It seems like something like this would already exist, but a bunch of searching didn't find anything.

Copy link
Collaborator

@dbr dbr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how tedious it to implement but would be better to include some kind of plain-text .sql file and generate the .sqlite3 file from this (or maybe even generate it from the main Rust lib?), instead of checking in the slightly-opaque binary integration-tests/test-db/taskchampion.sqlite3

lib/src/string.rs Outdated Show resolved Hide resolved
integration-tests/build.rs Outdated Show resolved Hide resolved
@djmitche
Copy link
Collaborator Author

Not sure how tedious it to implement but would be better to include some kind of plain-text .sql file and generate the .sqlite3 file from this (or maybe even generate it from the main Rust lib?), instead of checking in the slightly-opaque binary integration-tests/test-db/taskchampion.sqlite3

Oops, that's not supposed to be checked in :)

@dbr
Copy link
Collaborator

dbr commented Jan 26, 2022

How does the TCString support strike you? It's basically Cow but with CStr, CString, and String. It seems like something like this would already exist, but a bunch of searching didn't find anything.

It looks perfectly reasonable on first glance, and I agree it definitely seems like something that probably already exists (but such crates can be really hard to search for). @thomcc has a lot more experience with Rust strings+FFI things and may know of something?

@djmitche
Copy link
Collaborator Author

Thanks! I got directed toward ustr but that doesn't look useful (and certainly not the "never frees strings" part!).

@thomcc
Copy link

thomcc commented Jan 26, 2022

I don't know of a great utility crate for strings in FFI (I was pretty happy with the string support I wrote for https://crates.io/crates/ffi-support, but it doesn't support everything your TCString does). I think what you have here is probably not something you'll find off the shelf on crates.io.

I'm sick now, so can't give a thorough review, but did a quick skim, and I will note a lot of the code here is unsound, at least technically — Many safe functions accept a raw pointer, and dereference it (or Box::from_raw it). I'm not sure how much that matters to you, though, It's possibly fine given that this seems like an application not a library, and it wasn't clear if these cases are ever public.

P.S. Surprised to see two folks I (vaguely) know interacting :)

@djmitche
Copy link
Collaborator Author

:waves: I suppose Rust is a bit of a small world :)

My understanding is that FFI necessarily involves unsafe Rust, but I'm not sure how what I've done is unsound. Is it specifically functions that aren't export "C" { .. } like

impl<'a> TCString<'a> {
    pub(crate) fn from_arg(tcstring: *mut TCString<'a>) -> Self {
        debug_assert!(!tcstring.is_null());
        *(unsafe { Box::from_raw(tcstring) })
    }
...

, and would marking that function unsafe address the unsoundess? No need to answer now if you're not feeling well -- I'm sure this PR will be open for a while yet :)

@thomcc
Copy link

thomcc commented Jan 26, 2022

would marking that function unsafe address the unsoundess

Yes. Unsoundness is more or less "you can cause undefined behavior with this safe function". The issue I'm describing is just that the function isn't marked as unsafe, which is why I said it might not matter to you.

@djmitche
Copy link
Collaborator Author

OK, thanks -- that's easy to fix! I'm also planning to go back to every unsafe { .. } and add a comment as to why it's safe.

@djmitche
Copy link
Collaborator Author

OK, updated to make TCString pass-by-value (which was a major change!)

djmitche added a commit to djmitche/taskchampion that referenced this pull request Feb 22, 2022
djmitche added a commit to djmitche/taskchampion that referenced this pull request Feb 24, 2022
djmitche added a commit to djmitche/taskchampion that referenced this pull request Feb 24, 2022
@djmitche
Copy link
Collaborator Author

Looks like I need to work on build-system compatibility on mac, but otherwise I'd love a review of this.

@djmitche
Copy link
Collaborator Author

warning: warning: /Applications/Xcode_13.2.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: /Users/runner/work/taskchampion/taskchampion/target/debug/build/integration-tests-858ae231f57980e3/out/libbindings-tests.a(libtaskchampion.dylib) is a dynamic library, not added to the static library

Hey Apple, maybe "I didn't link the thing you asked me to link" should be more than a warning?

@djmitche
Copy link
Collaborator Author

Looks like this is rust-lang/cc-rs#250 -- it just happens to work on linux :(

@djmitche
Copy link
Collaborator Author

  = note: taskchampion.lib(getrandom-719baf4ba8413e3e.getrandom.5orpj5wj-cgu.4.rcgu.o) : error LNK2019: unresolved external symbol __imp_BCryptGenRandom referenced in function _ZN9getrandom3imp15getrandom_inner17hb74b95fb2783e364E

it looks like, on windows, a dependency on bcrypt has to be included. Maybe something similar to corrosion-rs/corrosion#120, but in this case it's occuring on Rust 1.47.0.

@djmitche
Copy link
Collaborator Author

Woo, all green (except lint, somehow)

@djmitche djmitche changed the title [WIP] build a C interface to taskchampion build a C interface to taskchampion Feb 27, 2022
@dbr
Copy link
Collaborator

dbr commented Mar 3, 2022

Had a look over the code and all looked good, although it's a lot of changes so I ended up skimming a lot!

Two things, one trivial and one maybe less so:

  1. I tried running the tests with cargo-nextest out of curiosity and it fails to find the libtaskchampion.so which is interesting (which is potentially a bug with that project as it I don't think it should change how compilation works at all, but also quite likely the way we find the library here might also be a quirk of regaulr cargo-test rather than any "official" file layout). Not worth looking into for this PR
  2. Running the test executable under valgrind's memcheck claims some memory is leaked. I guess this could potentially an artifact of how the tests are run rather than a problem with the C API itself
$ $ env LD_LIBRARY_PATH=target/debug/deps/ valgrind --leak-check=full target/debug/deps/bindings-9af6c48072e7dff1
[...]
==503== 
==503== HEAP SUMMARY:
==503==     in use at exit: 297 bytes in 5 blocks
==503==   total heap usage: 7,883 allocs, 7,878 frees, 1,133,607 bytes allocated
==503== 
==503== 193 (64 direct, 129 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 5
==503==    at 0x483577F: malloc (vg_replace_malloc.c:299)
==503==    by 0x48EB45B: alloc::alloc::alloc (alloc.rs:86)
==503==    by 0x48EB4E6: alloc::alloc::Global::alloc_impl (alloc.rs:166)
==503==    by 0x48EB709: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc.rs:226)
==503==    by 0x48EB3BC: alloc::alloc::exchange_malloc (alloc.rs:316)
==503==    by 0x48EDB54: new<taskchampion::replica::TCReplica> (boxed.rs:191)
==503==    by 0x48EDB54: taskchampion::traits::PassByPointer::return_ptr (traits.rs:128)
==503==    by 0x48D547A: tc_replica_new_in_memory (replica.rs:134)
==503==    by 0x12CEB4: test_replica_get_task_not_found (replica.c:300)
==503==    by 0x12DF9E: UnityDefaultTestRun (unity.c:1846)
==503==    by 0x12D0C6: replica_tests (replica.c:324)
==503==    by 0x127329: integration_tests::bindings_tests::replica_tests (mod.rs:21)
==503==    by 0x126E0D: bindings::replica_tests (bindings.rs:21)
==503== 
==503== LEAK SUMMARY:
==503==    definitely lost: 64 bytes in 1 blocks
==503==    indirectly lost: 129 bytes in 2 blocks
==503==      possibly lost: 0 bytes in 0 blocks
==503==    still reachable: 104 bytes in 2 blocks
==503==         suppressed: 0 bytes in 0 blocks
==503== Reachable blocks (those to which a pointer was found) are not shown.
==503== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==503== 
==503== For counts of detected and suppressed errors, rerun with: -v
==503== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

@djmitche
Copy link
Collaborator Author

djmitche commented Mar 3, 2022

Oh, cool! I thought about trying to run things under valgrind but assumed it would be much more complicated than that. It looks like that is, indeed, a bug in the test (it doesn't call tc_replica_free) rather than an issue in the API.

Your first point is a good one, too -- in general, I don't know how best to go about getting all this linking stuff lined up, and I haven't found any good packages to copy from. So I'm hoping we can land it and find out by people trying to link to it :)

Figuring out how to run the tests under valgrind in CI would be amazing, too -- at least worth an issue.

@djmitche djmitche merged commit a7f353b into GothenburgBitFactory:main Mar 4, 2022
@djmitche
Copy link
Collaborator Author

djmitche commented Mar 4, 2022

oops, i meant to squash that :/

@djmitche
Copy link
Collaborator Author

djmitche commented Mar 5, 2022

Filed #337 and #338 as followups.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create a C API for the taskchampion crate
3 participants