trouble building for 0.4.21

[MichaelMure]

New attempt to build this, but I still have troubles:

$ make go.mod IPFS_VERSION=v0.4.21
./set-target.sh v0.4.21
error writing go.mod: open /home/michael/go/pkg/mod/github.com/ipfs/[email protected]/go.mod883788688.tmp: permission denied
go mod edit: no flags specified (see 'go help mod edit').
make: *** [Makefile:13: go.mod] Error 1

Basically what's happening is that /home/michael/go/pkg/mod/**/**/* is read only on my system (debian), for a good reason IMHO. This is not something I have done specifically, it's how go modules behave normally.

I tried to remove the set-target script from the makefile and update go.mod myself:

$ make build
go build -buildmode=plugin -i -o "s3plugin.so" "plugin/main.go"
go build encoding: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/encoding.a: permission denied
go build unicode/utf16: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/unicode/utf16.a: permission denied
go build container/list: mkdir /usr/lib/go-1.12/pkg/linux_amd64_dynlink/container/: permission denied
go build crypto/internal/subtle: mkdir /usr/lib/go-1.12/pkg/linux_amd64_dynlink/crypto: permission denied
go build crypto/subtle: mkdir /usr/lib/go-1.12/pkg/linux_amd64_dynlink/crypto/: permission denied
go build internal/x/crypto/cryptobyte/asn1: mkdir /usr/lib/go-1.12/pkg/linux_amd64_dynlink/internal/x: permission denied
go build internal/nettrace: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/internal/nettrace.a: permission denied
go build internal/x/net/dns/dnsmessage: mkdir /usr/lib/go-1.12/pkg/linux_amd64_dynlink/internal/x: permission denied
go build internal/testlog: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/internal/testlog.a: permission denied
go build internal/x/crypto/curve25519: mkdir /usr/lib/go-1.12/pkg/linux_amd64_dynlink/internal/x: permission denied
go build io: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/io.a: permission denied
go build math/rand: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/math/rand.a: permission denied
go build sort: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/sort.a: permission denied
go build internal/singleflight: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/internal/singleflight.a: permission denied
go build internal/syscall/unix: mkdir /usr/lib/go-1.12/pkg/linux_amd64_dynlink/internal/syscall/: permission denied
go build time: open /usr/lib/go-1.12/pkg/linux_amd64_dynlink/time.a: permission denied
make: *** [Makefile:13: s3plugin.so] Error 1

It seems to be a problem with the -i flag of go build -buildmode=plugin -i -o "s3plugin.so" "plugin/main.go", in a similar fashion as golang/go#27285. Again, this seems to attempt to write the files shipped by go, which is a bad idea.

Removing the -i flag, the build finally succeed:

$ make install
go build -buildmode=plugin -o "s3plugin.so" "plugin/main.go"
chmod +x "s3plugin.so"
Built against v0.4.21
install -Dm700 s3plugin.so "/home/michael/.ipfs/plugins/go-ds-s3.so"

Alas, loading the plugin fail:

$ ipfs version
Error: error loading plugins: loading plugin /home/michael/.ipfs/plugins/go-ds-s3.so: plugin.Open("/home/michael/.ipfs/plugins/go-ds-s3"): plugin was built with a different version of package github.com/ipfs/go-ipfs/filestore/pb

Note that I compiled myself the go-ipfs binay, same machine, same environment, exactly from the v0.4.21 tag.

Any help ?

[MichaelMure]

I checked on my machine with go list -m -json all for each, the dependencies for go-ds-s3 are a strict superset of go-ipfs's ones (with most of them marked as indirect, obviously), same exact version and timestamps.

[Stebalien]

What go version are you using? go 1.11 has a bunch of issues with go mod.

Basically what's happening is that /home/michael/go/pkg/mod///* is read only on my system (debian), for a good reason IMHO. This is not something I have done specifically, it's how go modules behave normally.

That's odd. It should be editing the go-ds-s3's go.mod file, not go-ipfs's.

It seems to be a problem with the -i flag of go build -buildmode=plugin -i -o "s3plugin.so" "plugin/main.go", in a similar fashion as golang/go#27285. Again, this seems to attempt to write the files shipped by go, which is a bad idea.

That should have been fixed in go 1.12.

Note that I compiled myself the go-ipfs binay, same machine, same environment, exactly from the v0.4.21 tag.

It looks like make is re-generating github.com/ipfs/go-ipfs/filestore/pb/dataobj.pb.go every build. We need to fix that.

edit: hm, not every build

[MichaelMure]

go version go1.12.4 linux/amd64

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/michael/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/michael/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go-1.12"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.12/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/michael/dev/go-ipfs/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build099636867=/tmp/go-build -gno-record-gcc-switches"

[MichaelMure]

One thing I can think of is that when compiling go-ipfs with go modules (again, on the v0.4.21 tag), go.mod is changed that way:

diff --git a/go.mod b/go.mod
index 9c94e261f..570d0ec8f 100644
--- a/go.mod
+++ b/go.mod
@@ -122,6 +122,7 @@ require (
        go.uber.org/multierr v1.1.0 // indirect
        go4.org v0.0.0-20190313082347-94abd6928b1d // indirect
        golang.org/x/sys v0.0.0-20190522044717-8097e1b27ff5
+       google.golang.org/appengine v1.4.0 // indirect
        gopkg.in/cheggaaa/pb.v1 v1.0.28
        gotest.tools/gotestsum v0.3.4
 )

It's weird, but I don't see why it would make the plugin fail to load.

[Stebalien]

One thing I can think of is that when compiling go-ipfs with go modules

That's an annoying go modules bug. go mod tidy gets rid of that line.

Now I'm stumped.

[MichaelMure]

Just to be sure, I redid the whole process on another machine (but still a debian with the same go version), same thing on every point.

[MichaelMure]

Alternatively, if you can make it works, could you provide a working binary so I can at least evaluate this plugin ?

[Stebalien]

Yeah, I'm getting version issues too (goprocess for me).

To get this working, either:

1. Install IPFS with go get github.com/ipfs/go-ipfs/cmd/[email protected] and then build this plugin with make build IPFS_VERSION=v0.4.21.
2. Install IPFS from source and then build this plugin with make build IPFS_VERSION=/path/to/your/ipfs/source.

We should probably make the dist build scripts use go get for consistency.

[Stebalien]

Ah, the issue is the build environment. We need to trim the paths here.

[MichaelMure]

1. Install IPFS with `go get github.com/ipfs/go-ipfs/cmd/[email protected]` and then build this plugin with `make build IPFS_VERSION=v0.4.21`.

This one works ! I can at least get some errors specific to the plugin, so it's actually loaded and running !

2. Install IPFS from source and then build this plugin with `make build IPFS_VERSION=/path/to/your/ipfs/source`.

Error: error loading plugins: loading plugin /home/michael/.ipfs/plugins/go-ds-s3.so: plugin.Open("/home/michael/.ipfs/plugins/go-ds-s3"): plugin was built with a different version of package github.com/ipfs/go-ipfs/filestore/pb after removing the -i for the same reason as before.

[Stebalien]

Hm. I can now reproduce the -i issue... the set-target issue.

[Stebalien]

Ok, so I've now reproduced every single build issue. Here's a partial fix: #18

Unfortunately, I'm now seeing the filestore protobuf issue.

[Stebalien]

I'm guessing the filestore/pb issue is due to the build paths:

When building with go mod, the path is pkg/mod/.... When building with "dists", the path is src/github.com/....

[MichaelMure]

This one works ! I can at least get some errors specific to the plugin, so it's actually loaded and running !

So, I deleted my ~/.ipfs/ to have a clean repo to experiment, forgetting that it would delete the plugin as well. Now I can't make it load any more ...

[MichaelMure]

A short guide on how to use this plugin would be very welcome. I inferred what the config should looks like from the code but I end up with errors like Error: datastore configuration of '{"mounts":[{"mountpoint":"/blocks","path":"blocks","shardFunc":"/repo/flatfs/shard/v1/next-to-last/2","type":"flatfs"},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}' does not match what is on disk '{"mounts":[{"bucket":"arn:aws:s3:::blablabla","mountpoint":"/blocks","region":"us-east-1","rootDirectory":""},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}'

[Stebalien]

Unfortunately, we create a "lockfile" to prevent the datastore configuration from being modified after init (as some modifications will require migrations). The goal was to introduce an auto-migration feature we never finished that and it's not exactly a high priority.

In this case, you can replace the contents of the datastore_spec file with {"mounts":[{"bucket":"arn:aws:s3:::blablabla","mountpoint":"/blocks","region":"us-east-1","rootDirectory":""},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}.

---

We need to document this either way but I'd like to provide a nicer way to use this plugin first.

[MichaelMure]

I got it working, thanks :)

For the next guy, something like the following works:

{
  "Datastore": {

...

    "Spec": {
      "mounts": [
        {
          "child": {
            "type": "s3ds",
            "region": "us-east-1",
            "bucket": "blablabla", <-- no arn
            "accessKey": "",
            "secretKey": ""
          },
          "mountpoint": "/blocks",
          "prefix": "s3.datastore",
          "type": "measure"
        },

...

accessKey and secretKey are mandatory, but if not filled the plugin fallback to the ~/.aws/credential file.

[MichaelMure]

Another note on this: compiling as explained in #18 (comment) worked for me because my $GOPATH was empty. That is, is works if the path trimming is effectively disabled.

When I tried to compile with $GOPATH set, I couldn't load the plugin anymore, even after recompiling go-ipfs with ipfs/kubo#6412 cherry picked.

[MichaelMure]

For the next guy as well, the required permissions are:

• s3:PutObject
• s3:GetObject
• s3:ListBucket
• s3:DeleteObject

[ianopolous]

I'm also having similar but different build issues for v0.4.21. Is there a chance for PL to distribute go-ipfs builds with this plugin already included? I'm guessing this is going to be a very popular plugin for commercial users.

[Stebalien]

Could you try again with 0.4.22 and go 1.12.7? I think I've finally gotten the build working out of the box (at least I can get it to work in CI).

Unfortunately, it still requires the exact go version. Given that, I think we're going to need to distribute compiled plugins to keep users from pulling their hair out.

[ianopolous]

The latest builds fine out of the box (with go 1.12.6). Thank you!

[ianopolous]

Whilst it builds fine with go 1.12.6 it won't load in ipfs (0.4.22 downloaded from dists.ipfs.io):

Error: error loading plugins: loading plugin /root/.ipfs/plugins/go-ds-s3.so: plugin.Open("/root/.ipfs/plugins/go-ds-s3"): plugin was built with a different version of package internal/cpu

go 1.12.7 is not available on golang's download page, So I tried 1.12.8, but that gives the same error. :-(

[Stebalien]

Download link: https://dl.google.com/go/go1.12.7.linux-amd64.tar.gz

But yeah, that's why we need to pre-build.

[ianopolous]

@Stebalien Thanks for the link. That version loaded. Is there any concept of binary compatibility for plugins with go? i.e. If I want to use this plugin with a later version of ipfs, do I need to use the exact go version to build which was used on that ipfs version?

Now though, it fails with:

Error: datastore configuration of '{"mounts":[{"mountpoint":"/blocks","path":"blocks","shardFunc":"/repo/flatfs/shard/v1/next-to-last/2","type":"flatfs"},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}' does not match what is on disk '{"mounts":[{"bucket":"peergos-test","mountpoint":"/blocks","region":"us-east-1","rootDirectory":""},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}'

This is immediately after an ipfs init, and then changing the config as above.

[Stebalien]

If I want to use this plugin with a later version of ipfs, do I need to use the exact go version to build which was used on that ipfs version?

Unfortunately, yes. Specifically, the packages used need to be the same which means the standard library can't change.

---

This is immediately after an ipfs init, and then changing the config as above.

There's a second file, datastore_spec, that stores a serialized copy of the datastore config. The idea was to eventually write a migration tool that handled cases where the two disagreed and migrated data as necessary.

If you don't care about the repo, you can copy the config out of the repo, delete the repo, and then run ipfs init /path/to/my/config/file. That will re-create the repo with the new datastore specification.

[ianopolous]

For others (though I will document all this when done), you need to overwrite
$IPFS_DIR/datastore_spec with the expected value from the error, in my case:

{"mounts":[{"bucket":"peergos-test","mountpoint":"/blocks","region":"us-east-1","rootDirectory":""},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}

[Kubuxu]

There's a second file, datastore_spec, that stores a serialized copy of the datastore config. The idea was to eventually write a migration tool that handled cases where the two disagreed and migrated data as necessary.

I think @magik6k had such a tool.

[Stebalien]

https://github.com/ipfs/ipfs-ds-convert

But I don't think it supports anything but badger by default. It should probably use go-ipfs plugins.

[Stebalien]

Closing as building has now been fixed against go-ipfs v0.4.22.