Track p2p data store files using Git LFS

The large binary objects in p2p/src/main/resources/ are updated on every
Bisq release with the latest network data to avoid the need for new Bisq
clients to download all of this information from the network, which
would easily overload seed nodes and generally bog down the client.

This approach works well enough for its purposes, but comes with the
siginficant downside of storing all of this binary data in Git history
forever. The current version of these binary objects total about 65M,
and they grow with every release. In aggregate, this has caused the
total size of the repository to grow to 360M, making it cumbersome to
clone over a low-bandwith connection, and slowing down various local Git
operations.

To avoid further exacerbating this problem, this commit sets these files
up to be tracked via Git LFS. There's nothing we can do about the 360M
of files that already exist in history, but we can ensure it doesn't
grow in this unchecked way going forward. For an understanding of how
Git LFS works, see the reference material at [1], and see also the
sample project and README at [2].

We are using GitHub's built-in LFS service here, and it's important to
understand that there are storage and bandwidth limits in place. We have
1G total storage and 1G per month of bandwidth on the free tier. If we
exceed this, we will need to purchase a "data pack" from GitHub, which
will get us to 50G storage and bandwith. These are reasonably priced and
not the end of the world if it becomes necessary. In an attempt to avoid
this, however, the Travis CI build configuration has been updated to
cache Git LFS files, such that they are not re-downloaded on every CI
build, as this would very quickly cause us to exceed the free tier
bandwith limit (see [3] and [4]). With that out of the way, the variable
determining whether we exceed the monthly limit is how many clones we
have every month. Only the latest version of LFS-tracked files are
downloaded on a given clone, and at the current 65M that means we have
about 15 clones per month. This is not very many, and will almost
certainly mean that we exceed the limit. Our options at that point are
to buy a data pack or to run our own LFS server. We would almost
certainly do the former to start.

Tracking these files via LFS means that developers will need to have Git
LFS installed in order to properly synchronize the files. If a developer
does not have LFS installed, cloning will complete successfully and the
build would complete successfully, but the app would fail when trying to
actually load the p2p data store files. For this reason, the build has
been updated to proactively check that the p2p data store files have
been properly synchronized via LFS, and if not, the build fails with a
helpful error message. The docs/build.md instructions have also been
updated accordingly.

It is important that we make this change now, not only to avoid growing
the repository in the way described above as we have been doing now for
many releases, but also because we are now considering adding yet more
binary objects to the repository, as proposed at
bisq-network/projects#25.

[1]: https://git-lfs.github.com
[2]: https://github.com/cbeams/lfs-test
[3]: https://docs-staging.travis-ci.com/user/customizing-the-build/#git-lfs
[4]: travis-ci/travis-ci#8787 (comment)

.gitattributes

@@ -1,10 +1,8 @@
 # Auto detect text files and normalize line endings to LF
 # This will handle all files NOT found below
 * text=auto
-
 # These text files should retain Windows line endings (CRLF)
 *.bat           text eol=crlf
-
 # These binary files should be left untouched
 # (binary is a macro for -text -diff)
 *.bmp           binary
@@ -14,3 +12,4 @@
 *.jpg           binary
 *.jpeg          binary
 *.png           binary
+p2p/src/main/resources/*BTC_MAINNET filter=lfs diff=lfs merge=lfs -text

.travis.yml

@@ -3,8 +3,18 @@ jdk:
     - openjdk10
     - openjdk12
 
+cache:
+    directories:
+        - .git/lfs
+git:
+    lfs_skip_smudge: true
+
+install:
+    - git lfs pull
+
 before_install:
     grep -v '^#' assets/src/main/resources/META-INF/services/bisq.asset.Asset | sort --check --dictionary-order --ignore-case
+
 notifications:
     slack:
         on_success: change

build.gradle

@@ -269,6 +269,17 @@ configure(project(':p2p')) {
         testAnnotationProcessor "org.projectlombok:lombok:$lombokVersion"
         testCompile("org.mockito:mockito-core:$mockitoVersion")
     }
+
+    processResources.doFirst {
+        // Sanity check that Git LFS-managed data store files have actually been sync'd.
+        // If they have not, e.g. because Git LFS is not installed, they will be text files
+        // containing a sha256 hash of the remote object, indicating we should stop the
+        // build and inform the user how to fix the problem.
+        if (file('src/main/resources/ProposalStore_BTC_MAINNET').text.contains("oid sha256:"))
+            throw new GradleException("p2p data store files have not been synchronized. " +
+                "To fix this, ensure you have Git LFS installed and run `git lfs pull`. " +
+                "See docs/build.md for more information.")
+    }
 }
 
 

docs/build.md

@@ -1,6 +1,14 @@
 # Building Bisq
 
 
+## Install Git LFS
+
+Bisq uses Git LFS to track certain large binary files. Follow the instructions at https://git-lfs.github.com to install it, then run the following to command to verify the installation:
+
+    $ git lfs version
+    git-lfs/2.10.0 (GitHub; darwin amd64; go 1.13.6)
+
+
 ## Clone
 
     git clone https://github.com/bisq-network/bisq
@@ -20,7 +28,7 @@ If on Windows run `gradlew.bat build` instead.
 
 Bisq executables are now available in the root project directory. Run Bisq Desktop as follows:
 
-Note: bisq runs fine on jdk10 and jdk11. jdk12 is currently not supported. 
+Note: bisq runs fine on jdk10 and jdk11. jdk12 is currently not supported.
 
     ./bisq-desktop