diff --git a/DETAILS.md b/DETAILS.md index 8fee47d4..e58718f9 100644 --- a/DETAILS.md +++ b/DETAILS.md @@ -3,7 +3,7 @@ ## The premise -Conceptually, cimbar is built on top of `image hashing`: +Cimbar is a grid of colored tiles. Conceptually, it is built on the idea of `image hashing`: ![example image hash](https://github.com/sz3/cimbar-samples/blob/v0.5/docs/imagehash.png) @@ -84,7 +84,7 @@ These properties may appear to be magical as you consider them more, and they do 2. wirehair requires the file contents to be stored in RAM * this relates to the size limit! -This constraint is less of an obstacle than it may seem -- the fountain codes are essentially being used as a wire format, and the encoder and decoder could agree on a scheme to split up, and then reassemble, larger files. Cimbar does not yet implement this, however! +This constraint is less of an obstacle than it may seem -- the fountain codes are essentially being used as a wire format, and the encoder and decoder could agree on a scheme to split up, and then reassemble, larger files. Cimbar does not (yet?) implement this, however! ## Implementation: Decoder diff --git a/PERFORMANCE.md b/PERFORMANCE.md index e2a4d29b..42d2647d 100644 --- a/PERFORMANCE.md +++ b/PERFORMANCE.md @@ -9,22 +9,32 @@ * There are 4 or 8 possible colors, encoding an additional 2-3 bits per tile. * These 6-7 bits per tile work out to a maximum of 9300-10850 bytes per barcode, though in practice this number is reduced by error correction. * The default ecc setting is 30/155, which is how we go from 9300 -> 7500 bytes of real data for a 4-color cimbar image. - * Reed Solomon is not an ideal for this use case -- specifically, it corrects byte errors, and cimbar errors tend to involve 1-3 bits at a time. However, since Reed Solomon implementations are ubiquitous, I used it for this prototype. + * Reed Solomon is not perfect for this use case -- specifically, it corrects byte errors, and cimbar errors tend to involve 1-3 bits at a time. However, since Reed Solomon implementations are ubiquitous, it is currently in use. ## Current sustained benchmark * 4-color cimbar with ecc=30: - * 2,980,556 bytes (after compression) in 36s -> 662 kilobits/s (~82 KB/s) + * 4,717,525 bytes (after compression) in 45s -> 838 kilobits/s (~104 KB/s) * 8-color cimbar with ecc=30: - * 2,980,556 bytes in 31s -> 769 kilobits/s (~96 KB/s) + * 4,717,525 bytes in 40s -> 943 kilobits/s (~118 KB/s) * details: - * these numbers are use https://github.com/sz3/cfc, running with 4 CPU threads on a Qualcomm Snapdragon 625 + * cimbar has built-in compression using zstd. What's being measured here is bits over the wire, e.g. data after compression is applied. + * these numbers are using https://github.com/sz3/cfc, running with 4 CPU threads on a Qualcomm Snapdragon 625 * perhaps I will buy a new cell phone to inflate the benchmark numbers. - * the sender commandline is `./cimbar_send /path/to/file -s` - * the `shakycam` option allows cfc to quickly discard ghosted frames, and spend more time decoding real data. + * the sender is the cimbar.org wasm implementation. An equivalent command line is `./cimbar_send /path/to/file -s` + * cimbar.org uses the `shakycam` option to allow the receiver to detect/discard "in between" frames as part of the scan step. This allows it to spend more processing time decoding real data. * burst rate can be higher (or lower) * to this end, lower ecc settings *can* provide better burst rates - * 8-color cimbar is considerably more sensitive to lighting conditions. Notably, decoding has some issues with dim screens. + * 4-color cimbar is currently preferred, and will give more consistent transfer speeds. + * 8-color cimbar should be considered a prototype within a prototype. It is considerably more sensitive to lighting conditions and color tints. +* other notes: + * having better lighting in the frame often leads to better results -- this is why cimbar.org has a (mostly) white background. cfc uses android's auto-exposure, auto-focus, etc (it's a very simple app). Good ambient light -- or a white background -- can lead to more consitent quality frame capture. + * because of the lighting/exposure question, I usually "shoot" in landscape instead of portrait. + * cfc currently has a low resolution, so the cimbar frame should take up as much of the display as possible (trust the guide brackets) + * similarly, it's best to keep the camera angle straight-on -- instead of at an angle -- to decode the whole image successfully. Decodes should still happen at higher angles, but the "smaller" part of the image may have more errors than the ECC can deal with. + * other things to be wary of: + * glare from light sources. + * shaky hands. diff --git a/README.md b/README.md index c7eb1074..fe072bad 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,9 @@ Behold: an experimental barcode format for air-gapped data transfer. -It can sustain speeds of 770+ kilobits/s (~96 KB/s) using nothing but a smartphone camera! +It can sustain speeds of 943+ kilobits/s (~118 KB/s) using just a computer monitor and a smartphone camera! + +![A non-animated cimbar code](https://github.com/sz3/cimbar-samples/blob/v0.5/6bit/4cecc30f.png) ## Explain? @@ -21,11 +23,13 @@ No internet/bluetooth/NFC/etc is used. All data is transmitted through the camer `cimbar` is a high-density 2D barcode format. Data is stored in a grid of colored tiles -- bits are encoded based on which tile is chosen, and which color is chosen to draw the tile. Reed Solomon error correction is applied on the data, to account for the lossy nature of the video -> digital decoding. Sub-1% error rates are expected, and corrected. -`libcimbar`, this optimized implementation, includes a simple protocol for file encoding based on fountain codes (`wirehair`). Files of up to 33MB can be encoded in a series of cimbar codes, which can be output as images or a live video feed. Once enough distinct image frames have been decoded successfully, the file will be reconstructed successfully. This is true even if the images are received out of order, or if some have been corrupted or are missing. +`libcimbar`, this optimized implementation, includes a simple protocol for file encoding built on fountain codes (`wirehair`) and zstd compression. Files of up to 33MB (after compression!) are encoded in a series of cimbar codes, which can be output as images or a live video feed. Once enough distinct image frames have been decoded successfully, the file will be reconstructed and decompressed successfully. This is true even if the images are received out of order, or if some have been corrupted or are missing. ## Platforms -The code is written in C++, and developed/tested on amd64+linux, arm64+android, and emscripten+wasm. It probably works, or can be made to work, on other platforms. +The code is written in C++, and developed/tested on amd64+linux, arm64+android (decoder only), and emscripten+WASM (encoder only). It probably works, or can be made to work, on other platforms. + +Crucially, because the encoder compiles to asmjs and wasm, it can run on anything with a modern web browser. There are [releases](https://github.com/sz3/libcimbar/releases/latest) if you wish to run the encoder locally instead of via cimbar.org. ## Library dependencies diff --git a/TODO.md b/TODO.md index 306a45d3..48fbd553 100644 --- a/TODO.md +++ b/TODO.md @@ -8,7 +8,7 @@ libcimbar is fairly optimized, to achieve the *proof* part of proof-of-concept. Performance optimizations aside, there are a number of paths that might be interesting to pursue. Some I may take a look at, but most I will leave to any enterprising developer who wants to take up the cause: * proper metadata/header information? - * would be nice to be able to determine ecc/#colors/#smybols from the cimbar image itself? + * would be nice to be able to determine ecc/#colors/#symbols from the cimbar image itself? * The bottom right corner is the obvious place to reclaim space to make this possible. * multi-frame decoding? * when decoding a static cimbar image, it would be useful to be able to use prior (unsuccessful) decode attempts to inform a future decode, and -- hopefully -- increase the probability of success. Currently, all frames are decoded independently. @@ -16,7 +16,7 @@ Performance optimizations aside, there are a number of paths that might be inter * optimal symbol set? * the 16-symbol (4 bit) set is hand-drawn. I stared with ~40 or so hand-drawn symbols, and used the 16 that performed best with each other. * there is surely a more optimal set -- a more rigorous approach should yield lower error rates! - * but, more importantly, it may be possible to go up to 32 symbols, and encode 5 bits per tile? + * but, more importantly, it may be possible to go up to 32 symbols, and encode 5 symbol bits per tile? * optimal symbol size? * the symbols that make up each cell on the cimbar grid are 8x8 (in a 9x9 grid). * this is because imagehash was on 8x8 tiles! @@ -25,16 +25,24 @@ Performance optimizations aside, there are a number of paths that might be inter * optimal color set? * the 4-color (2 bit) pallettes seem reasonable. 8-color, perhaps less so? * this may be a limitation of the algorithm/approach, however. Notably, since each symbol is drawn with one pallette color, all colors need sufficient contrast against the backdrop (#000 or #FFF, depending). This constrains the color space somewhat, and less distinct colors == more errors. + * in addition to contrast, there is interplay (that I don't currently understand) between the overall brightness of the image and the exposure time needed for high framerate capture. More clean frames == more troughput. * optimal grid size? * 1024x1024 is a remnant of the early prototyping process. There is nothing inherently special about it (except that it fits on a 1920x1080 screen, which seems good) * the tile grid itself is 1008x1008 (1008 == 9x112 -- there are 112 tile rows and columns) * a smaller grid would be less information dense, but more resilient to errors. Probably. * optimal grid shape? * it's a square because QR codes are square. That's it. Should it be? + * I'm strongly considering 4:3 for the next revision. * more efficient ECC? - * LDPC? - * Reed Solomon operates on bytes. Most decode errors tend to average out at 1-3 bits. It's not a total disaster, because it works. However, it would be nice to have denser error correction codes. + * QC-LDPC? + * Reed Solomon operates on bytes. Most decode errors tend to average out at 1-3 bits. (In the pathological case, a single read error will span two bytes.) It's not a total disaster -- it still works. + * I expect that state of the art ECC will allow 6-15% better throughput. + * it's a wide range due to various unknowns (unknowns to me, anyway) * proper GPU support (OpenCV + openCL) on android? + * It *might* be useful. [CFC]((https://github.com/sz3/cfc) is the current test bed for this. +* wasm decoder? + * probably needs to use Web Workers + * in-browser GPGPU support would be interesting (but I'm not counting on it) * ??? * still reading? Of course there's more! There's always more! diff --git a/src/exe/cimbar/cimbar.cpp b/src/exe/cimbar/cimbar.cpp index 1990a60e..8f8c91e1 100644 --- a/src/exe/cimbar/cimbar.cpp +++ b/src/exe/cimbar/cimbar.cpp @@ -74,7 +74,7 @@ int main(int argc, char** argv) unsigned ecc = cimbar::Config::ecc_bytes(); options.add_options() ("i,in", "Encoded pngs/jpgs/etc (for decode), or file to encode", cxxopts::value>()) - ("o,out", "Output file or directory.", cxxopts::value()) + ("o,out", "Output file prefix (encoding) or directory (decoding).", cxxopts::value()) ("c,color-bits", "Color bits. [0-3]", cxxopts::value()->default_value(turbo::str::str(colorBits))) ("e,ecc", "ECC level", cxxopts::value()->default_value(turbo::str::str(ecc))) ("f,fountain", "Attempt fountain encode/decode", cxxopts::value()) diff --git a/src/exe/cimbar_send/send.cpp b/src/exe/cimbar_send/send.cpp index f20aeb51..a65e12ae 100644 --- a/src/exe/cimbar_send/send.cpp +++ b/src/exe/cimbar_send/send.cpp @@ -55,18 +55,11 @@ int main(int argc, char** argv) fps = defaultFps; unsigned delay = 1000 / fps; - bool dark = true; bool use_rotatecam = result.count("rotatecam"); bool use_shakycam = result.count("shakycam"); + int window_size = 1080; - cimbar::shaky_cam cam(cimbar::Config::image_size(), 1080, 1080, dark); - // if we don't need the shakycam, we'll just turn it off - // we could use a separate code path (just do a mat copyTo), - // but this is fine. - if (!use_shakycam) - cam.toggle(); - - cimbar::window w(cam.width(), cam.height(), "cimbar_send"); + cimbar::window w(window_size, window_size, "cimbar_send"); if (!w.is_good()) { std::cerr << "failed to create window :(" << std::endl; @@ -76,21 +69,22 @@ int main(int argc, char** argv) bool running = true; bool start = true; - auto draw = [&w, &cam, use_rotatecam, delay, &running, &start] (const cv::Mat& frame, unsigned) { + auto draw = [&w, use_rotatecam, use_shakycam, delay, &running, &start] (const cv::Mat& frame, unsigned) { if (!start and w.should_close()) return running = false; start = false; - cv::Mat& windowImg = cam.draw(frame); - w.show(windowImg, delay); + w.show(frame, delay); if (use_rotatecam) w.rotate(); + if (use_shakycam) + w.shake(); return true; }; Encoder en(ecc, cimbar::Config::symbol_bits(), colorBits); while (running) for (const string& f : infiles) - en.encode_fountain(f, draw, compressionLevel); + en.encode_fountain(f, draw, compressionLevel, 8.0, window_size); return 0; } diff --git a/src/lib/cimb_translator/CimbWriter.cpp b/src/lib/cimb_translator/CimbWriter.cpp index c5f87a0b..bb021ef7 100644 --- a/src/lib/cimb_translator/CimbWriter.cpp +++ b/src/lib/cimb_translator/CimbWriter.cpp @@ -33,39 +33,45 @@ namespace { string name = dark? "guide-vertical-dark" : "guide-vertical-light"; return cimbar::load_img(fmt::format("bitmap/{}.png", name)); } - - void paste(cv::Mat& canvas, const cv::Mat& img, int x, int y) - { - img.copyTo(canvas(cv::Rect(x, y, img.cols, img.rows))); - } } -CimbWriter::CimbWriter(unsigned symbol_bits, unsigned color_bits, bool dark) +CimbWriter::CimbWriter(unsigned symbol_bits, unsigned color_bits, bool dark, int size) : _positions(Config::cell_spacing(), Config::num_cells(), Config::cell_size(), Config::corner_padding(), Config::interleave_blocks(), Config::interleave_partitions()) , _encoder(symbol_bits, color_bits) { - unsigned size = cimbar::Config::image_size(); + if (size > cimbar::Config::image_size()) + _offset = (size - cimbar::Config::image_size()) / 2; + else + size = cimbar::Config::image_size(); cv::Scalar bgcolor = dark? cv::Scalar(0, 0, 0) : cv::Scalar(0xFF, 0xFF, 0xFF); _image = cv::Mat(size, size, CV_8UC3, bgcolor); + // from here on, we only care about the internal size + size = cimbar::Config::image_size(); + cv::Mat anchor = getAnchor(dark); - paste(_image, anchor, 0, 0); - paste(_image, anchor, 0, size - anchor.cols); - paste(_image, anchor, size - anchor.rows, 0); + paste(anchor, 0, 0); + paste(anchor, 0, size - anchor.cols); + paste(anchor, size - anchor.rows, 0); cv::Mat secondaryAnchor = getSecondaryAnchor(dark); - paste(_image, secondaryAnchor, size - anchor.rows, size - anchor.cols); + paste(secondaryAnchor, size - anchor.rows, size - anchor.cols); cv::Mat hg = getHorizontalGuide(dark); - paste(_image, hg, (size/2) - (hg.cols/2), 2); - paste(_image, hg, (size/2) - (hg.cols/2), size-4); - paste(_image, hg, (size/2) - (hg.cols/2) - hg.cols, size-4); - paste(_image, hg, (size/2) - (hg.cols/2) + hg.cols, size-4); + paste(hg, (size/2) - (hg.cols/2), 2); + paste(hg, (size/2) - (hg.cols/2), size-4); + paste(hg, (size/2) - (hg.cols/2) - hg.cols, size-4); + paste(hg, (size/2) - (hg.cols/2) + hg.cols, size-4); cv::Mat vg = getVerticalGuide(dark); - paste(_image, vg, 2, (size/2) - (vg.rows/2)); - paste(_image, vg, size-4, (size/2) - (vg.rows/2)); + paste(vg, 2, (size/2) - (vg.rows/2)); + paste(vg, size-4, (size/2) - (vg.rows/2)); +} + +void CimbWriter::paste(const cv::Mat& img, int x, int y) +{ + img.copyTo(_image(cv::Rect(x+_offset, y+_offset, img.cols, img.rows))); } bool CimbWriter::write(unsigned bits) @@ -77,7 +83,7 @@ bool CimbWriter::write(unsigned bits) CellPositions::coordinate xy = _positions.next(); cv::Mat cell = _encoder.encode(bits); - paste(_image, cell, xy.first, xy.second); + paste(cell, xy.first, xy.second); return true; } diff --git a/src/lib/cimb_translator/CimbWriter.h b/src/lib/cimb_translator/CimbWriter.h index b76321f2..9af71d9c 100644 --- a/src/lib/cimb_translator/CimbWriter.h +++ b/src/lib/cimb_translator/CimbWriter.h @@ -7,15 +7,19 @@ class CimbWriter { public: - CimbWriter(unsigned symbol_bits, unsigned color_bits, bool dark=true); + CimbWriter(unsigned symbol_bits, unsigned color_bits, bool dark=true, int size=0); bool write(unsigned bits); bool done() const; cv::Mat image() const; +protected: + void paste(const cv::Mat& img, int x, int y); + protected: cv::Mat _image; CellPositions _positions; CimbEncoder _encoder; + unsigned _offset = 0; }; diff --git a/src/lib/cimb_translator/test/CimbWriterTest.cpp b/src/lib/cimb_translator/test/CimbWriterTest.cpp index 35d64b46..0e047d96 100644 --- a/src/lib/cimb_translator/test/CimbWriterTest.cpp +++ b/src/lib/cimb_translator/test/CimbWriterTest.cpp @@ -20,5 +20,23 @@ TEST_CASE( "CimbWriterTest/testSimple", "[unit]" ) } cv::Mat img = cw.image(); + assertEquals(1024, img.cols); + assertEquals(1024, img.rows); assertEquals( 0xeecc8800efce8c08, image_hash::average_hash(img) ); } + +TEST_CASE( "CimbWriterTest/testCustomSize", "[unit]" ) +{ + CimbWriter cw(4, 2, true, 1040); + + while (1) + { + if (!cw.write(0)) + break; + } + + cv::Mat img = cw.image(); + assertEquals(1040, img.cols); + assertEquals(1040, img.rows); + assertEquals( 0xab00ab02af0abfab, image_hash::average_hash(img) ); +} diff --git a/src/lib/encoder/Encoder.h b/src/lib/encoder/Encoder.h index a296d7f9..280d376e 100644 --- a/src/lib/encoder/Encoder.h +++ b/src/lib/encoder/Encoder.h @@ -15,8 +15,8 @@ class Encoder : public SimpleEncoder using SimpleEncoder::SimpleEncoder; unsigned encode(const std::string& filename, std::string output_prefix); - unsigned encode_fountain(const std::string& filename, std::string output_prefix, int compression_level=6, double redundancy=1.2); - unsigned encode_fountain(const std::string& filename, const std::function& on_frame, int compression_level=6, double redundancy=4.0); + unsigned encode_fountain(const std::string& filename, std::string output_prefix, int compression_level=6, double redundancy=1.2, int canvas_size=0); + unsigned encode_fountain(const std::string& filename, const std::function& on_frame, int compression_level=6, double redundancy=4.0, int canvas_size=0); }; inline unsigned Encoder::encode(const std::string& filename, std::string output_prefix) @@ -39,7 +39,7 @@ inline unsigned Encoder::encode(const std::string& filename, std::string output_ return i; } -inline unsigned Encoder::encode_fountain(const std::string& filename, const std::function& on_frame, int compression_level, double redundancy) +inline unsigned Encoder::encode_fountain(const std::string& filename, const std::function& on_frame, int compression_level, double redundancy, int canvas_size) { std::ifstream infile(filename); fountain_encoder_stream::ptr fes = create_fountain_encoder(infile, compression_level); @@ -56,7 +56,7 @@ inline unsigned Encoder::encode_fountain(const std::string& filename, const std: unsigned i = 0; while (i < requiredFrames) { - auto frame = encode_next(*fes); + auto frame = encode_next(*fes, canvas_size); if (!frame) break; @@ -67,7 +67,7 @@ inline unsigned Encoder::encode_fountain(const std::string& filename, const std: return i; } -inline unsigned Encoder::encode_fountain(const std::string& filename, std::string output_prefix, int compression_level, double redundancy) +inline unsigned Encoder::encode_fountain(const std::string& filename, std::string output_prefix, int compression_level, double redundancy, int canvas_size) { std::function fun = [output_prefix] (const cv::Mat& frame, unsigned i) { std::string output = fmt::format("{}_{}.png", output_prefix, i); @@ -75,5 +75,5 @@ inline unsigned Encoder::encode_fountain(const std::string& filename, std::strin cv::cvtColor(frame, bgr, cv::COLOR_RGB2BGR); return cv::imwrite(output, bgr); }; - return encode_fountain(filename, fun, compression_level, redundancy); + return encode_fountain(filename, fun, compression_level, redundancy, canvas_size); } diff --git a/src/lib/encoder/SimpleEncoder.h b/src/lib/encoder/SimpleEncoder.h index f3b2cb26..2139c8c8 100644 --- a/src/lib/encoder/SimpleEncoder.h +++ b/src/lib/encoder/SimpleEncoder.h @@ -19,7 +19,7 @@ class SimpleEncoder void set_encode_id(uint8_t encode_id); // [0-127] -- the high bit is ignored. template - std::optional encode_next(STREAM& stream); + std::optional encode_next(STREAM& stream, int canvas_size=0); template fountain_encoder_stream::ptr create_fountain_encoder(STREAM& stream, int compression_level=6); @@ -28,6 +28,7 @@ class SimpleEncoder unsigned _eccBytes; unsigned _bitsPerSymbol; unsigned _bitsPerColor; + bool _dark; uint8_t _encodeId = 0; }; @@ -35,6 +36,7 @@ inline SimpleEncoder::SimpleEncoder(int ecc_bytes, unsigned bits_per_symbol, int : _eccBytes(ecc_bytes >= 0? ecc_bytes : cimbar::Config::ecc_bytes()) , _bitsPerSymbol(bits_per_symbol? bits_per_symbol : cimbar::Config::symbol_bits()) , _bitsPerColor(bits_per_color >= 0? bits_per_color : cimbar::Config::color_bits()) + , _dark(cimbar::Config::dark()) { } @@ -56,13 +58,13 @@ inline void SimpleEncoder::set_encode_id(uint8_t encode_id) * */ template -inline std::optional SimpleEncoder::encode_next(STREAM& stream) +inline std::optional SimpleEncoder::encode_next(STREAM& stream, int canvas_size) { if (!stream.good()) return std::nullopt; unsigned bits_per_op = _bitsPerColor + _bitsPerSymbol; - CimbWriter writer(_bitsPerSymbol, _bitsPerColor); + CimbWriter writer(_bitsPerSymbol, _bitsPerColor, _dark, canvas_size); reed_solomon_stream rss(stream, _eccBytes); bitreader br; diff --git a/src/lib/encoder/test/EncoderTest.cpp b/src/lib/encoder/test/EncoderTest.cpp index 3f786af9..60c06609 100644 --- a/src/lib/encoder/test/EncoderTest.cpp +++ b/src/lib/encoder/test/EncoderTest.cpp @@ -33,7 +33,6 @@ TEST_CASE( "EncoderTest/testVanilla", "[unit]" ) { std::string path = fmt::format("{}_{}.png", outPrefix, i); cv::Mat img = cv::imread(path); - cv::cvtColor(img, img, cv::COLOR_BGR2RGB); assertEquals( hashes[i], image_hash::average_hash(img) ); } } @@ -56,7 +55,6 @@ TEST_CASE( "EncoderTest/testFountain", "[unit]" ) { std::string path = fmt::format("{}_{}.png", outPrefix, i); cv::Mat img = cv::imread(path); - cv::cvtColor(img, img, cv::COLOR_BGR2RGB); assertEquals( hashes[i], image_hash::average_hash(img) ); } } @@ -69,13 +67,12 @@ TEST_CASE( "EncoderTest/testFountain.Compress", "[unit]" ) std::string inputFile = TestCimbar::getProjectDir() + "/LICENSE"; std::string outPrefix = tempdir.path() / "encoder.fountain"; - Encoder enc(40, 4, 2); + Encoder enc(30, 4, 2); assertEquals( 1, enc.encode_fountain(inputFile, outPrefix) ); - uint64_t hash = 0xf8cde200e90582e4; + uint64_t hash = 0x4fd34f01ee80a28d; std::string path = fmt::format("{}_0.png", outPrefix); cv::Mat img = cv::imread(path); - cv::cvtColor(img, img, cv::COLOR_BGR2RGB); assertEquals( hash, image_hash::average_hash(img) ); } @@ -102,3 +99,21 @@ TEST_CASE( "EncoderTest/testPiecemealFountainEncoder", "[unit]" ) uint64_t hash = 0xf8cde200e90582e4; assertEquals( hash, image_hash::average_hash(*frame) ); } + +TEST_CASE( "EncoderTest/testFountain.Size", "[unit]" ) +{ + MakeTempDirectory tempdir; + + std::string inputFile = TestCimbar::getProjectDir() + "/LICENSE"; + std::string outPrefix = tempdir.path() / "encoder.fountain"; + + Encoder enc(30, 4, 2); + assertEquals( 1, enc.encode_fountain(inputFile, outPrefix, 10, 2.0, 1080) ); + + uint64_t hash = 0x8985b70d93675786; + std::string path = fmt::format("{}_0.png", outPrefix); + cv::Mat img = cv::imread(path); + assertEquals( 1080, img.rows ); + assertEquals( 1080, img.cols ); + assertEquals( hash, image_hash::average_hash(img) ); +} diff --git a/src/lib/gui/gl_2d_display.h b/src/lib/gui/gl_2d_display.h index 3a4cf5e5..becd9fc3 100644 --- a/src/lib/gui/gl_2d_display.h +++ b/src/lib/gui/gl_2d_display.h @@ -3,10 +3,10 @@ #include "gl_program.h" #include "gl_shader.h" +#include "util/loop_iterator.h" #include #include - #include namespace cimbar { @@ -23,9 +23,32 @@ class gl_2d_display -1.0f, 1.0f, 0.0f }; + // just using sin and cos is probably better? + static constexpr std::array, 4> ROTATIONS = {{ + {-1, 0, 0, 1}, + {1, 0, 0, -1}, // right 180 + {0, 1, 1, 0}, // right 90 + {0, -1, -1, 0} // right 270 + }}; + + static std::array, 4> computeShakePos(float dim) + { + float shake = 8.0f / dim; // 1080 + float zero = 0.0f; + return {{ + {zero, zero}, + {zero-shake, zero-shake}, + {zero, zero}, + {zero+shake, zero+shake} + }}; + } + public: - gl_2d_display() + gl_2d_display(unsigned width, unsigned height) : _p(create()) + , _shakePos(computeShakePos(std::min(width, height))) + , _shake(_shakePos) + , _rotation(ROTATIONS) { glGenBuffers(3, _vbo.data()); glGenVertexArrays(1, &_vao); @@ -60,8 +83,13 @@ class gl_2d_display // pass in rotation matrix GLuint rotateUniform = glGetUniformLocation(prog, "rot"); - std::array vals = rotation_matrix(); - glUniformMatrix2fv(rotateUniform, 1, false, vals.data()); + std::array rot = *_rotation; + glUniformMatrix2fv(rotateUniform, 1, false, rot.data()); + + // pass in transform vector + GLuint transformUniform = glGetUniformLocation(prog, "tform"); + std::pair tform = *_shake; + glUniform2f(transformUniform, tform.first, tform.second); // Draw glDrawArrays(GL_TRIANGLES, 0, 6); @@ -83,25 +111,18 @@ class gl_2d_display void rotate(unsigned i=1) { - if (i == 0 or ++_rotation >= 4) - _rotation = 0; + if (i == 0) + _rotation.reset(); + else + ++_rotation; } - std::array rotation_matrix() const + void shake(unsigned i=1) { - // just using sin and cos is probably better? - switch (_rotation) - { - default: - case 0: - return {-1, 0, 0, 1}; - case 1: // right 90 - return {0, 1, 1, 0}; - case 2: // right 180 - return {1, 0, 0, -1}; - case 3: // right 270 - return {0, -1, -1, 0}; - } + if (i == 0) + _shake.reset(); + else + ++_shake; } protected: @@ -116,6 +137,7 @@ class gl_2d_display */ static const std::string VERTEX_SHADER_SRC = R"(#version 300 es uniform mat2 rot; + uniform vec2 tform; in vec4 vert; out vec2 texCoord; void main() { @@ -123,6 +145,7 @@ class gl_2d_display vec2 ori = vec2(vert.x, vert.y); ori *= rot; texCoord = vec2(1.0f - ori.x, 1.0f - ori.y) / 2.0; + texCoord -= tform; })"; static const std::string FRAGMENT_SHADER_SRC = R"(#version 300 es @@ -144,7 +167,10 @@ class gl_2d_display std::array _vbo; GLuint _vao; unsigned _i = 0; - unsigned _rotation = 0; + + std::array, 4> _shakePos; + loop_iterator _shake; + loop_iterator _rotation; }; } diff --git a/src/lib/gui/shaky_cam.h b/src/lib/gui/shaky_cam.h index 61847234..69ab785c 100644 --- a/src/lib/gui/shaky_cam.h +++ b/src/lib/gui/shaky_cam.h @@ -9,14 +9,14 @@ namespace cimbar { class shaky_cam { public: - static constexpr std::array, 8> SHAKE_POS = {{ - {0, 0}, {-8, -8}, {0, 0}, {8, 8}, {0, 0}, {-8, 8}, {0, 0}, {8, -8} + static constexpr std::array, 4> SHAKE_POS = {{ + {0, 0}, {-8, -8}, {0, 0}, {8, 8} }}; public: shaky_cam(unsigned img_size, unsigned w, unsigned h, bool dark) - : _shakycam(true) - , _shakePos(SHAKE_POS) + : _shakycam(true) + , _shakePos(SHAKE_POS) { unsigned minFrameSize = img_size + 16; w = std::max(w, minFrameSize); @@ -45,6 +45,12 @@ class shaky_cam _shakePos.reset(); } + void shake() + { + if (_shakycam) + ++_shakePos; + } + cv::Mat& draw(const cv::Mat& img) { _frame = _bgcolor; @@ -59,10 +65,10 @@ class shaky_cam offsetX += (*_shakePos).first; offsetY += (*_shakePos).second; } - ++_shakePos; } img.copyTo(_frame(cv::Rect(offsetX, offsetY, img.cols, img.rows))); + cv::cvtColor(_frame, _frame, cv::COLOR_BGR2RGB); return _frame; } diff --git a/src/lib/gui/window_cvhighgui.h b/src/lib/gui/window_cvhighgui.h index e5be7e35..bf8db4a4 100644 --- a/src/lib/gui/window_cvhighgui.h +++ b/src/lib/gui/window_cvhighgui.h @@ -3,12 +3,16 @@ #include "window_interface.h" +#include "shaky_cam.h" + namespace cimbar { class window_cvhighgui : public window_interface { public: - window_cvhighgui(unsigned, unsigned, std::string) + window_cvhighgui(unsigned width, unsigned height, std::string title) + : _cam(std::min(width, height), width, height, true) + , _title(title) {} bool is_good() const @@ -18,20 +22,28 @@ class window_cvhighgui : public window_interface bool should_close() const { - return cv::getWindowProperty("image", cv::WND_PROP_AUTOSIZE) < 0; + return cv::getWindowProperty(_title, cv::WND_PROP_AUTOSIZE) < 0; } void rotate(unsigned i=1) { } + void shake(unsigned i=1) + { + if (i > 0) + _cam.shake(); + } + void show(const cv::Mat& img, unsigned delay) { - cv::imshow("image", img); + cv::imshow(_title, _cam.draw(img)); cv::waitKey(delay); // functions as the frame delay... you can hold down a key to make it go faster } protected: + cimbar::shaky_cam _cam; + std::string _title; }; } diff --git a/src/lib/gui/window_glfw.h b/src/lib/gui/window_glfw.h index d3cb358b..64c183ae 100644 --- a/src/lib/gui/window_glfw.h +++ b/src/lib/gui/window_glfw.h @@ -16,6 +16,7 @@ class window_glfw : public window_interface { public: window_glfw(unsigned width, unsigned height, std::string title) + : _width(width) { if (!glfwInit()) { @@ -30,8 +31,9 @@ class window_glfw : public window_interface return; } glfwMakeContextCurrent(_w); + glfwSwapInterval(1); - _display = std::make_shared(); + _display = std::make_shared(width, height); glGenTextures(1, &_texid); init_opengl(width, height); } @@ -61,6 +63,12 @@ class window_glfw : public window_interface _display->rotate(i); } + void shake(unsigned i=1) + { + if (_display) + _display->shake(i); + } + void clear() { if (_display) @@ -88,6 +96,11 @@ class window_glfw : public window_interface std::this_thread::sleep_for(std::chrono::milliseconds(delay-millis)); } + unsigned width() const + { + return _width; + } + protected: void init_opengl(int width, int height) { @@ -110,8 +123,8 @@ class window_glfw : public window_interface GLFWwindow* _w; GLuint _texid; std::shared_ptr _display; + unsigned _width; bool _good = true; }; } - diff --git a/src/wasm/cimbar_js/CMakeLists.txt b/src/wasm/cimbar_js/CMakeLists.txt index 42dead63..b1746cbf 100644 --- a/src/wasm/cimbar_js/CMakeLists.txt +++ b/src/wasm/cimbar_js/CMakeLists.txt @@ -47,7 +47,7 @@ set (LINK_WASM_LIST -s USE_GLFW=3 -s FILESYSTEM=0 -s TOTAL_MEMORY=134217728 - -s EXPORTED_FUNCTIONS='["_render","_initialize_GL","_encode","_configure"]' + -s EXPORTED_FUNCTIONS='["_render","_next_frame","_initialize_GL","_encode","_configure"]' ) string(REPLACE ";" " " LINK_WASM_FLAGS "${LINK_WASM_LIST}") diff --git a/src/wasm/cimbar_js/wasm.cpp b/src/wasm/cimbar_js/wasm.cpp index ce6a958b..2bd7b93c 100644 --- a/src/wasm/cimbar_js/wasm.cpp +++ b/src/wasm/cimbar_js/wasm.cpp @@ -15,7 +15,9 @@ namespace { std::shared_ptr _window; std::shared_ptr _fes; - int _renders = 0; + std::optional _next; + + int _numFrames = 0; uint8_t _encodeId = 0; // settings @@ -39,34 +41,43 @@ int initialize_GL(int width, int height) return 1; } +// render() and next_frame() could be put in the same function, +// but it seems cleaner to split them. +// in any case, we're concerned with frame pacing (some encodes take longer than others) int render() { if (!_window or !_fes) return 0; - // we generate 2x the amount of required blocks -- unless everything fits in a single frame. + if (_next) + { + _window->show(*_next, 0); + _window->shake(); + return 1; + } + return 0; +} + +int next_frame() +{ + if (!_window or !_fes) + return 0; + + // we generate 8x the amount of required blocks -- unless everything fits in a single frame. unsigned required = _fes->blocks_required(); if (required > cimbar::Config::fountain_chunks_per_frame()) - required = required*4; + required = required*8; if (_fes->block_count() > required) { _fes->reset(); - _window->rotate(0); + _window->shake(0); } SimpleEncoder enc(_ecc, cimbar::Config::symbol_bits(), _colorBits); enc.set_encode_id(_encodeId); - std::optional img = enc.encode_next(*_fes); - if (!img) - { - std::cerr << "no image :(" << std::endl; - return 0; - } - - _window->show(*img, 0); - _window->rotate(); - return ++_renders; + _next = enc.encode_next(*_fes, _window->width()); + return ++_numFrames; } int encode(uint8_t* buffer, size_t size) @@ -84,6 +95,8 @@ int encode(uint8_t* buffer, size_t size) if (!_fes) return 0; + + _next.reset(); return 1; } @@ -101,8 +114,9 @@ int configure(unsigned color_bits) // if the data is too small, we should throw out _fes -- and clear the canvas. _fes = nullptr; _window->clear(); + _next.reset(); } - _window->rotate(0); + _window->shake(0); } } return 0; diff --git a/web/index.html b/web/index.html index f7e14d16..98d11f18 100644 --- a/web/index.html +++ b/web/index.html @@ -14,7 +14,8 @@ } body { - background-color: black; + background-color: white; + background-image: radial-gradient(circle at top left, rgb(7,0,0),rgb(244,244,244),rgb(255,255,255)); color: gray; height: 100vh; display: grid; @@ -39,6 +40,8 @@ margin: 0 auto; z-index: -1; color: #F0F0F0; + outline: 6px solid black; + box-shadow: 0px 0px 12px black, 0px 0px 18px black; } #dragdrop::before { @@ -57,6 +60,9 @@ opacity: 0; pointer-events: auto; } +#invisible_click.active { + cursor: none; +} #nav-container:focus-within + #invisible_click { pointer-events: none; display: none; @@ -110,6 +116,7 @@ pointer-events: auto; touch-action: manipulation; outline: 0; + background-image: linear-gradient(180deg, rgb(0,0,0,0) 10%, rgb(0,0,0) 40%, rgb(0,0,0) 60%, rgb(0,0,0,0) 90%); } .icon-bar { display: block; @@ -265,6 +272,7 @@ No file selected - +