Merge pull request #490 from SeeSpotRun/develop

Develop

.travis.yml

@@ -4,8 +4,8 @@ install:
     - sudo apt-get update
     - sudo apt-get install scons python3-sphinx gettext python3-setuptools
     - sudo apt-get install libblkid-dev libelf-dev libglib2.0-dev libjson-glib-dev
-    - sudo apt-get install clang
-    - sudo easy_install3 $(cat test-requirements.txt)
+    - sudo apt-get install clang python3-pip python3-cffi libffi-dev
+    - sudo pip3 install -r test-requirements.txt
 
 compiler:
   - clang

CHANGELOG.md

@@ -8,29 +8,38 @@ The format follows [keepachangelog.com]. Please stick to it.
 
 ### Fixed
 
-* Json encoding issues
+* Fix json UTF-encoding issues (see issues #463 and #464)
+* Fix ``rmlint --gui`` ignoring manual tagging of originals (see issue #385)
+* Fix various bugs in ``--replay`` mode (see issue #383)
+* Fix bug if ``--treemerge`` used in combination with ``--algorithm=SHA1``
+* Fix broken ``rmlint.py`` script after adding of part_of_directory types
+* Don't disable fiemap when gettext is disabled
+* Various fixes aimed at improving mac osx compatibility
 * Avoid generating ``rmlint.sh`` or other output files for ``rmlint --dedupe`` or ``rmlint --is-reflink`
-* Error message when trying to clone from original to its hardlink
-* Fix bug with ``--unmatched-basename`` option in ``--replay`` mode
-* Possible bug with ``--equal``
+* Fix possible compile problems scons' default CC flags are non-functional
 
 ### Added
 
 * Implement --hash-uniques option to generate full checksums of unique files too.
 * Implement --hash-unmatched option, similar to --hash-uniques but only for size twins.
 * Implement --rank-by f option to rank originals by directory full path
-* Can now atomically clone from original to its hardlink via ``rmlint --dedupe`` 
+* Can now atomically clone from original to its hardlink via ``rmlint --dedupe``
+* Option ``-c json:traversed`` to include list of fully-traversed dirs in json output
 
 ### Changed
 
-* Option ``--dedupe`` option parsing changed.
+* Option ``--dedupe`` option parsing changed.  See ``rmlint --dedupe -h`` for more.
   Previously:
   ``rmlint --dedupe [--dedupe-xattr] [--dedupe-readonly] [-v] [-V] <src> <dest>``
   Now:
   ``rmlint --dedupe [--xattr] [--readonly] [--followlinks] [--inline-extents] [-v] [-V] <src> <dest>``
 * Made json-glib a hard dependency
+* Improved progress reporting ETA algorithm when large number of small files remaining to scan
+* Small files (<4k) are now generally ignored for reflinking / cloning options since They
+  are typically stored as inline extents
+* docker: update to a more recent alpine and install full test dependencies
 
-### Deprecated
+### Removed
 * ``--write-unfinished``.  Use ``--hash-unmatched``.
 * ``--dedupe-xattr``.  Use ``--dedupe --xattr``
 * ``--dedupe-readonly``.  Use ``--dedupe --readonly``
@@ -367,7 +376,7 @@ Nothing was removed.
 - Progressbar uses timeout-based redraws which leads to much smoother drawing
   and less cpu footprint.
 - ``pretty`` formatter (default) produces now valid escaped commands.
-  It is still intented for visual output only. That's why a note for this was
+  It is still intended for visual output only. That's why a note for this was
   added.
 
 ### Added

README.rst

@@ -12,7 +12,7 @@ to remove it.
    :target: http://rmlint.rtfd.org
 
 .. image:: https://img.shields.io/travis/sahib/rmlint/develop.svg?style=flat
-   :target: https://travis-ci.org/sahib/rmlint
+   :target: https://travis-ci.com/sahib/rmlint
 
 .. image:: https://img.shields.io/github/issues/sahib/rmlint.svg?style=flat
    :target: https://github.com/sahib/rmlint/issues
@@ -95,8 +95,8 @@ AUTHORS
 Here's a list of developers to blame:
 
 ===================================  ============================= ===========================================
-*Christopher Pahl*                   https://github.com/sahib      2010-2017
-*Daniel Thomas*                      https://github.com/SeeSpotRun 2014-2017
+*Christopher Pahl*                   https://github.com/sahib      2010-2021
+*Daniel Thomas*                      https://github.com/SeeSpotRun 2014-2021
 ===================================  ============================= ===========================================
 
 There are some other people that helped us of course.
@@ -110,3 +110,17 @@ LICENSE
 See the
 `COPYING <https://raw.githubusercontent.com/sahib/rmlint/master/COPYING>`_ 
 file distributed along the source for details.
+
+DONATIONS
+---------
+
+If you think `rmlint` saved you some serious time [*]_ and/or space, you might
+consider a donation. 
+You can donate either via Flattr, PayPal or you buy us a
+beer if we ever meet. `See here for details <http://rmlint.readthedocs.org/en/latest/index.html#donations>`_. 
+
+.. [*] If it freed you from your beloved data: *Sorry.* [*]_
+.. [*] Please file a bug or read the source and provide a patch. [*]_
+.. [*] For more than 100GB of data loss we owe you one beer. [*]_
+.. [*] If you don't like beer or there's only Budweiser available, you can order
+   a Club Mate.

SConstruct

@@ -175,7 +175,7 @@ def check_gettext(context):
 def check_fiemap(context):
     rc = 1
 
-    if GetOption('with_gettext') is False:
+    if GetOption('with_fiemap') is False:
         rc = 0
 
     if rc and tests.CheckType(context, 'struct fiemap', header='#include <linux/fiemap.h>\n'):
@@ -546,7 +546,7 @@ if ARGUMENTS.get('VERBOSE') == "1":
     del options['CCCOMSTR']
     del options['LINKCOMSTR']
 
-# Actually instance the Environement with all collected information:
+# Actually instance the Environment with all collected information:
 env = Environment(**options)
 Export('env')
 
@@ -581,6 +581,34 @@ conf = Configure(env, custom_tests={
     'check_sysmacro_h': check_sysmacro_h
 })
 
+#######################################################################
+#                      Compiler Checks and Flags                      #
+#######################################################################
+
+if 'CC' in os.environ:
+    conf.env.Replace(CC=os.environ['CC'])
+    print(">> Using compiler: " + os.environ['CC'])
+
+if 'CFLAGS' in os.environ:
+    conf.env.Append(CCFLAGS=os.environ['CFLAGS'])
+    print(">> Appending custom build flags : " + os.environ['CFLAGS'])
+
+if 'LDFLAGS' in os.environ:
+    conf.env.Append(LINKFLAGS=os.environ['LDFLAGS'])
+    print(">> Appending custom link flags : " + os.environ['LDFLAGS'])
+
+if 'AR' in os.environ:
+    conf.env.Replace(AR=os.environ['AR'])
+    print(">> Using ar: " + os.environ['AR'])
+
+if 'NM' in os.environ:
+    conf.env.Replace(NM=os.environ['NM'])
+    print(">> Using nm: " + os.environ['NM'])
+
+if 'RANLIB' in os.environ:
+    conf.env.Replace(RANLIB=os.environ['RANLIB'])
+    print(">> Using ranlib: " + os.environ['RANLIB'])
+
 if not conf.CheckCC():
     print('Error: Your compiler and/or environment is not correctly configured.')
     Exit(1)
@@ -608,34 +636,6 @@ if conf.env['HAVE_BLKID']:
 if conf.env['HAVE_GIO_UNIX']:
     packages.append('gio-unix-2.0')
 
-###########################################################################
-#                           Compiler Flags                                #
-###########################################################################
-
-if 'CC' in os.environ:
-    conf.env.Replace(CC=os.environ['CC'])
-    print(">> Using compiler: " + os.environ['CC'])
-
-if 'CFLAGS' in os.environ:
-    conf.env.Append(CCFLAGS=os.environ['CFLAGS'])
-    print(">> Appending custom build flags : " + os.environ['CFLAGS'])
-
-if 'LDFLAGS' in os.environ:
-    conf.env.Append(LINKFLAGS=os.environ['LDFLAGS'])
-    print(">> Appending custom link flags : " + os.environ['LDFLAGS'])
-
-if 'AR' in os.environ:
-    conf.env.Replace(AR=os.environ['AR'])
-    print(">> Using ar: " + os.environ['AR'])
-
-if 'NM' in os.environ:
-    conf.env.Replace(NM=os.environ['NM'])
-    print(">> Using nm: " + os.environ['NM'])
-
-if 'RANLIB' in os.environ:
-    conf.env.Replace(RANLIB=os.environ['RANLIB'])
-    print(">> Using ranlib: " + os.environ['RANLIB'])
-
 # Support museums or other debian flavours:
 conf.check_c11()
 if conf.env['HAVE_C11']:
@@ -921,9 +921,9 @@ Type 'scons' to actually compile rmlint now. Good luck.
             compiler=env['CC'],
             prefix=GetOption('prefix'),
             actual_prefix=GetOption('actual_prefix') or GetOption('prefix'),
-            verbose=yesno(ARGUMENTS.get('VERBOSE')),
-            debug=yesno(ARGUMENTS.get('DEBUG')),
-            symbols=yesno(ARGUMENTS.get('SYMBOLS')),
+            verbose=yesno(ARGUMENTS.get('VERBOSE') == '1'),
+            debug=yesno(ARGUMENTS.get('DEBUG') == '1'),
+            symbols=yesno(ARGUMENTS.get('SYMBOLS') == '1'),
             version='{a}.{b}.{c} "{n}" (rev {r})'.format(
                 a=VERSION_MAJOR, b=VERSION_MINOR, c=VERSION_PATCH,
                 n=VERSION_NAME, r=env.get('gitrev', 'unknown')

docs/conf.py

@@ -137,7 +137,7 @@
         # an arbitrary url.
         'navbar_links': [
             ("GitHub", "https://github.com/sahib/rmlint", True),
-            ("Travis", "https://travis-ci.org/sahib/rmlint", True)
+            ("Travis", "https://travis-ci.com/sahib/rmlint", True)
         ],
 
         # Render the next and previous page links in navbar. (Default: true)

docs/developers.rst

@@ -63,7 +63,7 @@ Here are some other things to check before submitting your contribution:
   examples how they should **not** look like.
 - Is ``rmlint`` running okay inside of ``valgrind`` (i.e. no leaks and no memory violations)?
 
-.. _`TravisCI`: https://travis-ci.org/sahib/rmlint
+.. _`TravisCI`: https://travis-ci.com/sahib/rmlint
 
 .. _`test documentation`: https://github.com/sahib/rmlint/blob/develop/tests/README.rst
 

docs/index.rst

@@ -50,7 +50,7 @@ to remove it. It is able to find:
    :width: 8%
 
 .. image:: https://img.shields.io/travis/sahib/rmlint/develop.svg?style=flat
-   :target: https://travis-ci.org/sahib/rmlint
+   :target: https://travis-ci.com/sahib/rmlint
    :width: 8%
 
 .. image:: https://img.shields.io/github/issues/sahib/rmlint.svg?style=flat
@@ -153,3 +153,36 @@ License
 .. _GPLv3: http://www.gnu.org/copyleft/gpl.htm
 .. _sahib: https://github.com/sahib
 .. _SeeSpotRun: https://github.com/SeeSpotRun
+
+Donations
+---------
+
+If you think rmlint saved [*]_ you some serious time and/or space, you might
+consider a donation. You can donate either via *Flattr* or via *PayPal*:
+
+.. image:: http://api.flattr.com/button/flattr-badge-large.png
+   :target: https://flattr.com/@SeeSpotRun
+   :align: center
+   :width: 9%
+
+.. raw:: html
+
+   <br />
+   <center>
+        <form action="https://www.paypal.com/donate" method="post" target="_top">
+            <input type="hidden" name="business" value="ZZFE8WTYKT2KW" />
+            <input type="hidden" name="item_name" value="Save the world one duplicate at a time" />
+            <input type="hidden" name="currency_code" value="AUD" />
+            <input type="image" src="https://www.paypalobjects.com/en_AU/i/btn/btn_donate_SM.gif" border="0" name="submit" title="PayPal - The safer, easier way to pay online!" alt="Donate with PayPal button" />
+            <img alt="" border="0" src="https://www.paypal.com/en_AU/i/scr/pixel.gif" width="1" height="1" />
+        </form>
+   </center>
+   <br />
+
+Or just buy us a beer if we ever meet. Nice emails are okay too.
+
+.. [*] If it freed you from your beloved data: *Sorry.* [*]_
+.. [*] Please file a bug or read the source and provide a patch. [*]_
+.. [*] For more than 100GB of data loss we owe you one beer. [*]_
+.. [*] If you don't like beer or there's only Budweiser available, you can order
+   a Club Mate.

docs/install.rst

@@ -134,7 +134,7 @@ Here's a list of readily prepared commands for known operating systems:
 
   .. code-block:: bash
 
-    $ pkg install git scons py27-sphinx pkgconf
+    $ pkg install git scons-py37 py37-sphinx pkgconf
     $ pkg install glib gettext libelf json-glib
 
 -----

docs/rmlint.1.rst

@@ -122,7 +122,7 @@ General Options
     Examples::
 
     $ rmlint -o json                 # Stream the json output to stdout
-    $ rmlint -O csv:/tmp/rmlint.csv  # Output an extra csv fle to /tmp
+    $ rmlint -O csv:/tmp/rmlint.csv  # Output an extra csv file to /tmp
 
 :``-c --config=spec[=value]`` (**default\:** *none*):
 
@@ -1017,7 +1017,11 @@ PROGRAM AUTHORS
 
 ``rmlint`` was written by:
 
-* Christopher <sahib> Pahl 2010-2017 (https://github.com/sahib)
-* Daniel <SeeSpotRun> T.   2014-2017 (https://github.com/SeeSpotRun)
+* Christopher <sahib> Pahl 2010-2021 (https://github.com/sahib)
+* Daniel <SeeSpotRun> T.   2014-2021 (https://github.com/SeeSpotRun)
 
 Also see the  http://rmlint.rtfd.org for other people that helped us.
+
+If you consider a donation you can use *Flattr* or buy us a beer if we meet:
+
+https://flattr.com/@SeeSpotRun

docs/testing.rst

@@ -17,7 +17,7 @@ To ensure required test dependencies are present:
 
 On every commit, those tests are additionally run on `TravisCI`_.
 
-.. _`TravisCI`: https://travis-ci.org/sahib/rmlint
+.. _`TravisCI`: https://travis-ci.com/sahib/rmlint
 
 Control Variables
 ~~~~~~~~~~~~~~~~~

docs/tutorial.rst

@@ -574,7 +574,7 @@ as possible! Good practice includes adding a ``$`` anchor at the end of the rege
   .. code-block:: bash
 
     # Sort paths with ABC in them first, then DEF, then GHI.
-    # Everthing with »temp«, »tmp« or »cache« in it comes last,
+    # Everything with »temp«, »tmp« or »cache« in it comes last,
     # the rest is sandwhiched in between and sorted by their modification time (m).
     # If something is tied, the modification time is also used a sorting criteria.
     $ rmlint -S 'r<.*ABC.*>r<.*DEF.*>r<.*GHI.*>R<.*(tmp|temp|cache).*>m' /tmp/t
@@ -738,7 +738,7 @@ Replaying results
 Often it is useful to just re-output the results you got from ``rmlint``.
 That's kind of annoying for large datasets, especially when you have big files.
 For this, ``rmlint`` features a special mode, where it re-outputs the result of
-previous runs. By default, ``rmlint`` will spit out a ``.json`` file (ususally
+previous runs. By default, ``rmlint`` will spit out a ``.json`` file (usually
 called ``rmlint.json``). When ``--replay`` is given, you can pass one or more
 of those ``.json`` files to the commandline as they would be normal
 directories. ``rmlint`` will then merge and re-output then. Note however, that

gui/shredder/application.py

@@ -86,7 +86,7 @@ def do_activate(self, **kw):
     def do_startup(self, **kw):
         Gtk.Application.do_startup(self, **kw)
 
-        # Make tranlsating strings possible:
+        # Make translating strings possible:
         # (We use the same message catalouge as rmlint)
         gettext.install('rmlint')
 

gui/shredder/chart.py

@@ -415,7 +415,7 @@ def on_draw(self, area, ctx):
         # Figure out the background color of the drawing area
         alloc = area.get_allocation()
 
-        # Caluclate the font size of the inner label.
+        # Calculate the font size of the inner label.
         # Make it smaller if not enough place but cut off at a size of 12
         inner_circle = (1.4 / max_layers)
         inner_circle *= min(alloc.width, alloc.height) / 2

gui/shredder/query.py

@@ -192,7 +192,7 @@ def parse_count(value):
 
 def parse(query):
     """Actual lowlevel parsing function.
-    Extracts arbitary text and attr-value pairs.
+    Extracts arbitrary text and attr-value pairs.
     """
     attrs = ATTR_PATTERN.finditer(query)
     results = defaultdict(list)

gui/shredder/runner.py

@@ -256,7 +256,7 @@ def __init__(self, settings, untagged_paths, tagged_paths):
 
     def on_process_termination(self, process, result):
         """Called once GSuprocess sees its child die."""
-        # We dont emit process-finished yet here.
+        # We don't emit process-finished yet here.
         # We still might get some items from the stream.
         # Call process-finished once we hit EOF.