Commit Graph

37 Commits

Author SHA1 Message Date
Andrew Senetar e382683f66
Replace all relative imports 2022-05-09 01:40:08 -05:00
Andrew Senetar 63dd4d4561
Apply pyupgrade changes 2022-04-27 20:53:12 -05:00
Andrew Senetar 9f40e4e786
Squashed commit of the following:
commit 5eb515f666bfa1ff06c2e96bdc351a4b7456580e
Author: Andrew Senetar <arsenetar@gmail.com>
Date:   Sun Mar 27 22:19:39 2022 -0500

    Add fallback to md5 if xxhash not available

    Mainly here for the case when distributions have not packaged python3-xxhash.

commit 51b18d4c84
Author: Andrew Senetar <arsenetar@gmail.com>
Date:   Sat Mar 19 15:25:46 2022 -0500

    Switch file hashing to xxhash instead of md5

    - Improves performance significantly in some cases
    - Add xxhash to requirements.txt and sort requirements
    - Rename md5 based members to digest
    - Update all tests to use new member names and hashing methods
    - Update hash db code to upgrade schema

    NOTE: May consider supporting multiple hashing algorithms in the future.
2022-03-27 22:27:13 -05:00
Andrew Senetar e22d7d2fc9
Remove filtering of 0 size files in engine
Files size is already able to be filtered at a higher level, some users
may decide to see zero length files. Fix #321.
2021-08-28 18:16:22 -05:00
Andrew Senetar 0189c29f47
Misc cleanups in core/tests 2021-08-21 03:52:09 -05:00
Andrew Senetar 9446f37fad
Remove flake8 E731 Errors
Note: black formatting is now applying correctly as well.
2021-08-15 03:53:43 -05:00
glubsy 277bc3fbb8 Add unit tests for hash sample optimization
* Instead of keeping md5 samples separate, merge them as one hash computed from the various selected chunks we picked.
* We don't need to keep a boolean to see whether or not the user chose to optimize; we can simply compare the value of the threshold, since 0 means no optimization currently active.
2021-06-21 22:44:05 +02:00
glubsy c4dcfd3d4b Fix stripping (japanese) unicode characters
* Accents are getting removed from Unicode characters to generate similar "words".
* Non-latin characters which cannot be processed that way (eg. japanese, greek, russian, etc.) should not be filtered out at all otherwise files are erroneously skipped or detected as dupes if only some characters make it passed the filter.
* Starting from an arbitrary unicode codepoint (converted to decimal), above which we know it is pointless to try any sort of processing, we leave the characters as is.
* Fix #878.
2021-04-29 05:15:34 +02:00
Andrew Senetar de8a0a21b2
Update Packaging
- Add changes from OSX build to local hscommon/build.py
- Update package.py & srcpkg.sh
  - Remove invalid submodule references
  - Update srcpkg.sh to use xz
- Update package.py pyinstaller configuration
  - Call PyInstaller inline
  - Add --noconfirm option to be more script friendly
  - Add UCRT Redist location to path should fix #545 as now all the dlls
    are included
2019-12-31 21:36:52 -06:00
Andrew Senetar 7ba8aa3514
Format files with black
- Format all files with black
- Update tox.ini flake8 arguments to be compatible
- Add black to requirements-extra.txt
- Reduce ignored flake8 rules and fix a few violations
2019-12-31 20:16:27 -06:00
Virgil Dupras 9f2c3e7732 Fix failing test on OS X / py35
A 100 recursion limit was too low in that environment.
2016-07-01 15:29:50 -04:00
Virgil Dupras b1ef3dc8fe Simplify progress report during scanning
We now get less progress feedback, but in exchange, our progress job is
simpler. Previously, our progress bar would often get wonky towards the
end of the scan and I didn't have the energy to debug that.

Besides, people don't care about that level of progress feedback.
2016-06-08 12:29:28 -04:00
Virgil Dupras 130581db53 Apply flake8 checks to tests 2016-05-29 15:02:39 -04:00
Virgil Dupras e7076bc3bd Change license from BSD to GPLv3
See http://www.hardcoded.net/archive2014#2014-12-28 for context
2015-01-03 16:33:16 -05:00
Virgil Dupras fc16ea8c49 Change copyright year to 2015 2015-01-03 16:30:57 -05:00
Virgil Dupras ac32305532 Integrated the jobprogress library into hscommon
I have a fix to make in it and it's really silly to pretend that this
lib is of any use to anybody outside HS apps. Bringing it back here will
make things more simple.
2014-10-05 16:31:16 -04:00
Virgil Dupras ca709a60cf Updated copyright year to 2014 2014-04-19 12:19:11 -04:00
Virgil Dupras 4a8ce9b6c4 Updated copyright year to 2013. 2013-04-28 10:35:51 -04:00
Virgil Dupras 4f097a3a89 [#204 state:fixed] Added a message after re-prioritization telling how many dupe groups were changed by it. 2012-07-31 11:37:51 -04:00
Virgil Dupras 657f6743c2 Changed copyright year to 2012 2012-03-15 14:28:40 -04:00
Virgil Dupras 9fac97c147 Don't allow dupes from ref folders to step down from their ref position during reprioritization. 2011-09-23 13:14:57 -04:00
Virgil Dupras 518228a368 [#138] Added a very preliminary version of the prioritization dialog. A big part of this commit is about refactoring the tests and introducing a TestApp for dupeGuru (in core.tests.base). 2011-09-07 15:46:41 -04:00
Virgil Dupras 0fea59007c Updated copyright year to 2011. 2011-04-12 10:04:01 +02:00
Virgil Dupras eefe464fba Replaced dependencies from hsutil to hscommon. 2011-01-11 13:36:05 +01:00
Virgil Dupras 33c0ba808c Changed references to what has already been moved from hsutil to hscommon (io, path, testutil). 2011-01-11 11:59:53 +01:00
Virgil Dupras e0cc8ecda2 Stop using hsutil.testcase. 2011-01-05 11:11:21 +01:00
Virgil Dupras 0382ad1534 Adapted to the job-related code moving to the 'jobprogress' package. 2010-11-20 12:42:15 +01:00
Virgil Dupras 4886982d43 Re-licensed to BSD 2010-09-30 12:17:41 +02:00
Virgil Dupras 854d194f88 Converted to py3k. There's probably some bugs still. So far, I managed to run dupeGuru SE under pyobjc and qt. 2010-08-11 16:39:06 +02:00
Virgil Dupras 4d66b4667c Moved from nose to py.test (the former doesn't officially support py3k, which is limiting). 2010-07-13 11:10:45 +02:00
Virgil Dupras 371426a08e Adapted codebase to the hsutil/hscommon split and the hsmedia --> hsaudiotag rename. 2010-07-13 08:08:18 +02:00
Virgil Dupras 75eb005ba0 Fixed a flaky test which was broken in python 2.7rc1. 2010-06-07 10:15:58 -04:00
Virgil Dupras 601b67145c Fixed a flaky test which was broken in python 2.7rc1. 2010-06-07 09:41:59 -04:00
Virgil Dupras 47a6ceffbc Use lxml everywhere for xml save/load (instead of ElementTree and minidom). 2010-03-01 12:21:43 +01:00
Virgil Dupras d62ff40bed Removed svn keywords. 2010-01-02 16:52:18 +01:00
Virgil Dupras c14fecb415 Changed copyright year to 2010 2010-01-01 21:11:34 +01:00
hsoft 838f8ae352 Changed the build system (that commit is *huge*)
--HG--
rename : base/cocoa/AppDelegate.h => cocoa/base/AppDelegate.h
rename : base/cocoa/AppDelegate.m => cocoa/base/AppDelegate.m
rename : base/cocoa/Consts.h => cocoa/base/Consts.h
rename : base/cocoa/DetailsPanel.h => cocoa/base/DetailsPanel.h
rename : base/cocoa/DetailsPanel.m => cocoa/base/DetailsPanel.m
rename : base/cocoa/DirectoryPanel.h => cocoa/base/DirectoryPanel.h
rename : base/cocoa/DirectoryPanel.m => cocoa/base/DirectoryPanel.m
rename : base/cocoa/PyDupeGuru.h => cocoa/base/PyDupeGuru.h
rename : base/cocoa/ResultWindow.h => cocoa/base/ResultWindow.h
rename : base/cocoa/ResultWindow.m => cocoa/base/ResultWindow.m
rename : base/cocoa/dsa_pub.pem => cocoa/base/dsa_pub.pem
rename : base/cocoa/xib/DetailsPanel.xib => cocoa/base/xib/DetailsPanel.xib
rename : base/cocoa/xib/DirectoryPanel.xib => cocoa/base/xib/DirectoryPanel.xib
rename : base/cocoa/xib/MainMenu.xib => cocoa/base/xib/MainMenu.xib
rename : me/cocoa/AppDelegate.h => cocoa/me/AppDelegate.h
rename : me/cocoa/AppDelegate.m => cocoa/me/AppDelegate.m
rename : me/cocoa/Consts.h => cocoa/me/Consts.h
rename : me/cocoa/DetailsPanel.h => cocoa/me/DetailsPanel.h
rename : me/cocoa/DetailsPanel.m => cocoa/me/DetailsPanel.m
rename : me/cocoa/DirectoryPanel.h => cocoa/me/DirectoryPanel.h
rename : me/cocoa/DirectoryPanel.m => cocoa/me/DirectoryPanel.m
rename : me/cocoa/Info.plist => cocoa/me/Info.plist
rename : me/cocoa/PyDupeGuru.h => cocoa/me/PyDupeGuru.h
rename : me/cocoa/ResultWindow.h => cocoa/me/ResultWindow.h
rename : me/cocoa/ResultWindow.m => cocoa/me/ResultWindow.m
rename : me/cocoa/dupeguru.icns => cocoa/me/dupeguru.icns
rename : me/cocoa/dupeguru.xcodeproj/project.pbxproj => cocoa/me/dupeguru.xcodeproj/project.pbxproj
rename : me/cocoa/gen.py => cocoa/me/gen.py
rename : me/cocoa/main.m => cocoa/me/main.m
rename : me/cocoa/py/dg_cocoa.py => cocoa/me/py/dg_cocoa.py
rename : me/cocoa/py/setup.py => cocoa/me/py/setup.py
rename : me/cocoa/xib/Preferences.xib => cocoa/me/xib/Preferences.xib
rename : pe/cocoa/AppDelegate.h => cocoa/pe/AppDelegate.h
rename : pe/cocoa/AppDelegate.m => cocoa/pe/AppDelegate.m
rename : pe/cocoa/Consts.h => cocoa/pe/Consts.h
rename : pe/cocoa/DetailsPanel.h => cocoa/pe/DetailsPanel.h
rename : pe/cocoa/DetailsPanel.m => cocoa/pe/DetailsPanel.m
rename : pe/cocoa/DirectoryPanel.h => cocoa/pe/DirectoryPanel.h
rename : pe/cocoa/DirectoryPanel.m => cocoa/pe/DirectoryPanel.m
rename : pe/cocoa/Info.plist => cocoa/pe/Info.plist
rename : pe/cocoa/PictureBlocks.h => cocoa/pe/PictureBlocks.h
rename : pe/cocoa/PictureBlocks.m => cocoa/pe/PictureBlocks.m
rename : pe/cocoa/PyDupeGuru.h => cocoa/pe/PyDupeGuru.h
rename : pe/cocoa/ResultWindow.h => cocoa/pe/ResultWindow.h
rename : pe/cocoa/ResultWindow.m => cocoa/pe/ResultWindow.m
rename : pe/cocoa/dupeguru.icns => cocoa/pe/dupeguru.icns
rename : pe/cocoa/dupeguru.xcodeproj/project.pbxproj => cocoa/pe/dupeguru.xcodeproj/project.pbxproj
rename : pe/cocoa/gen.py => cocoa/pe/gen.py
rename : pe/cocoa/main.m => cocoa/pe/main.m
rename : pe/cocoa/py/dg_cocoa.py => cocoa/pe/py/dg_cocoa.py
rename : pe/cocoa/py/setup.py => cocoa/pe/py/setup.py
rename : pe/cocoa/xib/DetailsPanel.xib => cocoa/pe/xib/DetailsPanel.xib
rename : pe/cocoa/xib/Preferences.xib => cocoa/pe/xib/Preferences.xib
rename : se/cocoa/AppDelegate.h => cocoa/se/AppDelegate.h
rename : se/cocoa/AppDelegate.m => cocoa/se/AppDelegate.m
rename : se/cocoa/Consts.h => cocoa/se/Consts.h
rename : se/cocoa/DetailsPanel.h => cocoa/se/DetailsPanel.h
rename : se/cocoa/DetailsPanel.m => cocoa/se/DetailsPanel.m
rename : se/cocoa/DirectoryPanel.h => cocoa/se/DirectoryPanel.h
rename : se/cocoa/DirectoryPanel.m => cocoa/se/DirectoryPanel.m
rename : se/cocoa/Info.plist => cocoa/se/Info.plist
rename : se/cocoa/PyDupeGuru.h => cocoa/se/PyDupeGuru.h
rename : se/cocoa/ResultWindow.h => cocoa/se/ResultWindow.h
rename : se/cocoa/ResultWindow.m => cocoa/se/ResultWindow.m
rename : se/cocoa/dupeguru.icns => cocoa/se/dupeguru.icns
rename : se/cocoa/dupeguru.xcodeproj/project.pbxproj => cocoa/se/dupeguru.xcodeproj/project.pbxproj
rename : se/cocoa/gen.py => cocoa/se/gen.py
rename : se/cocoa/main.m => cocoa/se/main.m
rename : se/cocoa/py/dg_cocoa.py => cocoa/se/py/dg_cocoa.py
rename : se/cocoa/py/setup.py => cocoa/se/py/setup.py
rename : se/cocoa/xib/Preferences.xib => cocoa/se/xib/Preferences.xib
rename : base/core/LICENSE => core/LICENSE
rename : base/core/__init__.py => core/__init__.py
rename : base/core/app.py => core/app.py
rename : base/core/app_cocoa.py => core/app_cocoa.py
rename : base/core/data.py => core/data.py
rename : base/core/directories.py => core/directories.py
rename : base/core/engine.py => core/engine.py
rename : base/core/export.py => core/export.py
rename : base/core/fs.py => core/fs.py
rename : base/core/ignore.py => core/ignore.py
rename : base/core/results.py => core/results.py
rename : base/core/scanner.py => core/scanner.py
rename : base/core/tests/__init__.py => core/tests/__init__.py
rename : base/core/tests/app_cocoa_test.py => core/tests/app_cocoa_test.py
rename : base/core/tests/app_test.py => core/tests/app_test.py
rename : base/core/tests/data.py => core/tests/data.py
rename : base/core/tests/directories_test.py => core/tests/directories_test.py
rename : base/core/tests/engine_test.py => core/tests/engine_test.py
rename : base/core/tests/ignore_test.py => core/tests/ignore_test.py
rename : base/core/tests/results_test.py => core/tests/results_test.py
rename : base/core/tests/scanner_test.py => core/tests/scanner_test.py
rename : me/core/__init__.py => core_me/__init__.py
rename : me/core/app_cocoa.py => core_me/app_cocoa.py
rename : me/core/data.py => core_me/data.py
rename : me/core/fs.py => core_me/fs.py
rename : me/core/scanner.py => core_me/scanner.py
rename : me/core/tests/__init__.py => core_me/tests/__init__.py
rename : me/core/tests/scanner_test.py => core_me/tests/scanner_test.py
rename : pe/core/LICENSE => core_pe/LICENSE
rename : pe/core/__init__.py => core_pe/__init__.py
rename : pe/core/app_cocoa.py => core_pe/app_cocoa.py
rename : pe/core/block.py => core_pe/block.py
rename : pe/core/cache.py => core_pe/cache.py
rename : pe/core/data.py => core_pe/data.py
rename : pe/core/gen.py => core_pe/gen.py
rename : pe/core/matchbase.py => core_pe/matchbase.py
rename : pe/core/modules/block/block.pyx => core_pe/modules/block/block.pyx
rename : pe/core/modules/block/setup.py => core_pe/modules/block/setup.py
rename : pe/core/modules/cache/cache.pyx => core_pe/modules/cache/cache.pyx
rename : pe/core/modules/cache/setup.py => core_pe/modules/cache/setup.py
rename : pe/core/scanner.py => core_pe/scanner.py
rename : pe/core/tests/__init__.py => core_pe/tests/__init__.py
rename : pe/core/tests/block_test.py => core_pe/tests/block_test.py
rename : pe/core/tests/cache_test.py => core_pe/tests/cache_test.py
rename : se/core/LICENSE => core_se/LICENSE
rename : se/core/__init__.py => core_se/__init__.py
rename : se/core/app_cocoa.py => core_se/app_cocoa.py
rename : se/core/data.py => core_se/data.py
rename : se/core/fs.py => core_se/fs.py
rename : se/core/tests/__init__.py => core_se/tests/__init__.py
rename : se/core/tests/fs_test.py => core_se/tests/fs_test.py
rename : me/help/LICENSE => help_me/LICENSE
rename : me/help/__init__.py => help_me/__init__.py
rename : me/help/changelog.yaml => help_me/changelog.yaml
rename : me/help/gen.py => help_me/gen.py
rename : me/help/skeleton/hardcoded.css => help_me/skeleton/hardcoded.css
rename : me/help/skeleton/images/hs_title.png => help_me/skeleton/images/hs_title.png
rename : me/help/templates/base_dg.mako => help_me/templates/base_dg.mako
rename : me/help/templates/credits.mako => help_me/templates/credits.mako
rename : me/help/templates/directories.mako => help_me/templates/directories.mako
rename : me/help/templates/faq.mako => help_me/templates/faq.mako
rename : me/help/templates/intro.mako => help_me/templates/intro.mako
rename : me/help/templates/power_marker.mako => help_me/templates/power_marker.mako
rename : me/help/templates/preferences.mako => help_me/templates/preferences.mako
rename : me/help/templates/quick_start.mako => help_me/templates/quick_start.mako
rename : me/help/templates/results.mako => help_me/templates/results.mako
rename : me/help/templates/versions.mako => help_me/templates/versions.mako
rename : pe/help/LICENSE => help_pe/LICENSE
rename : pe/help/__init__.py => help_pe/__init__.py
rename : pe/help/changelog.yaml => help_pe/changelog.yaml
rename : pe/help/gen.py => help_pe/gen.py
rename : pe/help/skeleton/hardcoded.css => help_pe/skeleton/hardcoded.css
rename : pe/help/skeleton/images/hs_title.png => help_pe/skeleton/images/hs_title.png
rename : pe/help/templates/base_dg.mako => help_pe/templates/base_dg.mako
rename : pe/help/templates/credits.mako => help_pe/templates/credits.mako
rename : pe/help/templates/directories.mako => help_pe/templates/directories.mako
rename : pe/help/templates/faq.mako => help_pe/templates/faq.mako
rename : pe/help/templates/intro.mako => help_pe/templates/intro.mako
rename : pe/help/templates/power_marker.mako => help_pe/templates/power_marker.mako
rename : pe/help/templates/preferences.mako => help_pe/templates/preferences.mako
rename : pe/help/templates/quick_start.mako => help_pe/templates/quick_start.mako
rename : pe/help/templates/results.mako => help_pe/templates/results.mako
rename : pe/help/templates/versions.mako => help_pe/templates/versions.mako
rename : se/help/LICENSE => help_se/LICENSE
rename : se/help/changelog.yaml => help_se/changelog.yaml
rename : se/help/gen.py => help_se/gen.py
rename : se/help/skeleton/hardcoded.css => help_se/skeleton/hardcoded.css
rename : se/help/skeleton/images/hs_title.png => help_se/skeleton/images/hs_title.png
rename : se/help/templates/base_dg.mako => help_se/templates/base_dg.mako
rename : se/help/templates/credits.mako => help_se/templates/credits.mako
rename : se/help/templates/directories.mako => help_se/templates/directories.mako
rename : se/help/templates/faq.mako => help_se/templates/faq.mako
rename : se/help/templates/intro.mako => help_se/templates/intro.mako
rename : se/help/templates/power_marker.mako => help_se/templates/power_marker.mako
rename : se/help/templates/preferences.mako => help_se/templates/preferences.mako
rename : se/help/templates/quick_start.mako => help_se/templates/quick_start.mako
rename : se/help/templates/results.mako => help_se/templates/results.mako
rename : se/help/templates/versions.mako => help_se/templates/versions.mako
rename : base/qt/WARNING => qt/WARNING
rename : base/qt/__init__.py => qt/base/__init__.py
rename : base/qt/app.py => qt/base/app.py
rename : base/qt/details_table.py => qt/base/details_table.py
rename : base/qt/dg.qrc => qt/base/dg.qrc
rename : base/qt/directories_dialog.py => qt/base/directories_dialog.py
rename : base/qt/directories_dialog.ui => qt/base/directories_dialog.ui
rename : base/qt/directories_model.py => qt/base/directories_model.py
rename : base/qt/main_window.py => qt/base/main_window.py
rename : base/qt/main_window.ui => qt/base/main_window.ui
rename : base/qt/platform.py => qt/base/platform.py
rename : base/qt/platform_osx.py => qt/base/platform_osx.py
rename : base/qt/platform_win.py => qt/base/platform_win.py
rename : base/qt/preferences.py => qt/base/preferences.py
rename : base/qt/results_model.py => qt/base/results_model.py
rename : me/qt/app.py => qt/me/app.py
rename : me/qt/build.py => qt/me/build.py
rename : me/qt/details_dialog.py => qt/me/details_dialog.py
rename : me/qt/details_dialog.ui => qt/me/details_dialog.ui
rename : me/qt/dgme.spec => qt/me/dgme.spec
rename : me/qt/gen.py => qt/me/gen.py
rename : me/qt/installer.aip => qt/me/installer.aip
rename : me/qt/preferences.py => qt/me/preferences.py
rename : me/qt/preferences_dialog.py => qt/me/preferences_dialog.py
rename : me/qt/preferences_dialog.ui => qt/me/preferences_dialog.ui
rename : me/qt/profile.py => qt/me/profile.py
rename : me/qt/start.py => qt/me/start.py
rename : me/qt/verinfo => qt/me/verinfo
rename : pe/qt/app.py => qt/pe/app.py
rename : pe/qt/block.py => qt/pe/block.py
rename : pe/qt/build.py => qt/pe/build.py
rename : pe/qt/details_dialog.py => qt/pe/details_dialog.py
rename : pe/qt/details_dialog.ui => qt/pe/details_dialog.ui
rename : pe/qt/dgpe.spec => qt/pe/dgpe.spec
rename : pe/qt/gen.py => qt/pe/gen.py
rename : pe/qt/installer.aip => qt/pe/installer.aip
rename : pe/qt/main_window.py => qt/pe/main_window.py
rename : pe/qt/modules/block/block.pyx => qt/pe/modules/block/block.pyx
rename : pe/qt/modules/block/setup.py => qt/pe/modules/block/setup.py
rename : pe/qt/preferences.py => qt/pe/preferences.py
rename : pe/qt/preferences_dialog.py => qt/pe/preferences_dialog.py
rename : pe/qt/preferences_dialog.ui => qt/pe/preferences_dialog.ui
rename : pe/qt/profile.py => qt/pe/profile.py
rename : pe/qt/start.py => qt/pe/start.py
rename : pe/qt/verinfo => qt/pe/verinfo
rename : se/qt/app.py => qt/se/app.py
rename : se/qt/build.py => qt/se/build.py
rename : se/qt/details_dialog.py => qt/se/details_dialog.py
rename : se/qt/details_dialog.ui => qt/se/details_dialog.ui
rename : se/qt/dgse.spec => qt/se/dgse.spec
rename : se/qt/gen.py => qt/se/gen.py
rename : se/qt/installer.aip => qt/se/installer.aip
rename : se/qt/preferences.py => qt/se/preferences.py
rename : se/qt/preferences_dialog.py => qt/se/preferences_dialog.py
rename : se/qt/preferences_dialog.ui => qt/se/preferences_dialog.ui
rename : se/qt/profile.py => qt/se/profile.py
rename : se/qt/start.py => qt/se/start.py
rename : se/qt/verinfo => qt/se/verinfo
extra : convert_revision : svn%3Ac306627e-7827-47d3-bdf0-9a457c9553a1/trunk%40285
2009-12-30 16:34:41 +00:00