Commit Graph

141 Commits

Author SHA1 Message Date
Andrew Senetar e22d7d2fc9
Remove filtering of 0 size files in engine
Files size is already able to be filtered at a higher level, some users
may decide to see zero length files. Fix #321.
2021-08-28 18:16:22 -05:00
Andrew Senetar 3045361243
Add preference to ignore large files, close #430 2021-08-27 05:35:54 -05:00
Andrew Senetar 47dbe805bb
More cleanup and fixed a flake8 build issue 2021-08-25 01:11:24 -05:00
Andrew Senetar f11fccc889
More cleanups
- Cleanup columns.py and tables
- Other misc cleanups
- Remove text_field.py from qtlib as it is not used
- Remove unused variables from image_viewer method
2021-08-25 00:46:33 -05:00
Andrew Senetar d576a7043c
Code cleanups in core and other affected files 2021-08-21 18:02:02 -05:00
Andrew Senetar 0189c29f47
Misc cleanups in core/tests 2021-08-21 03:52:09 -05:00
Andrew Senetar ffe6b7047c
Format all files with black correcting line length 2021-08-15 04:10:18 -05:00
Andrew Senetar 9446f37fad
Remove flake8 E731 Errors
Note: black formatting is now applying correctly as well.
2021-08-15 03:53:43 -05:00
Andrew Senetar e11f996dfc
Merge pull request #908 from glubsy/hash_sample_optimization
Hash sample optimization
2021-08-13 23:41:17 -05:00
glubsy 3dccb686e2 Fix Directories regex test
The entire path to the file would match unless another path separator is added.
2021-08-06 17:18:23 +02:00
glubsy 277bc3fbb8 Add unit tests for hash sample optimization
* Instead of keeping md5 samples separate, merge them as one hash computed from the various selected chunks we picked.
* We don't need to keep a boolean to see whether or not the user chose to optimize; we can simply compare the value of the threshold, since 0 means no optimization currently active.
2021-06-21 22:44:05 +02:00
glubsy a6f83ad3d7 Fix missing regexp after rename
* Doing a full match should be safer to avoid partial results which would result in overly aggressive filtering.
* Add new tests to test suite to cover this issue.
* Fixes #903.
2021-06-19 02:00:25 +02:00
glubsy c4dcfd3d4b Fix stripping (japanese) unicode characters
* Accents are getting removed from Unicode characters to generate similar "words".
* Non-latin characters which cannot be processed that way (eg. japanese, greek, russian, etc.) should not be filtered out at all otherwise files are erroneously skipped or detected as dupes if only some characters make it passed the filter.
* Starting from an arbitrary unicode codepoint (converted to decimal), above which we know it is pointless to try any sort of processing, we leave the characters as is.
* Fix #878.
2021-04-29 05:15:34 +02:00
glubsy f0d3dec517 Fix exclude tests 2020-12-29 16:07:55 +01:00
glubsy 4b4cc04e87 Fix directories tests on Windows
Regexes did not match properly because the separator for Windows is '\\'
2020-12-29 05:35:30 +01:00
glubsy ea11a566af Highlight rows when testing regex string
* Add testing feature to Exclusion dialog to allow users to test regexes against an arbitrary string.
* Fixed test suites.
* Improve comments and help dialog box.
2020-09-01 23:02:58 +02:00
glubsy 4a1641e39d Add test suite, fix bugs 2020-08-31 20:35:56 +02:00
glubsy 470307aa3c Ignore path and filename based on regex
* Added initial draft for test suit
* Fixed small logging bug
2020-08-03 16:19:27 +02:00
Andrew Senetar 5cc439d846
Clean up rest of DeprecationWarnings 2020-06-30 00:51:06 -05:00
Andrew Senetar ee2671a5f3
More Test and Flake8 Cleanup
- Allow flake8 to check more files as well.
2020-06-27 01:08:12 -05:00
Andrew Senetar e05c72ad8c
Upgrade to latest pytest
- Currently some incompatibility in the hscommon tests, commented out
the ones with issues temporarily
- Also updated some deprecation warnings, still more to do
2020-06-25 23:26:48 -05:00
Andrew Senetar de8a0a21b2
Update Packaging
- Add changes from OSX build to local hscommon/build.py
- Update package.py & srcpkg.sh
  - Remove invalid submodule references
  - Update srcpkg.sh to use xz
- Update package.py pyinstaller configuration
  - Call PyInstaller inline
  - Add --noconfirm option to be more script friendly
  - Add UCRT Redist location to path should fix #545 as now all the dlls
    are included
2019-12-31 21:36:52 -06:00
Andrew Senetar 7ba8aa3514
Format files with black
- Format all files with black
- Update tox.ini flake8 arguments to be compatible
- Add black to requirements-extra.txt
- Reduce ignored flake8 rules and fix a few violations
2019-12-31 20:16:27 -06:00
Andrew Senetar 1a04f6ee86
Update to fix Travis Build & flake8 warning
- Fix Travis build by using dist: xenial so python 3.7 is available
- Correct flake8 warning in core/test/app_test.py
2018-12-04 20:16:15 -06:00
Virgil Dupras d5fef949e9 directories: un-recurse get_files() and get_state()
These methods were previously called recursively and it seemed to cause
problems in some cases. The recursive nature of these functions not
bringing any notable advantage and `os.walk()` being of better style
anyway, I removed that recursive nature.

Hopefully fixes #421
2017-10-08 20:35:58 -04:00
Virgil Dupras c58a4817ca Add shelve-based picture cache implementation
Hopefully, this will fix #394 for real this time, that is, without the
need for a messy python executable ship in the app.
2016-11-15 19:58:18 -05:00
Virgil Dupras cc01e8eb09 Move pe.cache.Cache into its own unit, cache_sqlite
This prepares us for an upcoming alternative cache implementation.
2016-11-13 17:01:20 -05:00
Virgil Dupras 9f2c3e7732 Fix failing test on OS X / py35
A 100 recursion limit was too low in that environment.
2016-07-01 15:29:50 -04:00
Virgil Dupras b1ef3dc8fe Simplify progress report during scanning
We now get less progress feedback, but in exchange, our progress job is
simpler. Previously, our progress bar would often get wonky towards the
end of the scan and I didn't have the energy to debug that.

Besides, people don't care about that level of progress feedback.
2016-06-08 12:29:28 -04:00
Virgil Dupras 80f659858c Fail with excplicit message when unable to load results file
Previously, we would simply show an empty results window. Not very
helpful.
2016-06-07 21:34:04 -04:00
Virgil Dupras ef8f8f0e44 Fix broken tests 2016-06-07 21:32:30 -04:00
Virgil Dupras 13fb06a693 Remove ContentsAusio scan type
It had few uses and had a confusing name. People though it did fuzzy
audio data matching, which it does not.
2016-06-06 17:08:41 -04:00
Virgil Dupras a65077f871 Merge core_{se,me,pe} into core.{se,me,pe} 2016-05-31 22:32:37 -04:00
Virgil Dupras 7d749779f2 qt: merge ME edition into SE
(breaks PE temporarily)

Adds a Standard/Music Application Mode button to SE and thus adds the
ability to run ME scan types in SE. When in Music mode, the
Music-specific results window, details panel and preferences panel will
show up.

All preferences except scan_type become shared between app modes
(changing the pref in a mode changes it in the other mode).

Results Window and Details Panel are now re-created at each scan
operation because they could change their type between two runs.

Preferences panel is instantiated on the fly and discarded after close.

This is a very big merge operation and I'm trying to touch as little
code as possible, sometimes at the cost of elegance. I try to minimize
the breakage that this change brings.
2016-05-29 22:37:38 -04:00
Virgil Dupras 5c57a2a8fc Instantiate Scanner on-the-fly
Previously, it would be instantiated on startup.

This will make our job easier for an upcoming SE/ME/PE merge.
2016-05-29 16:52:07 -04:00
Virgil Dupras 130581db53 Apply flake8 checks to tests 2016-05-29 15:02:39 -04:00
Virgil Dupras 9ed4b7abf0 refactoring: take ignore_list out of Scanner class
It's now `DupeGuru` that holds it and passes it to `get_dupe_groups()`,
the only place where it's actually used in `Scanner`.

This will make the SE/ME/PE merge easier by allowing us to instantiate
the Scanner on-the-fly since it doesn't hold state anymore.
2016-05-29 14:13:19 -04:00
Virgil Dupras e7076bc3bd Change license from BSD to GPLv3
See http://www.hardcoded.net/archive2014#2014-12-28 for context
2015-01-03 16:33:16 -05:00
Virgil Dupras fc16ea8c49 Change copyright year to 2015 2015-01-03 16:30:57 -05:00
Virgil Dupras ac32305532 Integrated the jobprogress library into hscommon
I have a fix to make in it and it's really silly to pretend that this
lib is of any use to anybody outside HS apps. Bringing it back here will
make things more simple.
2014-10-05 16:31:16 -04:00
Virgil Dupras ca709a60cf Updated copyright year to 2014 2014-04-19 12:19:11 -04:00
Virgil Dupras 781f13ae1a Overwrite subfolders' state when setting states in folder dialog
Fixes #248
2014-03-15 17:31:33 -04:00
Virgil Dupras c34c9562d3 Make non-numeric delta comparison case insensitive
Fixes #239.
2013-11-23 15:31:20 -05:00
Virgil Dupras 10dbfa9b38 Refactoring: Path API compatibility with pathlib
Refactored dupeGuru to make hscommon.path's API a bit close to pathlib's
API. It's not 100% compatible yet, but it's much better than before.

This is more of a hscommon refactoring than a dupeguru one, but since
duepGuru is the main user of Path, it was the driver behind the
refactoring.

This refactoring also see the introduction of @pathify, which ensure
Path arguments. Previously, we were often unsure of whether the caller
of a function was passing a Path or a str. This problem is now solved
and this allows us to remove hscommon.io, an ill-conceived attempt to
solve that same ambiguity problem.

Fixes #235.
2013-11-16 12:06:16 -05:00
Virgil Dupras e8c42740cf Fixed tests which were broken 2013-11-10 12:54:35 -05:00
Virgil Dupras 12cf9b800b Merge branch 'master' into develop 2013-11-09 16:21:59 -05:00
Virgil Dupras ba7e6494c6 Fixed crash on Dupe Count sorting with Delta + Dupes Only
Fixes #238
2013-11-09 16:20:33 -05:00
Virgil Dupras 72d8160b28 Fix boken tests 2013-11-08 16:45:14 -05:00
Virgil Dupras e1f532e2fd Fixed broken tests 2013-08-04 09:26:18 -04:00
Virgil Dupras b11b97dd7c Improved delta values to support non-numerical values
Delta values now work for non-numerical values. Any column,
when its value differs from its ref, becomes orange.

A column that was already a "delta column" keeps its previous
behavior (dupe cells for these columns are always displayed in
orange).

Sorting behavior, when Dupes Only and Delta Values are enabled
at the same time, has also been extended to non-numerical
values, making it easy to mass-mark dupe rows with orange
values.

Documentation was updated, unit tests were added.

Fixes #213
2013-07-28 17:45:23 -04:00