arsenetar/dupeguru
1
0
Fork 0
mirror of https://github.com/arsenetar/dupeguru.git synced 2024-11-16 20:29:02 +00:00
Code Releases Wiki Activity
2,000 Commits 7 Branches 108 Tags 24 MiB
5451f55219
Commit Graph

361 Commits

Author SHA1 Message Date
Andrew Senetar
0a4e61edf5
Additional cleanup per mypy 
- Add Callable type to hasher (should realy be more specific...)
- Add type hint to COLUMNS in qtlib/table.py
- Use Qt.ItemFlag.ItemIsEnabled instead of Qt.itemIsEnabled in qtlib/table.py
 2022-04-30 05:16:46 -05:00
Andrew Senetar
d73a85b82e
Add type hints for compiled modules 2022-04-30 05:11:54 -05:00
Andrew Senetar
63dd4d4561
Apply pyupgrade changes 2022-04-27 20:53:12 -05:00
Andrew Senetar
c5818b1d1f
Add option to profile scans 
- Add preference for profiling scans
- Move debug options to tab in preferences
- Add label with clickable link to debug output (appdata) to debug tab in preferences
- Update translation source files
 2022-03-31 00:16:37 -05:00
Andrew Senetar
a470a8de25
Update fs.py to optimize stat() calls 
- Update to get size and mtime at time of class creation when os.DirEntry is used for initialization.
- Folders still calculate size later for folder scans.
- Ref #962, #959
 2022-03-30 22:58:01 -05:00
Andrew Senetar
a37b5b0eeb
Fix #988 2022-03-30 01:06:51 -05:00
Andrew Senetar
efd500ecc1
Update directory scanning to use os.scandir() 
- Change to use os.scandir() instead of os.walk() to leverage DirEntry objects.
- Avoids extra calls to stat() on files during fs.can_handle()
- See 3x speed improvement on Windows in some cases
 2022-03-29 23:37:56 -05:00
Andrew Senetar
43fcc52291
Replace pathlib.glob() with os.scandir() in fs.py 2022-03-29 22:35:38 -05:00
Andrew Senetar
50f5db1543
Update fs to support DirEntry on get_file() 2022-03-29 22:32:36 -05:00
Andrew Senetar
a5b0ccdd02
Improve performance of Directories.get_state() 2022-03-29 21:48:14 -05:00
Andrew Senetar
ebb81d9f03
Remove pathlib function added in Python 3.9 2022-03-28 00:06:32 -05:00
Andrew Senetar
da9f8b2b9d
Squashed commit of the following: 
commit 8b15fe9a502ebf4841c6529e7098cef03a6a5e6f
Author: Andrew Senetar <arsenetar@gmail.com>
Date:   Sun Mar 27 23:48:15 2022 -0500

    Finish up changes to copy_or_move

commit 21f6a32cf3186a400af8f30e67ad2743dc9a49bd
Author: Andrew Senetar <arsenetar@gmail.com>
Date:   Thu Mar 17 23:56:52 2022 -0500

    Migrate from hscommon.path to pathlib
    - Part one, this gets all hscommon and core tests passing
    - App appears to be able to load directories and complete scans, need further testing
    - app.py copy_or_move needs some additional work
 2022-03-27 23:50:03 -05:00
Andrew Senetar
9f40e4e786
Squashed commit of the following: 
commit 5eb515f666bfa1ff06c2e96bdc351a4b7456580e
Author: Andrew Senetar <arsenetar@gmail.com>
Date:   Sun Mar 27 22:19:39 2022 -0500

    Add fallback to md5 if xxhash not available

    Mainly here for the case when distributions have not packaged python3-xxhash.

commit 51b18d4c84
Author: Andrew Senetar <arsenetar@gmail.com>
Date:   Sat Mar 19 15:25:46 2022 -0500

    Switch file hashing to xxhash instead of md5

    - Improves performance significantly in some cases
    - Add xxhash to requirements.txt and sort requirements
    - Rename md5 based members to digest
    - Update all tests to use new member names and hashing methods
    - Update hash db code to upgrade schema

    NOTE: May consider supporting multiple hashing algorithms in the future.
 2022-03-27 22:27:13 -05:00
Andrew Senetar
86bf9b39d0
Add update check function and call from about 
- Implement a update check against the GitHub releases via the api
- Add semantic-version dependency
- Add automatic check when opening about dialog
 2022-03-27 21:13:27 -05:00
Andrew Senetar
1bc206e62d
Bump version to 4.2.1 2022-03-19 19:02:41 -05:00
Andrew Senetar
cbfa8720f1
Update imports for objc module 2022-03-09 05:01:12 -06:00
Andrew Senetar
85e22089bd
Black formatting changes 2022-02-09 21:49:51 -06:00
Andrew Senetar
2c11eecf97
Update version and changelog to 4.2.0 2022-01-24 22:28:40 -06:00
Dobatymo
77460045c4 clean up abstraction 2021-10-29 15:24:47 +08:00
Dobatymo
9753afba74 change FilesDB to singleton class 
move hash calculation back in to Files class
clear cache now clears hash cache in addition to picture cache
 2021-10-29 15:12:40 +08:00
Dobatymo
1ea108fc2b changed cache filename 2021-10-29 15:12:40 +08:00
Dobatymo
2f02a6010d implement hash cache for md5 hash based on sqlite 2021-10-29 15:12:40 +08:00
Andrew Senetar
1d60e124ee
Update invoke_custom_command to run for all selected items 2021-09-02 20:48:25 -05:00
Andrew Senetar
e22d7d2fc9
Remove filtering of 0 size files in engine 
Files size is already able to be filtered at a higher level, some users
may decide to see zero length files. Fix #321.
 2021-08-28 18:16:22 -05:00
Andrew Senetar
78fb052d77
Add more progress details to getmatches, ref #700 2021-08-28 04:58:22 -05:00
Andrew Senetar
9805cba10d
Use different message for direct delete success, close #904 2021-08-28 04:27:34 -05:00
Andrew Senetar
4c3dfe2f1f
Provide more feedback during scans 
- Add output for number of collected files / folders
- Update to allow indeterminate progress bar
- Remove unused hscommon\jobprogress\qt.py
 2021-08-28 04:05:07 -05:00
Andrew Senetar
3045361243
Add preference to ignore large files, close #430 2021-08-27 05:35:54 -05:00
Andrew Senetar
809116c764
Fix CodeQL Alerts 
- Cast int to Py_ssize_t for multiplication
 2021-08-26 03:43:31 -05:00
Andrew Senetar
47dbe805bb
More cleanup and fixed a flake8 build issue 2021-08-25 01:11:24 -05:00
Andrew Senetar
f11fccc889
More cleanups 
- Cleanup columns.py and tables
- Other misc cleanups
- Remove text_field.py from qtlib as it is not used
- Remove unused variables from image_viewer method
 2021-08-25 00:46:33 -05:00
Andrew Senetar
d576a7043c
Code cleanups in core and other affected files 2021-08-21 18:02:02 -05:00
Andrew Senetar
1ef5f56158
Code cleanups in hscommon & external effects 2021-08-21 16:56:27 -05:00
Andrew Senetar
0189c29f47
Misc cleanups in core/tests 2021-08-21 03:52:09 -05:00
Andrew Senetar
58c04ff9ad
Switch from hsaudiotag to mutagen, close #440 
- This opens up the ability to support more tags and audio information
- Also makes progress on #333
 2021-08-19 00:14:26 -05:00
Andrew Senetar
be10b462fc
Add portable mode 
If settings.ini is present next to the executable, will run in portable mode.
This results in settings, data, and cache all being in same folder as dupeGuru.
 2021-08-17 21:12:32 -05:00
Andrew Senetar
ffe6b7047c
Format all files with black correcting line length 2021-08-15 04:10:18 -05:00
Andrew Senetar
9446f37fad
Remove flake8 E731 Errors 
Note: black formatting is now applying correctly as well.
 2021-08-15 03:53:43 -05:00
Andrew Senetar
e11f996dfc
Merge pull request #908 from glubsy/hash_sample_optimization 
Hash sample optimization
 2021-08-13 23:41:17 -05:00
glubsy
e95306e58f Fix flake 8 2021-08-14 02:52:00 +02:00
glubsy
891a875990 Cache constant expression 
Perhaps the python byte code is already optimized, but just in case it is not, keep pre-compute the constant expression.
 2021-08-13 21:33:21 +02:00
glubsy
545a5a75fb Fix for older python versions 
The "walrus" operator is only available in python 3.8 and later. Fall back to more traditional notation.
 2021-08-13 20:56:33 +02:00
glubsy
7b764f183e Avoid partially hashing small files 
Computing 3 hash samples for files less than 3MiB (3 * CHUNK_SIZE) is not efficient since spans of later samples would overlap a previous one.
Therefore we can simply return the hash of the entire small file instead.
 2021-08-13 20:47:01 +02:00
glubsy
3dccb686e2 Fix Directories regex test 
The entire path to the file would match unless another path separator is added.
 2021-08-06 17:18:23 +02:00
Andrew Senetar
0db66baace
Merge pull request #907 from glubsy/missing_renamed_regex 
Missing renamed regex
 2021-08-03 22:26:08 -05:00
glubsy
23c59787e5 Fix infinite recursion 
Force the Results to update its internal __dupes list whenever at least one group has re-prioritized and changed its dupes/ref.
 2021-06-23 05:36:10 +02:00
glubsy
a51f263632 Fix refs appearing in dupes-only view 
* Some refs appeared in the dupes-only view after a re-prioritization was done a second time.
* It seems the core.Results.__dupes list was not properly updated whenever core.app.Dupeguru.reprioritize_groups() -> core.Results.sort_dupes() was called.
When a re-prioritization is done, some refs became dupe, and some dupes became ref in their place. So we need to update the new state of the internal list of dupes kept by the Results object, instead of relying on the outdated cached one.
* Fix #757.
 2021-06-22 22:57:57 +02:00
glubsy
718ca5b313 Remove unused import 2021-06-22 02:41:33 +02:00
glubsy
277bc3fbb8 Add unit tests for hash sample optimization 
* Instead of keeping md5 samples separate, merge them as one hash computed from the various selected chunks we picked.
* We don't need to keep a boolean to see whether or not the user chose to optimize; we can simply compare the value of the threshold, since 0 means no optimization currently active.
 2021-06-21 22:44:05 +02:00
glubsy
e07dfd5955 Add partial hashes optimization for big files 
* Big files above the user selected threshold can be partially hashed in 3 places.
* If the user is willing to take the risk, we consider files with identical md5samples as being identical.
 2021-06-21 19:03:21 +02:00