343 Commits

Author	SHA1	Message	Date
Dobatymo	77460045c4	clean up abstraction	2021-10-29 15:24:47 +08:00
Dobatymo	9753afba74	change FilesDB to singleton class ... move hash calculation back in to Files class clear cache now clears hash cache in addition to picture cache	2021-10-29 15:12:40 +08:00
Dobatymo	1ea108fc2b	changed cache filename	2021-10-29 15:12:40 +08:00
Dobatymo	2f02a6010d	implement hash cache for md5 hash based on sqlite	2021-10-29 15:12:40 +08:00
Andrew Senetar	1d60e124ee	Update invoke_custom_command to run for all selected items	2021-09-02 20:48:25 -05:00
Andrew Senetar	e22d7d2fc9	Remove filtering of 0 size files in engine ... Files size is already able to be filtered at a higher level, some users may decide to see zero length files. Fix #321.	2021-08-28 18:16:22 -05:00
Andrew Senetar	78fb052d77	Add more progress details to getmatches, ref #700	2021-08-28 04:58:22 -05:00
Andrew Senetar	9805cba10d	Use different message for direct delete success, close #904	2021-08-28 04:27:34 -05:00
Andrew Senetar	4c3dfe2f1f	Provide more feedback during scans ... - Add output for number of collected files / folders - Update to allow indeterminate progress bar - Remove unused hscommon\jobprogress\qt.py	2021-08-28 04:05:07 -05:00
Andrew Senetar	3045361243	Add preference to ignore large files, close #430	2021-08-27 05:35:54 -05:00
Andrew Senetar	809116c764	Fix CodeQL Alerts ... - Cast int to Py_ssize_t for multiplication	2021-08-26 03:43:31 -05:00
Andrew Senetar	47dbe805bb	More cleanup and fixed a flake8 build issue	2021-08-25 01:11:24 -05:00
Andrew Senetar	f11fccc889	More cleanups ... - Cleanup columns.py and tables - Other misc cleanups - Remove text_field.py from qtlib as it is not used - Remove unused variables from image_viewer method	2021-08-25 00:46:33 -05:00
Andrew Senetar	d576a7043c	Code cleanups in core and other affected files	2021-08-21 18:02:02 -05:00
Andrew Senetar	1ef5f56158	Code cleanups in hscommon & external effects	2021-08-21 16:56:27 -05:00
Andrew Senetar	0189c29f47	Misc cleanups in core/tests	2021-08-21 03:52:09 -05:00
Andrew Senetar	58c04ff9ad	Switch from hsaudiotag to mutagen, close #440 ... - This opens up the ability to support more tags and audio information - Also makes progress on #333	2021-08-19 00:14:26 -05:00
Andrew Senetar	be10b462fc	Add portable mode ... If settings.ini is present next to the executable, will run in portable mode. This results in settings, data, and cache all being in same folder as dupeGuru.	2021-08-17 21:12:32 -05:00
Andrew Senetar	ffe6b7047c	Format all files with black correcting line length	2021-08-15 04:10:18 -05:00
Andrew Senetar	9446f37fad	Remove flake8 E731 Errors ... Note: black formatting is now applying correctly as well.	2021-08-15 03:53:43 -05:00
Andrew Senetar	e11f996dfc	Merge pull request #908 from glubsy/hash_sample_optimization ... Hash sample optimization	2021-08-13 23:41:17 -05:00
glubsy	e95306e58f	Fix flake 8	2021-08-14 02:52:00 +02:00
glubsy	891a875990	Cache constant expression ... Perhaps the python byte code is already optimized, but just in case it is not, keep pre-compute the constant expression.	2021-08-13 21:33:21 +02:00
glubsy	545a5a75fb	Fix for older python versions ... The "walrus" operator is only available in python 3.8 and later. Fall back to more traditional notation.	2021-08-13 20:56:33 +02:00
glubsy	7b764f183e	Avoid partially hashing small files ... Computing 3 hash samples for files less than 3MiB (3 * CHUNK_SIZE) is not efficient since spans of later samples would overlap a previous one. Therefore we can simply return the hash of the entire small file instead.	2021-08-13 20:47:01 +02:00
glubsy	3dccb686e2	Fix Directories regex test ... The entire path to the file would match unless another path separator is added.	2021-08-06 17:18:23 +02:00
Andrew Senetar	0db66baace	Merge pull request #907 from glubsy/missing_renamed_regex ... Missing renamed regex	2021-08-03 22:26:08 -05:00
glubsy	23c59787e5	Fix infinite recursion ... Force the Results to update its internal __dupes list whenever at least one group has re-prioritized and changed its dupes/ref.	2021-06-23 05:36:10 +02:00
glubsy	a51f263632	Fix refs appearing in dupes-only view ... * Some refs appeared in the dupes-only view after a re-prioritization was done a second time. * It seems the core.Results.__dupes list was not properly updated whenever core.app.Dupeguru.reprioritize_groups() -> core.Results.sort_dupes() was called. When a re-prioritization is done, some refs became dupe, and some dupes became ref in their place. So we need to update the new state of the internal list of dupes kept by the Results object, instead of relying on the outdated cached one. * Fix #757.	2021-06-22 22:57:57 +02:00
glubsy	718ca5b313	Remove unused import	2021-06-22 02:41:33 +02:00
glubsy	277bc3fbb8	Add unit tests for hash sample optimization ... * Instead of keeping md5 samples separate, merge them as one hash computed from the various selected chunks we picked. * We don't need to keep a boolean to see whether or not the user chose to optimize; we can simply compare the value of the threshold, since 0 means no optimization currently active.	2021-06-21 22:44:05 +02:00
glubsy	e07dfd5955	Add partial hashes optimization for big files ... * Big files above the user selected threshold can be partially hashed in 3 places. * If the user is willing to take the risk, we consider files with identical md5samples as being identical.	2021-06-21 19:03:21 +02:00
glubsy	a6f83ad3d7	Fix missing regexp after rename ... * Doing a full match should be safer to avoid partial results which would result in overly aggressive filtering. * Add new tests to test suite to cover this issue. * Fixes #903.	2021-06-19 02:00:25 +02:00
glubsy	ab8750eedb	Fix partial regex match yielding false positive	2021-06-17 03:49:59 +02:00
glubsy	22033211d6	Fix exception when deleting while in delta view	2021-05-31 23:49:21 +02:00
glubsy	f1ae478433	Fix including character at the border	2021-04-29 05:29:35 +02:00
glubsy	c4dcfd3d4b	Fix stripping (japanese) unicode characters ... * Accents are getting removed from Unicode characters to generate similar "words". * Non-latin characters which cannot be processed that way (eg. japanese, greek, russian, etc.) should not be filtered out at all otherwise files are erroneously skipped or detected as dupes if only some characters make it passed the filter. * Starting from an arbitrary unicode codepoint (converted to decimal), above which we know it is pointless to try any sort of processing, we leave the characters as is. * Fix #878.	2021-04-29 05:15:34 +02:00
Andrew Senetar	4a40b346a4	Update to 4.1.1	2021-03-21 22:50:33 -05:00
glubsy	528dedd813	Fix problematic string for translations ... Some languages have very different phrase syntaxes depending on which word is used. Better used two separate strings than a dynamically created one.	2021-02-09 01:40:00 +01:00
Sergey Zhuravlevich	32dcd90b50	Prioritize dialog: allow removing multiple prioritizations at once ... Removing prioritizations one-by-one can be tedious. This commit enables extended selection in the prioritizations list. Multiple items can be selected with conventional methods, such as holding down Ctrl or Shift key and clicking the items or holding down the left mouse button and hovering the cursor over the list. All items also can be selected with Ctrl+A. Multiple items drag-n-drop is also possible. To avoid confusion, the selection in the prioritizations list is cleared after the items are removed or drag-n-dropped. Signed-off-by: Sergey Zhuravlevich <sergey@zhur.xyz>	2021-01-07 17:42:30 +01:00
Sergey Zhuravlevich	c2fef8d624	Prioritize dialog: allow adding multiple criteria at once ... Adding criteria to the prioritizations list one-by-one can be tedious. This commit enables extended selection in the criteria list and implements adding multiple items. Multiple criteria can be selected with conventional methods, such as holding down Ctrl or Shift keys and clicking the items or holding down the left mouse button and hovering the cursor over the list. All items also can be selected with Ctrl+A. Signed-off-by: Sergey Zhuravlevich <sergey@zhur.xyz>	2021-01-07 17:42:07 +01:00
glubsy	b138dfad33	Fix exception when testing invalid regex ... * If a regex in the table is invalid and failed to compile, its "compiled" property is None. * Only test against the regex if its compilation worked.	2020-12-30 22:50:42 +01:00
glubsy	c1d94d6771	Merge branch 'master' into dev	2020-12-29 20:10:42 +01:00
glubsy	f0d3dec517	Fix exclude tests	2020-12-29 16:07:55 +01:00
glubsy	e533a396fb	Remove redundant check	2020-12-29 05:39:26 +01:00
glubsy	4b4cc04e87	Fix directories tests on Windows ... Regexes did not match properly because the separator for Windows is '\\'	2020-12-29 05:35:30 +01:00
glubsy	6bc619055e	Change version to 4.1.0	2020-12-06 20:13:03 +01:00
glubsy	680cb581c1	Merge branch 'master' into exclude_list	2020-10-28 03:58:05 +01:00
glubsy	32d66cd19b	Move up to 4.0.5 ... * Initial push to 4.0.5 milestone * Update changelog	2020-10-27 19:38:51 +01:00
glubsy	2875448c71	Merge branch 'save_directories' into dev	2020-10-27 16:23:49 +01:00