dedupe

Star

Here are 69 public repositories matching this topic...

restic / restic

Star

Fast, secure, efficient backup program

go backup dedupe secure-by-default restic deduplication

Updated Oct 17, 2022
Go

borgbackup / borg

Star

Deduplicating archiver with compression and authenticated encryption.

python c ssh backup compression encryption dedupe cython borgbackup python-3 deduplication

Updated Oct 19, 2022
Python

dedupeio / dedupe

Star

🆔 A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

python clustering dedupe record-linkage python-library entity-resolution datamade dedupe-library de-duplicating

Updated Oct 17, 2022
Python

scinos / yarn-deduplicate

Star

Deduplication tool for yarn.lock files

yarn dedupe duplicates lock-file yarn-lock duplicated-packages

Updated Oct 18, 2022
TypeScript

jbruchon / jdupes

Star

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Updated Oct 17, 2022
C

J535D165 / recordlinkage

Sponsor

Star

A powerful and modular toolkit for record linkage and duplicate detection in Python

python machine-learning privacy dedupe record-linkage python-library entity-resolution similarity string-distance deduplication data-matching utrecht-university

Updated Apr 19, 2022
Python

blakeembrey / free-style

Sponsor

Star

Make CSS easier and more maintainable by using JavaScript

javascript css typescript js dedupe css-in-js hash minification css-string

Updated Nov 9, 2021
TypeScript

zinggAI / zingg

Star

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Updated Oct 19, 2022
Java

nil0x42 / duplicut

Sponsor

Star

Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

c dictionary dedupe password wordlist remove-duplicates uniq hashes cracking wordlist-generator hashcat unique duplicate-detection password-cracking wordlists

Updated Jun 25, 2022
C

Zygo / bees

Star

Best-Effort Extent-Same, a btrfs dedupe agent

dedupe btrfs dedup

Updated Oct 13, 2022
C++

dedupeio / csvdedupe

Star

🆔 Command line tool for deduplicating CSV files

cli dedupe record-linkage csv-files entity-resolution

Updated Mar 31, 2020
Python

dedupeio / dedupe-examples

Star

🆔 Examples for using the dedupe library

python dedupe record-linkage entity-resolution

Updated Jan 19, 2022
Python

knjcode / imgdupes

Star

Finding and deleting near-duplicate images based on perceptual hash.

image dedupe perceptual-hashing perceptual-hashes deduplicate

Updated Sep 30, 2022
Python

kornelski / dupe-krill

Star

A fast file deduplicator

macos dedupe rust-library file-deduplication hardlinks dupes

Updated Oct 19, 2022
Rust

kdeldycke / mail-deduplicate

Sponsor

Star

📧 CLI to deduplicate mails from mail boxes.

python cli mail email dedupe mailbox cleanup maildir mbox deduplication mh babyl mmdf

Updated Oct 18, 2022
Python

Lakshmipathi / dduper

Star

Fast block-level out-of-band BTRFS deduplication tool.

dedupe btrfs deduplication btrfs-progs btrfs-tools

Updated Jun 5, 2021
Python

dssg / pgdedupe

Star

A simple command line interface to the datamade/dedupe library.

python database dedupe record-linkage postgresql data-cleaning deduplication

Updated Jun 21, 2022
Jupyter Notebook

jRimbault / yadf

Star

Yet Another Dupes Finder

dedupe deduplication fdupes duplicate-detection file-deduplication dupes-finder

Updated Sep 29, 2022
Rust

jchristn / WatsonDedupe

Sponsor

Star

Self-contained C# library for data deduplication using Sqlite

compression storage nuget dedupe sqlite-database data-deduplication chunk compress deduplication chunk-data duplicate-data chunk-key

Updated Jan 7, 2022
C#

mighty-justice / django-super-deduper

Star

Utilities for de-duping Django model instances

python django dedupe

Updated Jul 30, 2021
Python

Improve this page

Add a description, image, and links to the dedupe topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the dedupe topic, visit your repo's landing page and select "manage topics."

Learn more