#

dedupe

Here are 67 public repositories matching this topic...

restic / restic

Fast, secure, efficient backup program

go backup dedupe secure-by-default discover restic deduplication

Updated Dec 8, 2021
Go

borgbackup / borg

Open

borg list --format: improve docs, extend placeholders

5

xeruf commented Nov 23, 2021

Read more

enhancement documentation easy good first issue

Open

debug log all cache / index full paths

2

Open

using compression for files cache?

6

Find more good first issues

dedupeio / dedupe

🆔 A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

python clustering dedupe record-linkage python-library entity-resolution datamade dedupe-library de-duplicating

Updated Oct 14, 2021
Python

atlassian / yarn-deduplicate

Deduplication tool for yarn.lock files

yarn dedupe duplicates lock-file yarn-lock duplicated-packages

Updated Oct 18, 2021
JavaScript

jbruchon / jdupes

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Updated Nov 10, 2021
C

blakeembrey / free-style

Sponsor

Make CSS easier and more maintainable by using JavaScript

javascript css typescript js dedupe css-in-js hash minification css-string

Updated Nov 9, 2021
TypeScript

J535D165 / recordlinkage

A toolkit for record linkage and duplicate detection in Python

python machine-learning privacy dedupe record-linkage python-library entity-resolution similarity string-distance deduplication data-matching

Updated Apr 28, 2021
Python

duplicut

nil0x42 / duplicut

Sponsor

Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

c dictionary dedupe password wordlist remove-duplicates uniq hashes cracking wordlist-generator hashcat unique duplicate-detection password-cracking wordlists

Updated Aug 23, 2021
C

dedupeio / csvdedupe

🆔 Command line tool for deduplicating CSV files

cli dedupe record-linkage csv-files entity-resolution

Updated Mar 31, 2020
Python

zingg

zinggAI / zingg

Scalable data mastering, deduplication and entity resolution.

identity data-science identity-resolution spark etl analytics dedupe entity-resolution data-transformation ml fuzzy-matching deduplication datalake masterdata dataengineering fuzzymatch dataquality analytics-engineering data-transformations modern-data-stack

Updated Dec 7, 2021
Java

dedupeio / dedupe-examples

🆔 Examples for using the dedupe library

python dedupe record-linkage entity-resolution

Updated Jun 17, 2021
Python

Zygo / bees

Best-Effort Extent-Same, a btrfs dedupe agent

beer filesystem dedupe extent hash btrfs dedup

Updated Nov 30, 2021
C++

knjcode / imgdupes

Finding and deleting near-duplicate images based on perceptual hash.

image dedupe perceptual-hashing perceptual-hashes deduplicate

Updated Oct 12, 2021
Python

kornelski / dupe-krill

A fast file deduplicator

macos dedupe rust-library file-deduplication hardlinks dupes

Updated Jul 14, 2021
Rust

mail-deduplicate

kdeldycke / mail-deduplicate

Sponsor

📧 CLI to deduplicate mails from mail boxes.

python cli mail email dedupe mailbox cleanup maildir mbox deduplication mh babyl mmdf

Updated Dec 7, 2021
Python

Lakshmipathi / dduper

Open

Add test scripts for RAID config.

Lakshmipathi commented Aug 24, 2020

Right dduper has minimal test script to check basic functionality See ci/gitlab/*.sh . Enhance it add RAID tests.

Read more

good first issue qa

Open

Add travisci.yml for basic sanity test

dssg / pgdedupe

A simple command line interface to the datamade/dedupe library.

python database dedupe record-linkage postgresql data-cleaning deduplication

Updated Nov 15, 2021
Jupyter Notebook

jchristn / WatsonDedupe

Sponsor

Self-contained C# library for data deduplication using Sqlite

compression storage nuget dedupe sqlite-database data-deduplication chunk compress deduplication chunk-data duplicate-data chunk-key

Updated Nov 14, 2021
C#

mighty-justice / django-super-deduper

Utilities for de-duping Django model instances

python django dedupe

Updated Jul 30, 2021
Python

jRimbault / yadf

Yet Another Dupes Finder

dedupe deduplication fdupes duplicate-detection file-deduplication dupes-finder

Updated Dec 8, 2021
Rust

zirkonit / duplex

Duplicate code finder for Elixir

lint elixir escript static-code-analysis dedupe static-analysis duplex

Updated Apr 12, 2021
Elixir

kevinpollet / pocket-deduper

Remove duplicates from your Pocket list

go cli golang tool dedupe duplicates pocket

Updated Apr 13, 2020
Go

dedupeio / dedupe-variable-address

Address Variable Type for dedupe

dedupe dedupe-variable

Updated Mar 31, 2020
Python

dedupeio / dedupe-variable-name

name variable type for dedupe

dedupe dedupe-variable

Updated Jun 30, 2020
Python

hotio / borg

Borg backup in a container.

docker backup dedupe borg

Updated Dec 1, 2021

dedupeio / dedupe-variable-employer

dedupe dedupe-variable

Updated Mar 31, 2020
Python

dedupeio / dedupe-variable-fuzzycategory

Dedupe Variable for Fuzzy Categories

dedupe dedupe-variable

Updated Mar 31, 2020
Python

amoghe / dedup

Deduplicator

compression dedupe compressor deduplication

Updated Aug 9, 2017
Go

dedupeio / parseratorvariable

Base class for dedupe variables for parsed fields

dedupe dedupe-variable

Updated Mar 31, 2020
Python

mterron / swuniq

A command-line tool for deduplicating entries in a file or stream with constant memory usage

cli dedupe filter uniq deduplication deduplicate sliding-window deduping

Updated Apr 19, 2021
C

Improve this page

Add a description, image, and links to the dedupe topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the dedupe topic, visit your repo's landing page and select "manage topics."