deduplication

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

cloud backup encryption google-cloud-storage hacktoberfest deduplication

Updated Aug 28, 2022
Go

sahib / rmlint

Star

Extremely fast tool to remove duplicates and other lint from your filesystem

lint python c filesystem duplicates deduplication fdupes

Updated Aug 20, 2022
C

borgmatic-collective / borgmatic

Star

Simple, configuration-driven backup software for servers and workstations

python backup compression borg borgbackup servers deduplication

Updated Aug 25, 2022
Python

jbruchon / jdupes

Star

A powerful duplicate file finder and an enhanced fork of 'fdupes'.

Updated Mar 18, 2022
C

mhx / dwarfs

Star

A fast high compression read-only file system

Updated Aug 4, 2022
C++

dpc / rdedup

Star

Data deduplication engine, supporting optional compression and public key encryption.

backup encryption data-deduplication deduplication

Updated Aug 25, 2022
Rust

J535D165 / recordlinkage

Sponsor

Star

A powerful and modular toolkit for record linkage and duplicate detection in Python

python machine-learning privacy dedupe record-linkage python-library entity-resolution similarity string-distance deduplication data-matching utrecht-university

Updated Apr 19, 2022
Python

Yomguithereal / talisman

Sponsor

Star

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

machine-learning natural-language-processing information-retrieval clustering record-linkage fuzzy-matching deduplication

Updated May 5, 2021
JavaScript

zinggAI / zingg

Star

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Updated Aug 25, 2022
Java

cupcakearmy / autorestic

Sponsor

Star

Config driven, easy backup cli for restic.

config cli backup incremental pruning restic deduplication incremental-backup config-driven

Updated Aug 26, 2022
Go

J535D165 / data-matching-software

Sponsor

Star

A list of free data matching and record linkage software.

open-source machine-learning awesome record-linkage entity-resolution fuzzy-matching software awesome-list deduplication data-matching

Updated Jul 29, 2022

moj-analytical-services / splink

Star

Fast, accurate and scalable probabilistic data linkage using your choice of SQL backend

spark record-linkage entity-resolution fuzzy-matching deduplication em-algorithm data-matching deduplicate-data

Updated Aug 27, 2022
Python

mattilyra / LSH

Star

Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents

cython lsh minhash deduplication duplicate-documents

Updated Jun 7, 2020
Python

dm-vdo / kvdo

Star

A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.

compression storage linux-kernel kernel-modules deduplication vdo

Updated Aug 22, 2022
C

rin-nas / postgresql-patterns-library

Star

Коллекция готовых SQL запросов для PostgreSQL по часто возникающим задачам (получение и модификация данных, ускорение запросов, обслуживание БД)

syntax recipes postgres csv parse validation sql detect howto email phone function postgresql update batch check gender deduplication determine

Updated Aug 23, 2022
PLpgSQL

Improve this page

Add a description, image, and links to the deduplication topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the deduplication topic, visit your repo's landing page and select "manage topics."

Learn more