Fast, secure, efficient backup program
-
Updated
Sep 12, 2021 - Go
#
deduplication
Here are 240 public repositories matching this topic...
Prometheus Alertmanager
-
Updated
Sep 15, 2021 - Go
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
c
nlp
machine-learning
natural-language-processing
record-linkage
address
address-parser
international
deduplication
deduping
-
Updated
Sep 8, 2021 - C
Extremely fast tool to remove duplicates and other lint from your filesystem
-
Updated
Sep 9, 2021 - C
budachst
commented
Apr 13, 2021
I am synching my local repo to a Wasabi S3 bucket daily. I am capturing the output from the cron job and have that one sent as an e-mail to myself. The output of the sync-to command is reallly… lengthy and I am actually not interested in that output.
How about adding the --no-progress option to this command, so it only outputs any errors ans not the whole progress?
Simple, configuration-driven backup software for servers and workstations
-
Updated
Sep 14, 2021 - Python
A powerful duplicate file finder and an enhanced fork of 'fdupes'.
c
windows
macos
linux
fast
dedupe
duplicate-files
bsd
macosx
btrfs
mit-license
win32
symlinks
deduplication
fdupes
symlink-files
delete-files
hard-links
hardlinking
deduplication-command
-
Updated
Jun 4, 2021 - C
Data deduplication engine, supporting optional compression and public key encryption.
-
Updated
Sep 7, 2021 - Rust
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
machine-learning
natural-language-processing
information-retrieval
clustering
record-linkage
fuzzy-matching
deduplication
-
Updated
May 5, 2021 - JavaScript
A toolkit for record linkage and duplicate detection in Python
python
machine-learning
privacy
dedupe
record-linkage
python-library
entity-resolution
similarity
string-distance
deduplication
data-matching
-
Updated
Apr 28, 2021 - Python
High level CLI utility for restic
-
Updated
Sep 1, 2021 - Go
A list of free data matching and record linkage software.
-
Updated
Apr 20, 2020
Locality Sensitive Hashing using MinHash in Python/Cython to detect near duplicate text documents
-
Updated
Jun 7, 2020 - Python
A pair of kernel modules which provide pools of deduplicated and/or compressed block storage.
-
Updated
Sep 1, 2021 - C
Quickly detect already witnessed data.
go
golang
command-line
memory
history
duplicates
command-line-tool
duplicate-values
probabilistic
deduplication
-
Updated
Jul 16, 2017 - Go
Userspace tools for managing VDO volumes.
-
Updated
Sep 1, 2021 - C
zouzias
commented
Apr 21, 2019
Is your feature request related to a problem? Please describe.
Currently, MapType are not supported for Spark DataFrames
Describe the solution you'd like
Add support for MapType Spark DataFrame columns
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other co
Implementation in Apache Spark of the EM algorithm to estimate parameters of Fellegi-Sunter's canonical model of record linkage.
spark
record-linkage
entity-resolution
fuzzy-matching
deduplication
em-algorithm
data-matching
deduplicate-data
-
Updated
Jul 5, 2021 - Roff
RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用
-
Updated
Aug 18, 2021 - Java
You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash
-
Updated
Jul 19, 2020 - Go
Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
-
Updated
May 5, 2021 - Python
Benji Backup: A block based deduplicating backup software for Ceph RBD images, iSCSI targets, image files and block devices
-
Updated
Sep 3, 2021 - Python
CLI utility to find duplicate files
-
Updated
Aug 26, 2021 - C
Tool for managing data-deduplication within extant compressed archive files, along with a relatively performant BK tree implementation for fuzzy image searching.
-
Updated
Apr 11, 2019 - Python
PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
python
deep-learning
record-linkage
entity-resolution
pytorch
embeddings
representation-learning
deduplication
entity-matching
data-matching
approximate-nearest-neighbors
-
Updated
Jul 16, 2021 - Jupyter Notebook
Record Linkage ToolKit (Find and link entities)
record-linkage
entity-resolution
similarity
deduplication
linkage
similarity-metric
string-similarity
-
Updated
Sep 3, 2021 - Python
Scalable fuzzy matching for data mastering, deduplication and entity resolution.
data-science
identity-resolution
spark
etl
dedupe
entity-resolution
data-transformation
ml
fuzzy-matching
deduplication
masterdata
dataengineering
fuzzymatch
datapreparation
analytics-engineering
data-transformations
-
Updated
Sep 14, 2021 - Java
Improve this page
Add a description, image, and links to the deduplication topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the deduplication topic, visit your repo's landing page and select "manage topics."
Considering issues like #5954 are bad to solve / debug without this information, borg should debug log full paths of all caches / indexes it uses.