Skip to content
master
Switch branches/tags
336 branches 52 tags
Code

Latest commit

@blt
blt chore: Slim unused dependencies from the project (#11541)
7556dc3 Feb 26, 2022
chore: Slim unused dependencies from the project (#11541) 
* Slim unused dependencies from the project

This commit enables two global lints: `unused_extern_crates` and
`unused_crate_dependencies`. The remainder of the changes chase what those lints
flagged. This should, ideally, mean we generate less dependabot updates, require
slightly less build time. I have not applied these lints to sub-crates.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* criterion ding

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* incremental progress on cargo-hack

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more incremental progress

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* further incremental progress

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* slim features

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more incremental progress

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* fix shutdown tests

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* dinged by rusoto_es, assert_cmd

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* less dings

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Back off unused_crate_dependencies

While this is an interesting lint it's overly strict, at least for the state of
the root crate. Our dev-dependencies get flagged quite a bit and, while we do
have inconsistent use of them, that I'm less concerned about. To resolve we'd
need to pull these into `src/lib.rs` as _ imports.

I do think the basic idea is a good one and can apply to sub-crates well
enough. Feasible to pick this back up after we slim feature flags out of the
project.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* indoc dings

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more dings

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* remove dev-dependency dance

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* warp is non-optional

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* bollard non-optional

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* more reqwest/json for es

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* reqwest dings

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* value ding

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
7556dc3

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
.cargo
chore(dev): Avoid unnecessary cargo check/clippy rebuilds (#10450)
Dec 14, 2021
.github
chore(ci): Refine testing for needed env vars for datadog-agent integ…
Feb 26, 2022
benches
chore: Run cargo hack on each workspace member and all targets (#11463)
Feb 18, 2022
config
chore(elasticsearch sink): remove deprecated config (#10488)
Dec 21, 2021
cue.mod
chore: Remove more references to timberio (#11111)
Feb 9, 2022
distribution
chore(releasing): Update kubernetes manifests (#11408)
Feb 16, 2022
docs
chore(internal docs): Clarify error tag in component spec (#11317)
Feb 11, 2022
lib
chore(core): Disambiguate LogEvent and TraceEvent (#11584)
Feb 26, 2022
proto
enhancement(datadog_metrics sink): rewrite to the new model + add ske…
Oct 29, 2021
rfcs
chore: rename ElasticSearch to Elasticsearch (#11437)
Feb 18, 2022
scripts
chore(tests): Update prometheus version for integration tests (#10462)
Feb 26, 2022
skaffold
skaffold: Update .metadata.name to fix local vector binary build (#…
Jan 27, 2022
soaks
chore: Avoid using registry for soak images (#11474)
Feb 18, 2022
src
chore: Slim unused dependencies from the project (#11541)
Feb 26, 2022
testing
enhancement(topology): Send arrays of events through the topology (#1…
Feb 25, 2022
tests
chore: Slim unused dependencies from the project (#11541)
Feb 26, 2022
website
fix(api): Change vector top --human_metrics short flag (#11581)
Feb 26, 2022
.dockerignore
chore(deps): Upgrade to Rust 1.56.1 (#9858)
Nov 4, 2021
.editorconfig
chore: Separate internal and external docs (#8557)
Aug 12, 2021
.gitattributes
chore: Update .gitattributes rules
Oct 29, 2021
.gitignore
chore: Address sources of non-vector variability (#10730)
Jan 11, 2022
.graphqlconfig
feat(observability): vector top, release candidate (#4886)
Nov 9, 2020
.rustfmt.toml
chore: reformatting imports (#10496)
Dec 16, 2021
CHANGELOG.md
chore: Add link to CHANGELOG.md (#8885)
Aug 26, 2021
CODE_OF_CONDUCT.md
chore: Remove more references to timberio (#11111)
Feb 9, 2022
CONTRIBUTING.md
docs: Add link to CONTRIBUTING.md to root directory (#9357)
Sep 29, 2021
Cargo.lock
chore: Slim unused dependencies from the project (#11541)
Feb 26, 2022
Cargo.toml
chore: Slim unused dependencies from the project (#11541)
Feb 26, 2022
Cross.toml
chore(platforms): Pass through jemalloc page size env vars (#10940)
Jan 20, 2022
LICENSE
chore: Switch to the MPL 2.0 license (#3654)
Sep 3, 2020
Makefile
chore(ci): Refine testing for needed env vars for datadog-agent integ…
Feb 26, 2022
NOTICE
chore: Remove more references to timberio (#11111)
Feb 9, 2022
PRIVACY.md
chore: Adjust privacy policy language (#5276)
Nov 28, 2020
README.md
chore: Remove more references to timberio (#11111)
Feb 9, 2022
RELEASES.md
chore: Remove more references to timberio (#11111)
Feb 9, 2022
SECURITY.md
chore: Remove more references to timberio (#11111)
Feb 9, 2022
VERSIONING.md
chore: Remove more references to timberio (#11111)
Feb 9, 2022
build.rs
enhancement(datadog_metrics sink): rewrite to the new model + add ske…
Oct 29, 2021
clippy.toml
chore: Attempt to enable disallowed-methods clippy lint (#9013)
Sep 3, 2021
deny.toml
chore: Remove more references to timberio (#11111)
Feb 9, 2022
kustomization.yaml
Update the path to the base at the skaffold-related kustomization.yaml (
Nov 25, 2020
netlify.toml
chore: Fix url for discussions redirect (#10683)
Jan 4, 2022
rust-toolchain.toml
chore(deps): Upgrade Rust to 1.58.1 (#10941)
Jan 20, 2022
skaffold.yaml
chore(dev): Get ./scripts/skaffold.sh working on OSX (#8022)
Jun 24, 2021
What is Vector? Principles Use cases Community Documentation About Setup Reference Administration Resources Comparisons Performance Correctness Features

README.md

Quickstart  •   Docs  •   Guides  •   Integrations  •   Chat  •   Download

Vector

What is Vector?

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. Collect, transform, and route all your logs, metrics, and traces to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.

To get started, follow our quickstart guide or install Vector.

Principles

  • Reliable - Built in Rust, Vector's primary design goal is reliability.
  • End-to-end - Deploys as an agent or aggregator. Vector is a complete platform.
  • Unified - Logs, metrics, and traces (coming soon). One tool for all of your data.

Use cases

  • Reduce total observability costs.
  • Transition vendors without disrupting workflows.
  • Enhance data quality and improve insights.
  • Consolidate agents and eliminate agent fatigue.
  • Improve overall observability performance and reliability.

Community

  • Vector is relied on by startups and enterprises like Atlassian, T-Mobile, Comcast, Zendesk, Discord, Fastly, CVS, Trivago, Tuple, Douban, Visa, Mambu, Blockfi, Claranet, Instacart, Forcepoint, and many more.
  • Vector is downloaded over 100,000 times per day.
  • Vector's largest user processes over 30TB daily.
  • Vector has over 100 contributors and growing.

Documentation

About

Setup

Reference

Administration

Resources

Comparisons

Performance

The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.

Test Vector Filebeat FluentBit FluentD Logstash SplunkUF SplunkHF
TCP to Blackhole 86mib/s n/a 64.4mib/s 27.7mib/s 40.6mib/s n/a n/a
File to TCP 76.7mib/s 7.8mib/s 35mib/s 26.1mib/s 3.1mib/s 40.1mib/s 39mib/s
Regex Parsing 13.2mib/s n/a 20.5mib/s 2.6mib/s 4.6mib/s n/a 7.8mib/s
TCP to HTTP 26.7mib/s n/a 19.6mib/s <1mib/s 2.7mib/s n/a n/a
TCP to TCP 69.9mib/s 5mib/s 67.1mib/s 3.9mib/s 10mib/s 70.4mib/s 7.6mib/s

To learn more about our performance tests, please see the Vector test harness.

Correctness

The following correctness tests are not exhaustive, but they demonstrate fundamental differences in quality and attention to detail:

Test Vector Filebeat FluentBit FluentD Logstash Splunk UF Splunk HF
Disk Buffer Persistence ✓ ✓ ⚠ ✓ ✓
File Rotate (create) ✓ ✓ ✓ ✓ ✓ ✓ ✓
File Rotate (copytruncate) ✓ ✓ ✓
File Truncation ✓ ✓ ✓ ✓ ✓ ✓ ✓
Process (SIGHUP) ✓ ⚠ ✓ ✓
JSON (wrapped) ✓ ✓ ✓ ✓ ✓ ✓ ✓

To learn more about our correctness tests, please see the Vector test harness.

Features

Vector is an end-to-end, unified, open data platform.

Vector Beats Fluentbit Fluentd Logstash Splunk UF Splunk HF
End-to-end ✓
Agent ✓ ✓ ✓ ✓
Aggregator ✓ ✓ ✓ ✓
Unified ✓
Logs ✓ ✓ ✓ ✓ ✓ ✓ ✓
Metrics ✓ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠
Traces 🚧
Open ✓ ✓ ✓
Open-source ✓ ✓ ✓ ✓ ✓
Vendor-neutral ✓ ✓ ✓
Reliability ✓
Memory-safe ✓
Delivery guarantees ✓ ✓ ✓
Multi-core ✓ ✓ ✓ ✓ ✓ ✓ ✓

⚠ = Not interoperable, metrics are represented as structured logs

Developed with ❤️ by Datadog - Security Policy - Privacy Policy

About

A high-performance observability data pipeline.

vector.dev

Topics

rust parser events router pipeline metrics vector logs stream-processing forwarder observability

Resources

Readme

License

MPL-2.0 License

Code of conduct

Code of conduct

Stars

9.3k stars

Watchers

110 watching

Forks

718 forks

Releases 38

v0.20.0 Latest
Feb 11, 2022
+ 37 releases

Contributors 201

+ 190 contributors

Languages