It’s been just over two years since we open sourced Scalding and today we are very excited to release the 0.9 version. Scalding at Twitter powers everything from internal and external facing dashboards, to custom relevance and ad targeting algorithms, including many graph algorithms such as PageRank, approximate user cosine similarity and many more.

There have been a wide breadth of new features added to Scalding since the last release:Read more…

Finagle is our fault tolerant, protocol-agnostic RPC framework built atop Netty. Twitter’s core services are built on Finagle, from backends serving user profile information, Tweets, and timelines to front end API endpoints handling HTTP requests.Read more…

As Twitter has moved from a monolithic to a distributed architecture, our scalability has increased dramatically.Read more…

Today we are open sourcing Summingbird on GitHub under the ALv2.Read more…

Last summer, we open-sourced Zipkin, a distributed tracing system that helps us gather timing and dependency data for the many services involved in managing requests to Twitter. As we continually improve Zipkin, today we’re adding a Firefox extension to Zipkin that makes it easy to see trace visualizations in your browser as you navigate your website.Read more…

Earlier this year we open sourced Scalding, a Scala API for Cascading that makes it easy to write big data jobs in a syntax that’s simple and concise. We use Scalding heavily — for everything from custom ad targeting algorithms to PageRank on the Twitter graph.Read more…

Iago is a load generator that we created to help us test services before they encounter production traffic. While there are many load generators available in the open source and commercial software worlds, Iago provides us with capabilities that are uniquely suited for Twitter’s environment and the precise degree to which we need to test our services. There are three main properties that make Iago a good fit for Twitter:Read more…

Zipkin is a distributed tracing system that we created to help us gather timing data for all the disparate services involved in managing a request to the Twitter API. As an analogy, think of it as a performance profiler, like Firebug, but tailored for a website backend instead of a browser. In short, it makes Twitter faster.Read more…

Finagle is a protocol-agnostic, asynchronous RPC system for the JVM that makes it easy to build robust clients and servers in Java, Scala, or any JVM-hosted language.Read more…