Blog

Welcome Ted Kaemming

We’re excited to announce that Ted Kaemming is joining the engineering team at Sentry.

Previously, Ted worked on infrastructure projects at Disqus, including Samsa (a Python client for Apache Kafka 0.7, which was later renamed to pykafka), a change capture and replication system for PostgreSQL, and other foundational systems. He also refuses to drink anything below a 90 on Beer Advocate.

Ted will be continuing to work on similar projects at Sentry, where he will be helping to improve the performance, reliability, and usefulness of both hosted and open-source Sentry.

Help us welcome Ted by giving him a shout on Twitter, or following him on GitHub.

Welcome Matt Robenolt

We’re excited to announce that Matt Robenolt is officially joining the engineering team at Sentry.

Matt is a long time member of the Sentry project and avid open source contributor. Five years ago, Matt started fixing bugs and contributing to numerous Raven clients. When we eventually launched the SaaS offering, Matt stuck around as a core contributor and general advisor. During the time, Matt led the operations team at Disqus and has given a number of talks about #webscale and JavaScript.

Matt will be owning the Sentry infrastructure as we grow and continue to push open source to the limits.

Help us welcome Matt by giving him a shout on Twitter or following him on GitHub.

Sentry 8 starts rolling out this week

Earlier this month we announced a brand new version of Sentry, “version 8”, which you can try today at beta.getsentry.com.

Besides a brand new look and feel, Sentry 8 introduces some powerful new features, including:

  • Release tracking
  • User assignment
  • Event merging
  • And much more …

Not only have we been dogfooding Sentry 8 for the past few months, but so have 20% of Sentry users, who’ve given us a steady supply of bug reports, feedback, and thumbs ups. We’ve been iterating rapidly over the past few weeks to get Sentry 8 ready for a wider release, and after hundreds of bug fixes and improvements, we feel that day has come.

Starting today, we are going to begin opting-in groups of Sentry organizations into Sentry 8.

We realize that there may still be edge cases we haven’t caught, so if you come across a critical issue that is interrupting your workflow, we’ve got good news – you will be able to temporarily opt-out of the new version. Our goal is to have everybody on Sentry 8 by the start of September.

That’s it! Keep an eye out for opt-in emails, and as always, feedback is greatly appreciated. Please don’t hesitate to contact us.

– The Sentry Team

rb: A Redis parallelization toolkit for Python

rb logo

We love Redis at Sentry. Since the early days it has driven many parts of our system, ranging from rate limiting and caching, as well as powering the entirety of our time series data storage. We have enough data in Redis that a single machine won’t cut it.

Previously we had been using the Nydus library to communicate with multiple independent Redis nodes. It is based on the idea that you can route individual requests to individual nodes and it largely automates this process for you. Tools like twemproxy achieve a similar result, but they do it via a proxy layer whereas Nydus does it at the application level (thus avoiding an infrastructure dependency).

One of the most useful features of Nydus is the ability to issue many requests in parallel. Given a large set of operations we want to execute them in the most efficient way possible. At a basic level that means parallelizing operations. Nydus does this using Python’s internal threading and a simple worker pool.

Executing Operations in Parallel

Sentry exposes numerous API endpoints where we fetch hundreds of Redis keys from different nodes. To do that you run some code akin to:

def look_up_keys(keys):
    results = []
    for key in keys:
        client_for_host = get_host_for_key(key)
        results.append(client_for_host.get(key))
    return results

Now the problem here is that you’re doing a single request per key. At the very least you’d want to pipeline all requests that go to the same node. Even better would be if you could fetch from different hosts in parallel. Nydus does that and more for you in a very simple abstraction:

def look_up_keys(keys):
    results = []
    # when the context manager exits the commands are executed
    # and the resulting object (a sort-of proxy to a promise) gets
    # assigned the result
    with cluster.map() as conn:
        for key in keys:
            results.append(conn.get(key))
    return results

What we didn’t expect was how much slower these lookups became after increasing the node count. Nydus internally uses threads for concurrency to achieve parallel data fetching and that turns out to be problematic in Python. There’s a number of problems here, but they all boil down to performance. In our case we consistently saw locking as being the number one bottleneck. This is to be expected as the way a threaded worker pool operates is by using a thread-safe queue, thus consistently taking out locks.

Changing Nydus to not use threads wasn’t an easy task because it doesn’t just support Redis, but also other databases like Cassandra. It’s a very high level library that aims to provide basic cluster management so a lot of the code is generalized to support other connections. Given this it becomes very hard to support changing the threading abstraction as underlying connection libraries function differently.

A Better Approach

After determining that the threading (and resulting locks) were our biggest issue we decided to create a new library which would take a better approach. The new library we wrote, rb (Redis Blaster) only works with Redis, but that offers us a few advantages.

We switched the parallel query implementation to use a select loop (select/poll/kqueue). This allows us to reuse the entire connection and client system from the Python Redis library. It also means no thread spawning, no locking, and much higher performance.

Additionally the routing layer was overhauled to support more correct behavior. Because rb knows about the structure of all Redis commands, it correctly knows which parameters are keys and which are not. This ended up pointing out some potentially bad uses of Nydus in our codebase. Now if you try to issue a command like MGET (to get multiple keys) and you pass more than one key, the library will error out instead of producing garbage.

Nydus is dead. Long live rb.

Improving Routing

The most common operation is to work with commands that operate on keys which get routed to one of the many nodes. For this the map() function comes in handy which works similar to the one in Nydus:

results = []
with cluster.map() as client:
    for key in range(100):
        results.append(client.get(key))

print 'Sum: %s' % sum(int(x.value or 0) for x in results)

Something else we found ourselves doing frequently was running commands across all hosts. This was more complex in Nydus as you effectively needed to iterate each host and run the command on it. With rb you can do that through the all() API. For instance to delete all Redis server’s databases you can do this:

# a common pattern for our testsuite fixtures
with cluster.all() as client:
    client.flushdb()

Simplifying Promises

As part of the process we decided to also clean up the API. We use the same ideas as Nydus, but use an explicit promise object instead of the proxy. This is both more explicit but also less error prone, and once again gives us a slight performance boost. Nydus used to use proxy objects that would eventually resolve to the value that comes back from Redis but this required being careful when to use it, and to type convert it before passing it on to other APIs that might be either confused by the promise or that would hold on to it and create a small memory leak.

The promises are heavily inspired by the ES6 implementation, with a slightly modified API to make it work better in Python. While the lack of anonymous block functions makes Promises a little less elegant in Python, for our use cases simple lambda functions can go a long way.

In the end, the API is very easy to reason about:

results = []
with cluster.map() as client:
    for key in range(100):
        client.get(key).then(lambda x: results.append(int(x or 0)))

In Production

We set off on this adventure to improve the response time on a very specific endpoint. Once we got it out there we were pleasantly surprised to see noticeable impact on other pages, such as our store API endpoint (the majority of Sentry’s requests):

8ms improvement

The real kicker here is that this endpoint does far fewer operations than many of our others, but we still managed to get a very measurable performance boost out of the change.

Get it on Github

rb is available for general consumption under the Apache 2 License. Documentation can be found on rb.readthedocs.org.

We would love to get your feedback and hope it’s as useful to you as to us.

Take Part in the Sentry 8 Beta

Sentry 8.0 Beta Preview

Nearly 18 months ago we began exploring a brand new look for Sentry. Around the same time we also decided to modernize Sentry’s frontend. After many iterations on the technology and the design, we’re happy to finally be able to share it with you.

Today we’re opening up the beta to the general public.

If you’re a Sentry customer, you’ll be able to access the new Sentry at beta.getsentry.com. This is an early release candidate for version 8 of our platform and is made up of more than 2,000 individual commits. We plan to run the beta for at least the next two weeks, and will be actively finishing up minor items and listening to your feedback.

What’s new?

From a high level, there’s a few major things you’ll notice almost immediately:

  • A brand new design. Most things will feel similar, though you’ll see some significant changes on pages like Stream and Event Details.
  • Faster rendering. Many pages are now rendered via React and simply communicate with our web API.
  • Release Tracking. Let Sentry know about the version of your application and many magical things will happen. We’ll be greatly expanding on support in this area in the future.
  • Assignment. While many of us use external issue trackers, sometimes duplicating effort isn’t worth it. We’ve added basic assign-to-user workflow to Sentry to help with that situation.
  • Event Merging. On the stream you’ll find a lot of added power around bulk actions. One of the newly exposed features allows you to merge event groups.
  • … and much more!

We’ll be talking more about these features in the coming weeks, but as part of the beta we want you to begin exploring for yourself. It’s important to us that things feel obvious and aren’t confusing.

We’re also not done yet. We’ll be moving faster than you’ve ever seen before, and things will be improving and changing very quickly.

Caveats

As this is still a beta, we want to you to know that we’re still actively working on things. While everything is generally stable, you’ll find various display issues as well as pages that may not seem fast enough. We’re working on this, and you can track our progress via GitHub.

You also shouldn’t send data to the beta site (i.e. don’t configure your clients with a beta DSN). You also shouldn’t configure release hooks or any other service to communicate with the beta site. These endpoints are generally available in production (by swapping out beta for app), but if you have questions we’re happy to answer them.

Feedback

We’d love if you’d provide us feedback about the new release. A few key areas to explore:

  • the new Stream, as it’s quite a bit different
  • the Group details, and changes to how events function (overview vs individual event)
  • Release tracking — is there integration you’d like to see supported?

The preferred place for feedback would be our public issue tracker on GitHub. Just create an issue akin to “David Cramer’s Feedback on v8” and throw your comments and concerns in there.

Alternatively if you’re not comfortably sharing publicly, you can always drop us a line via hello@getsentry.com.