Announcing the Sentry API

Today we’re excited to introduce the Sentry API, which will help ops teams and developers integrate Sentry deeper into their workflows.

Although the API is in its early stages, it’s expanding rapidly. Here are a few things you can do right off the bat:

  • Retrieve/update an individual event
  • Retrieve/update/delete a group of events
  • List all events in a group
  • List all groups in a project
  • Bulk mutate a list of groups

For example, here’s how you might resolve everything after a deploy:

PUT /api/0/projects/{project_id}/groups/
  "status": "resolved"

For more examples, view the full API Documentation.

The Sentry API is the starting point for many things to come. We’re excited to see what you do with it.

Introducing Organizations

Historically, managing org structure, membership, and billing in Sentry has been a pain. Today we’re announcing several giant steps towards making this better.

The most significant of these changes is the introduction of Organizations. Organizations are a new parent entity that will encapsulate Teams and Projects. Organizations will now be where you manage membership and billing. We’ve also included a couple of new tools to increase visibility across your organization.

Billing and Membership

Billing and membership is now managed at the organization level. Teams across an organization will share a quota. This means admins can create new teams without having to setup an additional subscription.

Stats and Audit Log

To make it easier to see usage across your organization, we’ve included a basic Stats page:


We’ve also added a simple Audit Log to highlight important actions taken across your organization:

Audit Log

Simple member management

We’ve made a lot of progress towards simplifying our membership model. Some major changes:

Phasing out Access Groups in favor of Teams

Now that the complexities of Teams have been moved to the organization level, Access Groups no longer make sense. Although Access Groups will continue to function as before, you will no longer be able to create or modify them. Teams will replace the role of Access Groups from this point forward.

We’ve provided an Access Group Migration Tool to make this a snap.

No more Team Owner

There’s no longer a concept of ‘Team Owner.’ This is now handled on the organization level.

Action items

  • Rename your Organization - We generated these based on your team names and that is probably not ideal for everyone.
  • Rename your Teams - Teams no longer share a global namespace. Rename them to whatever you want.
  • Be aware of broken links - We unfortunately had to break most pre-existing URLs as part of this change. This means links in emails too. Sorry :(
  • Migrate your Access Groups to Teams

We’d be happy to assist you with any of these. Let us know if we can help.

State of the union: JavaScript Exceptions and Stacktraces

Watch our very own Matt Robenolt speak about the current state of javascript exceptions and stacktraces at this year’s JSConf.

OpenSSL Vulnerability and SSL Certificate Changes

You’ve likely already heard about the recent OpenSSL vulnerability. We have no indication that any of our systems were compromised, and we’ve taken the best measures we can to ensure integrity throughout. As always, it’s worth noting that it’s never a bad idea to cycle your passwords.

As part of our security updates, we will be revoking our current SSL certificate. This has been a long time coming, and this has just pressed the issue. Unfortunately, this means that anyone who had to update their keychains with our SSL certificate will have to go through the process again. While this doesn’t affect all users, it’s important to be aware of this transition.

Primarily, if you’re a user of raven-java, or you’ve explicitly enabled ssl verification in any other clients, you’ll want to ensure that our new certificate is registered on your servers before April 14th when we do the cutover.

For more information, as well as links to obtain the new certificates, visit our SSL documentation.

CCP Games brings Sentry to the PlayStation

Dust 514 - CCP Games

CCP Games, the game development powerhouse behind Eve Online and Dust 514, has brought Sentry to the PS3. As they’re one of our largest and most unique customers, we decided to sit down with CCP Game’s Kristján Valur Jónsson to find out more about how his team utilizes Sentry.

Sentry: Kristján, thanks for taking the time to speak with us. First of all, why did you choose to start using something like Sentry over traditional logging?

Kristján: We needed to re-visit client logging for our latest product, Dust 514. While games like EVE use a home-grown logging solution, the last 10 years have seen tremendous progress in off the shelf infrastructure. We wanted to use a tried-and-tested stand-alone solution. We were impressed by the detailed stacktrace information available through sentry and didn’t want to waste manpower re-inventing that wheel. We also wanted it to be as independent of the rest of our network infrastructure as possible, so that for example errors could continue to be logged even if the regular client-server connection were to be interrupted. Having a hosted service helps with this. The hosted service also brings the huge benefit of not having to requisition server resources for logging, and frees our infrastructure department from worrying about that part.

Sentry: How large is your organization? How many are using Sentry?

Kristján: CCP is about 600 people strong these days. The part chiefly involved in Dust 514 is about 110 people. thereof about 30 engineers and 20 QA people. Most of QA knows Sentry, and a healthy portion of our engineers are aware of it as well.

CCP Games
CCP Games offices in Reykjavik

Sentry: What are some of the unique challenges that your team faces?

Kristján: All of our games are constantly being updated. This is true for EVE Online, as well as Dust 514. In the latter case, we are currently pushing out client updates every few weeks. This necessarily makes efficient QA vitaly important and thus it is imperative that we continue to monitor the health of our game clients during actual use. This makes logging from clients an essential part of our business. Also, the reality of the internet and the complexity of modern web-based server architecture means that some problems don’t occur within a testing environment, how realistic as it may be. So, we need to be constantly aware of any problems that the users are experiencing and improving our software appropriately.

Sentry: Which pieces of Sentry do you find most useful throughout the day?

Kristján: We primarily use Sentry to log and analyze Python tracebacks. Our QA teams watch the event stream closely and notice if any new problems crop up. Since the events are then uniquely available, we can augment our internal defect tracking with Sentry URLs and our developers can then use the detailed traceback information study the problems.

Sentry: Are things different at CCP after utilizing Sentry?

Kristján: It has reduced our reliance on log files considerably. And since we are using a hosted service, the server is always on. QA now monitors the Sentry server and defects in our defect tracking system cross-reference the events there. This means that often actual client problems don’t need to be reproduced, but can be diagnosed from the Sentry events directly. We can also use Sentry to validate code fixes that would otherwise be hard to verify through defect reproduction, since we can simply monitor the event feed and watch them disappear.

Sentry: It’s clear that CCP uses Sentry in a unique and interesting way. Can you tell us a little about your implementation?

Kristján: Perhaps the single most unusual bit is that in addition to our battle-servers, we are logging from player’s own client machines. There are hundreds of thousands of these, scattered around the globe, so identifiers such as hostnames and ip-addresses become meaningless. I’m guessing that Sentry was never really intended for that but rather for server-side logging, but we weren’t aware of that initially. This hasn’t been a problem, however and Sentry is able to cope beautifully.

The client implementation is unusual as well. We are uniquely using Stackless Python to drive important parts of the game logic on a Sony PS3 console. On this generation of hardware we are very constrained by memory. We have a custom version of Python tuned to use as little of this precious resource as possible. For this reason, we could not take, for example, the Raven library and use it as-is. To save program memory we carved out its essential parts into something we call Krunk, the Icelandic name for the raven’s call. Performance is also important, since error reporting can not be allowed to noticeably affect frame rate. So, we are using the ultrajson JSON encoder, which is written in C. But this didn’t allow the custom hooks required by Raven, i.e. for UUIDs and timestamps. Since we are using Stackless Python, we also use a Stackless transport to send each event on a separate tasklet, so as not to block the main execution loop.

Another interesting fact is that we can dynamically change the activation state of Sentry logging on each client. We can choose the percentage of the clients that have it enabled, and also which logging channels are connected and on what level. This happens via the game server and allows us to throttle logging at the source and focus logging on certain areas of the code.

Sentry: In closing, could you summarize your experience with Sentry for developers who are considering giving it a try?

Kristján: Using Sentry has been a simple and intuitive experience and support has been virtually instant for any questions that have come up.

Bonus As a gift to our readers, Kristján and his team have provided 20 PlayStation 3 voucher codes for the Dust 514 Origin Pack:

  • 7LL6-6JN8-DT7K
  • NJ58-G4N6-P89R
  • 5G7B-8RNF-KGDG
  • F9HT-H8N6-A7ER
  • L4FJ-PHN9-Q6L3
  • P6EQ-CGN4-T3KJ
  • 296B-BGNT-8BHP
  • 3KLG-PLND-P58B
  • H938-GPN6-QCLH
  • 2AJ7-ALNH-H8HR
  • CDT5-FKN6-2KFD
  • JR56-PPNB-AM5B
  • CG55-NBNG-Q7FP
  • DR5P-G3NP-9K4N
  • F687-T3N2-BQHD