You’ve likely already heard about the recent OpenSSL vulnerability. We have no indication that any of our systems were compromised, and we’ve taken the best measures we can to ensure integrity throughout. As always, it’s worth noting that it’s never a bad idea to cycle your passwords.
As part of our security updates, we will be revoking our current SSL certificate. This has been a long time coming, and this has just pressed the issue. Unfortunately, this means that anyone who had to update their keychains with our SSL certificate will have to go through the process again. While this doesn’t affect all users, it’s important to be aware of this transition.
Primarily, if you’re a user of raven-java, or you’ve explicitly enabled ssl verification in any other clients, you’ll want to ensure that our new certificate is registered on your servers before April 14th when we do the cutover.
For more information, as well as links to obtain the new certificates, visit our SSL documentation.
CCP Games, the game development powerhouse behind Eve Online and Dust 514, has brought Sentry to the PS3. As they’re one of our largest and most unique customers, we decided to sit down with CCP Game’s Kristján Valur Jónsson to find out more about how his team utilizes Sentry.
Sentry: Kristján, thanks for taking the time to speak with us. First of all, why did you choose to start using something like Sentry over traditional logging?
Kristján: We needed to re-visit client logging for our latest product, Dust 514. While games like EVE use a home-grown logging solution, the last 10 years have seen tremendous progress in off the shelf infrastructure. We wanted to use a tried-and-tested stand-alone solution. We were impressed by the detailed stacktrace information available through sentry and didn’t want to waste manpower re-inventing that wheel. We also wanted it to be as independent of the rest of our network infrastructure as possible, so that for example errors could continue to be logged even if the regular client-server connection were to be interrupted. Having a hosted service helps with this. The hosted service also brings the huge benefit of not having to requisition server resources for logging, and frees our infrastructure department from worrying about that part.
Sentry: How large is your organization? How many are using Sentry?
Kristján: CCP is about 600 people strong these days. The part chiefly involved in Dust 514 is about 110 people. thereof about 30 engineers and 20 QA people. Most of QA knows Sentry, and a healthy portion of our engineers are aware of it as well.
CCP Games offices in Reykjavik
Sentry: What are some of the unique challenges that your team faces?
Kristján: All of our games are constantly being updated. This is true for EVE Online, as well as Dust 514. In the latter case, we are currently pushing out client updates every few weeks. This necessarily makes efficient QA vitaly important and thus it is imperative that we continue to monitor the health of our game clients during actual use. This makes logging from clients an essential part of our business. Also, the reality of the internet and the complexity of modern web-based server architecture means that some problems don’t occur within a testing environment, how realistic as it may be. So, we need to be constantly aware of any problems that the users are experiencing and improving our software appropriately.
Sentry: Which pieces of Sentry do you find most useful throughout the day?
Kristján: We primarily use Sentry to log and analyze Python tracebacks. Our QA teams watch the event stream closely and notice if any new problems crop up. Since the events are then uniquely available, we can augment our internal defect tracking with Sentry URLs and our developers can then use the detailed traceback information study the problems.
Sentry: Are things different at CCP after utilizing Sentry?
Kristján: It has reduced our reliance on log files considerably. And since we are using a hosted service, the server is always on. QA now monitors the Sentry server and defects in our defect tracking system cross-reference the events there. This means that often actual client problems don’t need to be reproduced, but can be diagnosed from the Sentry events directly. We can also use Sentry to validate code fixes that would otherwise be hard to verify through defect reproduction, since we can simply monitor the event feed and watch them disappear.
Sentry: It’s clear that CCP uses Sentry in a unique and interesting way. Can you tell us a little about your implementation?
Kristján: Perhaps the single most unusual bit is that in addition to our battle-servers, we are logging from player’s own client machines. There are hundreds of thousands of these, scattered around the globe, so identifiers such as hostnames and ip-addresses become meaningless. I’m guessing that Sentry was never really intended for that but rather for server-side logging, but we weren’t aware of that initially. This hasn’t been a problem, however and Sentry is able to cope beautifully.
The client implementation is unusual as well. We are uniquely using Stackless Python to drive important parts of the game logic on a Sony PS3 console. On this generation of hardware we are very constrained by memory. We have a custom version of Python tuned to use as little of this precious resource as possible. For this reason, we could not take, for example, the Raven library and use it as-is. To save program memory we carved out its essential parts into something we call Krunk, the Icelandic name for the raven’s call. Performance is also important, since error reporting can not be allowed to noticeably affect frame rate. So, we are using the ultrajson JSON encoder, which is written in C. But this didn’t allow the custom hooks required by Raven, i.e. for UUIDs and timestamps. Since we are using Stackless Python, we also use a Stackless transport to send each event on a separate tasklet, so as not to block the main execution loop.
Another interesting fact is that we can dynamically change the activation state of Sentry logging on each client. We can choose the percentage of the clients that have it enabled, and also which logging channels are connected and on what level. This happens via the game server and allows us to throttle logging at the source and focus logging on certain areas of the code.
Sentry: In closing, could you summarize your experience with Sentry for developers who are considering giving it a try?
Kristján: Using Sentry has been a simple and intuitive experience and support has been virtually instant for any questions that have come up.
Bonus As a gift to our readers, Kristján and his team have provided 20 PlayStation 3 voucher codes for the Dust 514 Origin Pack:
We recently sat down with Yonas Beshawred, founder of Leanstack.io, and talked about how Sentry came to be.
There were no filters, no management, for better or worse. Just me and David. It’s a scary and exciting freedom….That’s why open source projects have always interested me. You get to play every role you want. You succeed or fail purely based on your decisions.
Read the entire interview over on the leanstack blog.
Disqus Team at PyCon 2013 by Ed Schipul
Since Sentry’s conception at Disqus in early 2010, its use inside has grown considerably. The entire Disqus engineering team (16 people and counting) use Sentry in one way or another, deploying 20 Sentry instances across a wide range of services.
A little history: As Disqus started taking off in 2009 (by 2010 it would be handling 200M unique visitors), traditional logging became problematic. A need existed for real aggregation, but also the power of introspection. In early 2010, Sentry was conceived to solve exception logging (primarily for Django, as it’s what powered Disqus). Today, Disqus is the #1 online discussion and commenting service, serving 1.1 billion unique monthly visitors at 2.5 million sites. While it’s average is only 1-2 events per second in Sentry, at times there are spikes of up to 500 errors a second.
On the operations side, Sentry is used primarily as an early warning system, allowing for rapid response to issues as they arise. Disqus engineer Mike Clarke explains, “As we ship code, which we do at least daily, I’m constantly staring at Sentry. Watching the trends to see if something new is broken versus something that’s been broken for a while.”
As we ship code, which we do at least daily, I’m constantly staring at Sentry.
On the development side, Sentry helps prioritize resources. “When we’re building stuff, there’s always going to be errors. It helps to find out how frequently those errors are occurring to prioritize which things get tackled first. Being able to look at Sentry over a time period, or just overall, tells us what’s really most impactful to the user”, explains developer Ted Kaemming.
Sentry is also used by support. When a user gets an error, they’re given a hash they can include in a support email. This hash gives the support person a clue as to what might be broken, and who in engineering to talk to. If the email becomes a support ticket for engineering, engineering can then use Sentry to debug, using the search and lookup features (among others) to examine the specifics of the error.
Being able to look at Sentry over a time period, or just overall, tells us what’s really most impactful to the user.
Disqus also uses Sentry for diagnostic logging, using it to see how errors in one part of the system affect other systems. It logs warnings in addition to errors, and even uses Sentry for auditing and security tasks. “If you’re masquerading as another user doing super admin type things, we’ll log that to Sentry so you can follow if someone is unexpectedly doing bad things to somebody else. The grouping is so effective that when something out of the ordinary does happen, regardless of it’s an error or not, it’s a convenient way of visualizing it”, explains Mike Clarke.
In all, Disqus uses Sentry in a variety of places, more than 20 projects at this point. They report that setup is surprisingly easy for new projects, and adoption shows no signs of slowing. “Sentry is great in terms of visualizing stack traces and that sort of thing, but there’s many ways we can know when our system is broken. How Sentry consolidates that into a single location is what makes it truly useful for us”, explains Mike Clarke.