Manage Your Error Quota
Sending all your application errors to Sentry ensures you'll be notified in real-time when errors occur in your code. However, with a basic setup, you may consume your error
- Receive more actionable and meaningful error events.
- Reserve real-time notifications for errors that actually break your code.
- Manage your error quota.
Applying the proper filters, SDK configuration, and rate limits is an iterative and on-going process, but these tips will show you how to get the most out of your error events. These tips are ordered from easiest or least time-consuming to most challenging or potentially time-consuming. Options 1-5 are all things you can do in sentry.io, whereas the remaining ones are things you'll do in the SDK.
Before You Begin: Check Your Quota Usage
You can look at your events in aggregate in the "Usage Stats" tab of Stats. This information will help you answer key questions about the breakdown of your incoming events or which projects are consuming your
This page is accessible to all members of your organization, so Owners in your Sentry org can share this page with the developers directly responsible for a given project. Also, you can come back to this page to check if the changes you've made are having the desired effect.
How can I see a breakdown of incoming events?
The Usage Stats tab displays details about the total amount of data Sentry has received across your entire organization for up to 90 days. The page breaks down the events (by project) into three categories: accepted, dropped, or filtered. Only accepted events affect your
What are my busiest projects?
The "Project" table in the "Usage Stats" tab of Stats breaks down your data by project, so you can see which ones are consuming your
Which issues are consuming my quota?
You can only update the columns of the Results table in Discover if your organization is on a Business plan.
You can set up a query in Discover to take a more proactive approach to resolving your busiest issues. When you're building the query, search for event.type:error
and then set the columns issue
, title
, project
, and count()
, as shown below:
Once the changes are applied, sort the "Results" table by the "COUNT()" column to display your busiest issues:
Once you know where your
Grace Period
All accounts have access to a one-time grace period. This means that if an account depletes its error event volume and has never previously triggered a grace period, we continue to accept events for the next three days at a maximum rate of 10,000 events per hour. If you choose not to increase your error volume, Sentry will stop accepting events for the remainder of the current billing month. However, if you increase your volume, error events accepted during the grace period will be counted as consumed volume.
While this grace period isn't dependent on the Spike Protection feature, you'll probably get a notification that you've entered a grace period along with a notification of a spike.
Spike Protection
A spike is both a large and temporary increase in error event volume. Sentry applies a dynamic rate limit to your account designed to protect you from short-term spikes.
Spike protection is enabled for your organization by default, and when it's enabled, Sentry continually monitors for spikes. You can confirm that it's enabled in Settings > Subscription. This page can only be accessed by a Billing or Owner member of your Sentry organization.
How Does Spike Protection Work?
Because Sentry bills based on monthly event volume, spikes can easily consume your
We use your historical error event volume to implement a dynamic rate limit, and then discard error events when you hit its threshold. When spike protection is activated, we limit the number of error events accepted in any minute to:
maximum(20, 6 x average error events per minute over the last 24 hours)
The 24-hour window ends at the beginning of the current hour, not at the current minute.
This means if you experience a spike, we'll temporarily protect you, but if the increase in volume is sustained, the spike protection limit will gradually increase until Sentry accepts all events.
For example, in the last 24 hours, your organization has been receiving, on average, 10 events per minute (after any inbound filters have been applied). That means your current per-minute limit is 6 * 10 or 60 events. There have been no spikes in that time, so spike control is "inactive". Something breaks, and in the next minute, your organization sends Sentry 100 events. When we see the 61st event, three things happen:
Spike protection becomes "active".
If your organization has used more than 25% of your monthly quota, Sentry sends the organization owner a notification including which project the 61st event came from. This is likely, but not guaranteed to be, the project in which something broke.
Events 61-100 are dropped with a 429 error code.
When the next minute begins, we again record up to 60 events, dropping the rest until the minute is up. It continues like this for the remainder of the hour. After an hour, we re-calculate your per-minute limit, and for the next hour use that limit to decide whether events get dropped each minute. We repeat this process every hour until Sentry eventually accepts all events.
If, instead of a single big spike (or an overall, permanent, increase in traffic), you experience many small spikes, there may be many days in a row when at least a few events are dropped. In that scenario, spike protection remains active the entire time.
Spike protection is an organization-level setting, so once it's triggered, it affects all the projects in the organization, regardless of which project triggered the spike.
Managing a Spike
If spike protection has been triggered for your account, you'll receive an email notifying you:
To manage the current spike and avoid future spikes, we recommend taking the following steps:
- Set up an on-demand budget to ensure you have time to adjust your volume in the event of a future spike in errors.
- Set better rate limits on the DSN keys for the projects related to the spike.
- If it's a specific release version that has caused the spike, add the version identifier to the project's inbound filters to avoid accepting events from that release.
To review the error events dropped because of spike protection, go to the "Usage Stats" tab of Stats for your Sentry org and select "Errors" in the "Category" dropdown.
When Does Spike Protection Become "Inactive?"
Events will not be dropped during any minute in which you don't send more than the hourly limit that Sentry has calculated for you. After 24 hours without any dropped events, spike protection becomes "inactive" again. This means that it is no longer dropping events, but it does not mean the system has stopped paying attention. The next time events are dropped, spike protection will be "reactivated".
New Spike Protection Calculations
This feature is in beta and is only available if your organization is participating in its limited release. Features in beta are still in-progress and may have bugs. We recognize the irony.
Limited availability spike protection is a project-level tool that helps prevent
Our spike protection algorithm does the following:
- Uses a weighted average of your events over the past 168 hours (seven days)
- Applies a multiplier to that number
- Compares this final number against a minimum number of events, determined using your quota, to trigger a spike
- Sets this as your spike limit
Setting the Spike Limit
There are two ways that we can set your spike limit, or the number of events that trigger a spike:
- Minimum Event Calculation - A calculation that determines a minimum number of events
- Usage-Based Calculation - A projection based on your past usage
The spike limit for each hour is set using either the minimum event or usage-based calculation — whichever is higher. This is done for a number of reasons. Firstly, using a minimum event calculation protects smaller or new projects. New projects that don't have a week’s worth of data to use to calibrate spike limits can use this minimum number of events, an adaptation of the organization’s
Spike limits are recalculated in real time throughout the duration of the spike to adjust for the increasing volume of incoming events. This allows the limit to grow at a steady rate such that quota is protected from being quickly consumed. An example of how this works during a spike is shown below.
Minimum Event Calculation
This calculation, which is the first step of our algorithm, identifies a minimum number of events, using your
(3 \* your quota)/(720 \* number of projects)
. The equation represents your project using up three times your overall quota in 30 days if events are continually ingested at this hourly rate, thus flagging the project for a potential spike.Usage-Based Calculation
This calculation, which is the second step of our algorithm, calculates hourly data from the past seven days to determine spike limit projections for the next seven days. This data is used to calculate weighted averages, which takes into account weekly and hourly seasonality. For example, the weighted average calculated for Monday at 3 pm is more heavily influenced by data points on Monday or the hours around 3 pm. This weighted average is then multiplied by a multiplier that is 5
times the overall standard deviation of the past week — this multiplier is bounded between 3
and 6
.
Example
In this example, the project usually ingests 100-200 events per hour. There's been a spike that’s reached 50,000 events, as shown in the graph below:
In the following graph, we can see a zoomed in perspective of the 12-hour period of the spike, along with a line indicating the spike limit as it’s being recalculated over the course of the spike:
Throughout the spike, the recalulating limit has the following effect:
- 1st hour: 6k events ingested, limit is recalculated to 2083, 3917 events dropped
- 2nd hour: 34k events ingested, limit is recalculated to 2873, 31217 events dropped
- 3rd hour: 55k events ingested, limit is recalculated to 5452, ~49k events dropped
- 4th hour: 49k events ingested, limit is recalculated to 7628, ~41k events dropped
- 5th hour: 41k events ingested, limit is recalculated to 9371, ~31k events dropped
For this particular example:
- Org quotaThe monthly number of events that you pay Sentry to track.: 500k
- Events ingested during the spike: ~478k
- Events accepted overall: ~157k
Here's an example of spike limit projections for a week, taking into account seasonality:
These regular differences in event ingestion don't cause a spike to occur.
Bursty Projects
There may be instances where a project routinely accepts a high volume of events in a very short period of time by design — for example projects that orchestrate cron/Airflow jobs or task runners. The screenshot below shows an example of this kind of behavior:
If this is expected behavior for a given project in your organization, you may want to consider turning off spike protection in the project settings to ensure necessary events aren't dropped.
Adjusting Quotas
Quotas can only be updated by a Billing or Owner member of your Sentry organization.
Once your event volume is approaching or has exceeded the
If this is your first time exceeding quota and you're on a paid plan, however, you'll be entered into a one-time grace period. Learn more about the grace period in this Help article.
Increasing Quotas
If you're dropping events that you want to keep because you've exceeded your
When you exceed your quota threshold, the server will respond with a 429 HTTP status code, which communicates to SDKs and clients to stop sending events. This status code comes with a Retry-After
header that indicates the time for which this rate limit is active. However, clients are not supposed to retry events, but instead drop events until the rate limit has expired, to prevent queue backlogs. Note, that since event ingestion and rate limiting happen asynchronously, the 429 HTTP status code is always slightly delayed.
To increase your quota, go to Settings > Subscription and click the "Manage Subscription" button to access your subscription options. When you increase your quota, the change goes into effect immediately.
If you're on a Developer plan and want to increase your quota, you'll need to upgrade to a Team or Business plan. On these plans you can prepay for more event volume and purchase on-demand volume, as needed. Learn about Sentry's plans on our pricing page.
If you upgrade from Team to Business mid-billing period, your on-demand pricing changes retroactively.
Reserved Volume
If the number of events you need is steadily increasing, you may want to increase your reserved capacity or volume. Reserved volume is less expensive than on-demand volume since you prepay for it. It also allows you to choose the number of events that you want to have available beforehand rather than just setting an arbitrary on-demand budget. Learn more about reserved volume in our pricing documentation.
You shouldn't increase your reserved volume if you think your need for more events is temporary, since reducing your reserved volume is tied to your billing period.
On-Demand Budget
If you need to increase your event
Decreasing Quotas
Plan downgrades and decreases in reserved volume are processed at the end of your billing period, and remaining volume cannot be refunded. For example, if you have a monthly billing period that starts on the 5th of the month, and you decrease your reserved volume on June 20th, then this change will be processed on July 4th. Your billing period beginning on July 5th will reflect your new reserved volume.
If you have an annual billing period, plan downgrades and decreases in reserved volume go into effect at the beginning of your next billing year.
Changes to on-demand volume typically go into effect immediately and are guaranteed to go into effect within 24 hours. However, you can't decrease your on-demand budget to less than what you've consumed in the current period.
To decrease your
We strongly recommend that you make subscription changes before the last day of your billing period. Depending on your time zone, in some cases, changes made on the last day of the billing period will not go into effect until the next billing period.
Rate Limiting
This feature is available only if your organization is on a Business or Enterprise plan.
Rate limiting allows you to set the maximum volume of error events a project key will accept during a period of time. For example, if you have a project in production that generates a lot of noise, a rate limit allows you to set the maximum amount of data, such as “500 events per minute”. Additionally, you can create a second key for the same project for your staging environment, which is unlimited, ensuring your QA process is still untouched.
In [Project] > Settings > Client Keys (DSN), click "Configure", and you can create multiple DSN keys per project and assign different (or no) rate limits to each key. This will allow you to dynamically allocate keys (with varying thresholds) depending on release, environment, and so on.
Once you've set the limit, events dropped because of rate limiting generate a 429 error code.
While rate limiting is quite useful for managing your monthly event
Setting Useful Rate Limits
A good way to set a project rate limit is by figuring out the expected event volume based on your average traffic. Here's how to do that:
- Go [Project] > Settings > Client Keys (DSN) and open the project DSN key configuration under by clicking "Configure".
- In the "KEY USAGE IN THE LAST 30 DAYS" graph, look for the highest point, or the maximum daily rate. In the example below, the maximum daily rate in the last month is less than 34K:
- Based on the rate, choose a daily maximum value or ceiling. In this example, we calculated a daily maximum of approximately 35K, which is around 1458 events an hour, or about 24 events a minute.
- Set a daily, hourly, or minute-based rate limit. We recommend using a minute-based rate to avoid situations where a random event spike might exhaust your daily or hourly quotaThe monthly number of events that you pay Sentry to track.and leave you without event data for a long period.
You should periodically go back and check the graph to see the number of events dropped due to rate limiting and, if needed, revisit your settings.
Review Event Repetition
The following rules apply for error event repetition and your
- If you previously resolved an issue, a new error event counts toward your quota because it may represent a regression in your code.
- If you have chosen to ignore alerts about error events with the same fingerprintThe set of characteristics that define an event., a new event counts toward your quota because the event is still occurring.
- If you "Delete & Discard" an issue, then future error events with the same fingerprint will not count toward your quota.
Delete & Discard
This feature is available only if your organization is on a Business or Enterprise plan.
If there is an irrelevant, reoccurring issue that you are unable or unwilling to resolve, you can delete and discard it from the Issue Details page by clicking "Delete and discard future events". This will remove the issue and event data from Sentry and filter out future matching events.
You can find a list of deleted and discarded issues in the "Discarded Issues" tab in [Project] > Settings > Inbound Filters. From here, you can un-discarded any of these issues to receive future events.
Once you've identified a set of discarded issues, it might make sense to go back to your SDK configuration and add the related errors into your beforeSend
client-side filtering.
Inbound Data Filters
While SDK configuration requires changes to your source code and depends on your next deployment, server-side filters can be easily configured per project in the "Data Filters" section of [Project] > Settings > Inbound Filters.
Once applied, you can track the filtered events (numbers and cause) using the graph provided at the top of the "Inbound Data Filters" page.
Sentry provides several methods to filter all events and attachments server-side, which are applied before checking for potential rate limits. Inbound filters include:
- Common browser extension errors
- Events coming from
localhost
- Known legacy browsers errors
- Known web crawlers
- Error messages
- Specific release versions of your code
- Certain IP addresses
The first four options above are settings that you simply toggle on or off. Setting the others is explained further below.
After these checks are processed, events and attachments that aren't dropped based on these filters count toward your
IP Filters
If you have a rogue client, Sentry supports blocking an IP from sending data. Navigate to [Project] > Settings > Inbound Filters to add the IP addresses (or subnets) in the "IP Addresses" field.
Filter by Release
This feature is available only if your organization is on a Business or Enterprise plan.
If you discover a problematic release causing excessive noise, Sentry supports ignoring all events and attachments from that release. Navigate to [Project] > Settings > Inbound Filters, then add the release to the "Releases" field.
Filter by Error Message
This feature is available only if your organization is on a Business or Enterprise plan.
Sentry supports filtering out a specific or certain kind of error as well. Navigate to [Project] > Settings > Inbound Filters, then add the error message to the "Error Message" field.
To ensure you’re adding the correct message to the inbound filter setting, check the JSON for an event in the issue. The filter by error message setting matches the data found in the title
field near the end of the file.
SDK Sample Rate
If a sample rate is defined for the SDK, the SDK evaluates whether this event should be sent as a representative fraction of events. Setting a sample rate is documented for each SDK.
The SDK sample rate is not dynamic; changing it requires re-deployment. Also, keep in mind that setting an SDK sample rate limits visibility into the source of events. Setting a rate limit for your project may better suit your needs.
SDK Filtering: beforeSend
All Sentry SDKs support the beforeSend
callback method. Once implemented, the method is invoked when the SDK captures an error event, right before sending it to your Sentry account. It receives the event object as a parameter, so you can use that to modify the event's data or drop it completely (by returning null
) based on your custom logic and the data available on the event, like tags, environment, release version, error attributes, and so on. Note that only error and message events pass through beforeSend
. Tansaction events have a separate method, beforeSendTransaction
, though it's not yet supported in all SDKs. Learn more about both methods in Filtering Events.
SDK Configuration
The Sentry SDKs have several configuration options that can be used to filter unwanted errors from leaving your application's runtime. A lot of these options are platform-specific, so make sure you look for yours in our platform docs under Configuration. Following are some configuration examples for JavaScript and other SDKs.
JavaScript
The JavaScript SDK includes multiple integrations: functional plugins that you can configure, enable, or disable. Learn more in JavaScript SDK Integrations. Several integrations allow you to configure the types of events you want Sentry to monitor; two of them, InboundFilters and GlobalHandlers are explained below:
InboundFilters
The integration is enabled by default and adds the following configuration options to the SDK:
allowUrls
: Domains that might raise acceptable exceptions represented in a regex pattern format.denyUrls
: A list of strings or regex patterns which match error URLs that should be blocked from sending events. Configuring bothallowUrls
anddenyUrls
on the SDK can be used to block subdomains of the domains listed inallowUrls
.ignoreErrors
: Instructs the SDK to never send an error to Sentry if it matches any of the listed error messages or regular expressions. The SDK will try to match against the message or, if there is no message, a string containing the exception type and value formatted like so:ExceptionType: value
.
To learn more and see code samples, check out:
GlobalHandlers
This integration attaches global handlers to capture uncaught exceptions (onerror
) and unhandled rejections (onunhandledrejection
). Both handlers are enabled by default, but can be disabled through configuration. Learn more in GlobalHandlers Integration.
Check out additional configuration options with the tryCatch and ReportingObserver integrations.
Other SDKs
Java - Sentry's Java SDK provides integrations with common Java logging libraries. The configuration allows you to set a logging threshold determining the minimum level to capture a log message as breadcrumb. It also provides a setting to configure the minimum log level to capture an event.
PHP - The error_types
configuration option allows you to set the error types you want Sentry to monitor. Learn more in PHP Basic Options.
Ruby - The excluded_exceptions
configuration option allows you to set the exception types you wish to suppress. Learn more in the Ruby configuration docs
Our documentation is open source and available on GitHub. Your contributions are welcome, whether fixing a typo (drat!) to suggesting an update ("yeah, this would be better").