With our smart scorecards you can confidently deploy releases in production and validate their performance instantly. In this section, learn more about how scorecards are graded.
We autonomously detect new deployments for all AWS Lambda and Elastic Kubernetes Service (EKS) resources, and assess their performance by looking for latency, error, or saturation deviations in varying traffic scenarios. Each release receives a scorecard that grades its performance quality.
The Release Intelligence feature empowers your team to build and iterate faster by pushing releases into production and validating their performance instantly. Our smart scorecards analyze releases for unexpected behavior so that you can quickly view and track their success.
We recommend using these scorecards as a decision making tool during your review process. Scorecards provide a quick summary of a release’s performance compared to prior releases, and can be used as a fast way to track a release’s performance against your expectations as well as our own prediction for a resource’s behavior. If a scorecard indicates a resource deviates from expected behavior, it is best practice to review and verify it is functioning as intended.
We analyze a resource’s historical performance and compare that to how we predict the new release should behave. The score itself primarily emphasizes the extent to which behavior changes (which could be considered good or bad).
While we generate a preliminary score shortly after a release is deployed, we continue to analyze its performance and update it after 24 hours (assuming we had sufficient data for our analysis).
The analysis is based on datasets pulled from varying time windows; each dataset is graded based on the following:
- Whether latency is relatively longer or shorter compared to previous releases
- Whether the error count is relatively higher or lower compared to previous releases
- The amount of traffic relative to the resource (classified as Low, Typical, or High)
We then aggregate results into an overarching score that prioritizes datasets with considerable change in high traffic scenarios. The score is graded based on the amount of deviation detected.
The score is based on a scale of 0 to 10, which is broken down into three levels of deviation based on which metrics changed and by how much.
- Minimal (10): The release is performing as expected compared to prior deployments with similar metric behavior.
- Moderate (5.0 to 9.9): The release is exhibiting inconsistent behavior, which means:
- Latency is either slightly longer or shorter, and/or
- Errors are slightly higher or lower.
- Significant (0 to 4.9): The release distinctly varies from previous observed behavior, which means:
- Latency is either considerably longer or shorter, and/or
- Errors are considerably higher or lower.
Keep in mind that we rate the level of deviations based on each resource’s respective history. You might receive a scorecard of significant deviations with considerably more errors, but the resource could still have a reasonable error rate. Compared to its historical error trends, we flag the increase as a departure from the resource’s norm.
A preliminary score is generated as soon as enough data is available. However, the final score may vary significantly based on further analysis. The initial score reflects performance immediately following the release, but we continue to analyze the release for up to 24 hours to assess its performance in varying traffic scenarios.
Scores are based on an analysis of available data following a release. If there is insufficient traffic to complete the analysis, the score will be blank or unavailable:
- Blank ( — ): Incomplete analysis; we will attempt to analyze performance for up to a week post-release.
- Unavailable (N/A): Unsuccessful analysis; no data available from the 7 days post-release.
You should factor in your release’s goal when reviewing scores since context matters. For example, the goal of a release might be a change in metric behavior that ultimately leads to an improvement in performance (for example, fewer errors and a shorter duration). However, that might not be your primary goal and those unexpected deviations could indicate a deeper underlying issue, such as bad code that is functioning incorrectly.
- If you’re attempting to improve latency, a lower score that indicates significant deviations is a good thing.
- If you’re pushing a brand new feature that is graded with moderate deviations in errors, you likely want to dig in and look for ways to improve your code.
- If you pushed an update with minimal deviations but you expected to improve the error count, you missed your goal.
- If you pushed an update with an unexpected and significant decline in latency, you might have mistakenly broken the service and should revisit your code.
Navigate to Release Intelligence from the side navigation. This page provides an overview of all of your accounts and summarizes their total scorecards broken down by the past week, the past month, and the past three months (note: scorecards are only saved in Sedai for 90 days).
Choose an account and select View all scorecards. The page will refresh to display a timeline of all scorecards by resource in that account, ordered by the most recent. You can narrow down the list of scorecards by using the filters on the lefthand side of the page, or search for a single resource to view all of its releases in the past three months.
The list groups scorecards by date. You can scroll back in time to look at older releases, or quickly jump to a specified date in the upper righthand corner of the screen (select Go to date).
The scorecard itself has three distinct sections:
- The far left portion provides the overall score
- The middle portion includes when the release occurred, as well as details about the score
- The far right portion identifies the resource
Select a scorecard to open the side drawer with its details. The side drawer has four sections:
- At the top, you can view a summary of the resource itself and copy its ARN code by clicking the blue link icon.
- Below that is the overall score summary (depending on when you view a scorecard, this could be preliminary or final).
- Following the summary is the Assessment tab. This shows each dataset that contributed to the overall score and identifies the level of deviation per metric as well as the amount of traffic during that time period. If the score is final, you can toggle to view the preliminary score and its respective datasets. You can also select a dataset to view its metric and traffic charts.
- The Related Signals tab shows any activity (including availability, optimization, or SLO signals) Sedai detected within 24 hours of the release.
While Release Intelligence automatically generates scorecards for all Lambda serverless functions and EKS resources once a deployment is detected, you can manually trigger a scorecard analysis for any resource at any time.
There are two primary reasons you might want to do this:
- The resource is not a Lambda serverless function or EKS resource. We currently are unable to autonomously detect deployments for other resource types, but can still analyze their releases when provided the resource name and release time.
- There was insufficient data due to low traffic following the release. For example, your Lambda is a low priority service that is infrequently used. When it’s last release was deployed, we were unable to analyze its performance because we didn’t have any datasets in the week following the release.
In either scenario, you can manually trigger an analysis for an on-demand scorecard and select how much time post-release you want Sedai to review performance.
To trigger a scorecard analysis, from within an account list of scorecards select the Analyze release button in the upper righthand corner. The side drawer will pop out and allow you to select the resource type and enter the resource name.