๐จAlerting
Carpool supports automated alerts via Elasticsearch queries on your indexed data. This is a very powerful tool, and we'll show you how to get started.
Creating Alerts
You can configure alerts on either Account or Instruction data. In either case, to create an alert, visit the Alerting page and click the New Alert button and enter the following configurations:
A name for your alert
Wether the alert is on instruction or account data
The indexed program that you want to monitor
The account type (for account alerts) or instruction (for instruction alerts)
Next, you need to configure the query body and comparison for the alert.
Body
Here, youโll configure an Elasticsearch query for the data that youโll be monitoring. You need to use the Elasticsearch DSL. This body consists of two parts, a Query and an (optionally) Aggregation. If youโre just looking for certain types of accounts or instruction calls, a query will be sufficient, but if youโre examining state across multiple entities, combine a query with an aggregation. Hereโs an example that filters for a subset of account data, then performs an aggregation on that filtered data.
In this example, we are filtering on the foo
field of the account's data (account specific data is prefixed with data.
, unlike top-level information such as lamports and writeVersion). Our aggregation will only apply to accounts which have their foo
field set to bar
. Next, we perform an aggregation to sum the values of the amount
field, but only for those accounts which matched the filter.
Once youโve entered a valid body for the alert, you can click the Test button to execute the query and see what data will be returned. This is important for the next step, configuring the Condition.
Transaction Timestamps
While account-based data reflects the current state of the world, instruction data is indexed with timestamps. When querying instruction data, make sure to include a timestamp filter, such as the following:
now
references the evaluation time of the query. This filter will only include data from the last 5 minutes.
Condition
Next, you need to define a trigger condition based on the Body youโve defined. This is again defined via Elasticsearch syntax. The two useful condition types are compare and array-compare. You can access the results of the Body inside the Condition via ctx.payload
. For example, to access the total number of documents which match a Bodyโs query, use ctx.payload.hits.total
. To access an aggregation, use ctx.payload.aggregations.AGGREGATION_NAME.value
where AGGREGATION_NAME
is the name of the aggregation you defined in the Body.
Examples
To fire an alert if any documents match your query, use this:
To alert on the result of an aggregation (such as the one we defined above), you can use a Condition like this. Note that the exact access path for your data will vary depending on the aggregation you've used.
With all of these fields filled out, you can now click Submit to create you alert.
Configuring Alerts
With your Alert defined, you will now see it on the main Alerting page. Before it will actually fire, you need to add one or more Actions. We currently support two types of Actions, Webhooks and PagerDuty. In both cases, expand the Alertโs panel in the main Alerting page, make your changes, and click Update at the bottom of the panel.
Testing Conditions
In the future, we will add the ability to test-fire your alerts with their associated Conditions and Actions. Until then, we suggest using an always condition to verify your Actions, and a lower threshold (something that will always be true) to verify your Condition.
Webhook
Use Webhook Actions to fire an HTTP request to an endpoint of your choosing with a specified payload. This is useful for triggering automated actions based on the nature of the alert youโve configured.
A Webhook Action is comprised of three parts:
Endpoint
HTTP Method
Body
The first two are self-explanatory, but the body of a Webhook action has some unique properties. It will be sent as raw text to the endpoint (youโll need to parse the JSON on your end), and can also make use of some special Elasticsearch functions, such as transforming your Body result to JSON (using {{#toJson}}ctx.payload{{/toJson}}
). We strongly recommend including a unique key in this body to verify that calls to your endpoint actually came from Carpool.
PagerDuty
Carpool also supports a direct PagerDuty integration. Simply specify the following fields:
Routing Key
Severity Level
Summary
Source
Check the PagerDuty documentation for more information about these fields.
Integration Support
In Beta, our integration is limited to sending Trigger events to the PagerDuty API. We havenโt built de-duplication keys or the ability to resolve PagerDuty alerts through our UI. When handling alerts, first Ack the alert in Carpool (so we stop pinging PagerDuty), then Ack the alert in PagerDuty.
Managing Alerts
Acknowledging Alerts
When one of your Alerts fires, you will see a Firing status indicator on the Alerting page.
In order to Ack your Alert, simply expand the Alertโs panel and click the Ack button at the top.
Once you Ack a firing alert, it will be marked as acknowledged. Itโs very important to understand that, in this state, the alert will not fire again until its Condition no longer evaluates to โtrueโ. This is expected behavior, but worth pointing out. Alerts can only enter the โfiringโ state from the โokโ state.
Updating Alert Body or Condition
You can freely update your Alertโs Body or Condition at any time. Simply make the changes in the Alertโs panel and hit the Update button. Note that updating an Alert will reset it to the โokโ state.
Deleting an Alert
Simply click the Delete button at the bottom of the Alert panel to remove an alert.
Last updated