🚨Alerting

Carpool supports automated alerts via Elasticsearch queries on your indexed data. This is a very powerful tool, and we'll show you how to get started.

Alerts

The current process to create and manage alerts is still based on using the raw Elasticsearch DSL. This will be updated to use our Query API soon, but in the meantime, please feel free to reach out to the team for help in setting up your alerts.

Creating Alerts

You can configure alerts on either Account or Instruction data. In either case, to create an alert, visit the Alerting page and click the New Alert button and enter the following configurations:

• A name for your alert

• Wether the alert is on instruction or account data

• The indexed program that you want to monitor

• The account type (for account alerts) or instruction (for instruction alerts)

An example alert using BidReceipt account data from the Auction House program

Next, you need to configure the query body and comparison for the alert.

Body

Here, you’ll configure an Elasticsearch query for the data that you’ll be monitoring. You need to use the Elasticsearch DSL. This body consists of two parts, a Query and an (optionally) Aggregation. If you’re just looking for certain types of accounts or instruction calls, a query will be sufficient, but if you’re examining state across multiple entities, combine a query with an aggregation. Here’s an example that filters for a subset of account data, then performs an aggregation on that filtered data.

{
   "query":{
      "bool":{
         "filter":{
            "term":{
               "data.foo":"bar"
            }
         }
      },
      "aggs":{
         "total-amount":{
            "sum":{
               "field":"data.amount"
            }
         }
      }
   }
}

In this example, we are filtering on the foo field of the account's data (account specific data is prefixed with data., unlike top-level information such as lamports and writeVersion). Our aggregation will only apply to accounts which have their foo field set to bar. Next, we perform an aggregation to sum the values of the amount field, but only for those accounts which matched the filter.

Once you’ve entered a valid body for the alert, you can click the Test button to execute the query and see what data will be returned. This is important for the next step, configuring the Condition.

Aggregation Keys

When defining an aggregation, the JSON keys immediately under aggs can be whatever name you’d like. You can even have multiple aggregations. You will reference those names in the Condition.

Transaction Timestamps

While account-based data reflects the current state of the world, instruction data is indexed with timestamps. When querying instruction data, make sure to include a timestamp filter, such as the following:

"range": {
  "timestamp": {
    "gte": "now-5m",
    "lte": "now"
  }
}

now references the evaluation time of the query. This filter will only include data from the last 5 minutes.

Condition

Next, you need to define a trigger condition based on the Body you’ve defined. This is again defined via Elasticsearch syntax. The two useful condition types are compare and array-compare. You can access the results of the Body inside the Condition via ctx.payload. For example, to access the total number of documents which match a Body’s query, use ctx.payload.hits.total. To access an aggregation, use ctx.payload.aggregations.AGGREGATION_NAME.value where AGGREGATION_NAME is the name of the aggregation you defined in the Body.

Examples

To fire an alert if any documents match your query, use this:

{
  "compare": {
    "ctx.payload.hits.total": {
      "gte": 1
    }
  }
}

To alert on the result of an aggregation (such as the one we defined above), you can use a Condition like this. Note that the exact access path for your data will vary depending on the aggregation you've used.

{
  "compare": {
    "ctx.payload.aggregations.total-amount.value": {
      "gte": 100000000000
    }
  }
}

With all of these fields filled out, you can now click Submit to create you alert.

Configuring Alerts

With your Alert defined, you will now see it on the main Alerting page. Before it will actually fire, you need to add one or more Actions. We currently support two types of Actions, Webhooks and PagerDuty. In both cases, expand the Alert’s panel in the main Alerting page, make your changes, and click Update at the bottom of the panel.

Testing Conditions

In the future, we will add the ability to test-fire your alerts with their associated Conditions and Actions. Until then, we suggest using an always condition to verify your Actions, and a lower threshold (something that will always be true) to verify your Condition.

Webhook

Use Webhook Actions to fire an HTTP request to an endpoint of your choosing with a specified payload. This is useful for triggering automated actions based on the nature of the alert you’ve configured.

A Webhook Action is comprised of three parts:

• Endpoint

• HTTP Method

• Body

The first two are self-explanatory, but the body of a Webhook action has some unique properties. It will be sent as raw text to the endpoint (you’ll need to parse the JSON on your end), and can also make use of some special Elasticsearch functions, such as transforming your Body result to JSON (using {{#toJson}}ctx.payload{{/toJson}}). We strongly recommend including a unique key in this body to verify that calls to your endpoint actually came from Carpool.

In this example, the Webhook Action sends a payload to our endpoint via a post request. The body includes a “secret” field and the JSON-serialized Body result.

Pardon Our Dust

The webhook body field doesn’t strictly have to be JSON, so we need to create a special editor for it to improve ease of use. For now, it’s a raw text input. As mentioned above, test your webhooks with an “always” condition to make sure it behaves as you’d expect.

PagerDuty

Carpool also supports a direct PagerDuty integration. Simply specify the following fields:

• Routing Key

• Severity Level

• Summary

• Source

Check the PagerDuty documentation for more information about these fields.

Integration Support

In Beta, our integration is limited to sending Trigger events to the PagerDuty API. We haven’t built de-duplication keys or the ability to resolve PagerDuty alerts through our UI. When handling alerts, first Ack the alert in Carpool (so we stop pinging PagerDuty), then Ack the alert in PagerDuty.

Managing Alerts

Alert Cadence

By default, all Alerts are evaluated on a 5 minute interval. As our Beta progresses and we understand the load this causes on our backend, we will make this customizable. If you have an immediate need for an alert that fires more frequently, contact us and we can help.

Acknowledging Alerts

When one of your Alerts fires, you will see a Firing status indicator on the Alerting page.

“My Instruction Alert” is firing, while “My Alert” has been acknowledged.

In order to Ack your Alert, simply expand the Alert’s panel and click the Ack button at the top.

Once you Ack a firing alert, it will be marked as acknowledged. It’s very important to understand that, in this state, the alert will not fire again until its Condition no longer evaluates to “true”. This is expected behavior, but worth pointing out. Alerts can only enter the “firing” state from the “ok” state.

Updating Alert Body or Condition

You can freely update your Alert’s Body or Condition at any time. Simply make the changes in the Alert’s panel and hit the Update button. Note that updating an Alert will reset it to the “ok” state.

Deleting an Alert

Simply click the Delete button at the bottom of the Alert panel to remove an alert.