TiDB Cloud Built-in Alerting
TiDB Cloud provides you with an easy way to view alerts, edit alert rules, and subscribe to alert notification emails.
This document describes how to do these operations and provides the TiDB Cloud built-in alert conditions for your reference.
View alerts
In TiDB Cloud, you can view both active and closed alerts on the Alerts page.
In the TiDB Cloud console, navigate to the Clusters page of your project.
Click the name of the target cluster. The cluster overview page is displayed.
Click Alerts in the left navigation pane.
The Alerts page displays the active alerts by default. You can view the information of each active alert such as the alert name, trigger time, and duration.
If you also want to view the closed alerts, just click the Status drop-down list and select Closed or All.
Edit alert rules
In TiDB Cloud, you can edit the alert rules by disabling or enabling the alerts or updating the alert threshold.
On the Alerts page, click Edit Rules.
Disable or enable alert rules as needed.
Click Edit to update the threshold of an alert rule.
Subscribe to alert notification emails
To get alert notification emails of clusters in your project, take the following steps:
On the Alerts page , click Subscribe Alerts.
Enter your email address, and then click Subscribe.
Alternatively, you can also add the subscription from the Alert Subscription page as follows:
- Log in to the TiDB Cloud console.
- Click in the lower-left corner, switch to the target project if you have multiple projects, and then click Project Settings.
- On the Project Settings page of your project, click Alert Subscription in the left navigation pane.
- Click Add Subscriber, enter your email address in the displayed dialog, and then click Add.
If an alert condition remains unchanged, the alert sends email notifications every 3 hours.
Unsubscribe from alert notification emails
If you no longer want to receive alert notification emails of clusters in your project, take the following steps:
- Log in to the TiDB Cloud console.
- Click in the lower-left corner, switch to the target project if you have multiple projects, and then click Project Settings.
- On the Project Settings page of your project, click Alert Subscription in the left navigation pane.
- Locate your email address and click Unsubscribe.
- Click Delete to confirm the unsubscription.
TiDB Cloud built-in alert conditions
The following table provides the TiDB Cloud built-in alert conditions and the corresponding recommended actions.
Resource usage alerts
Condition | Recommended Action |
---|---|
Total TiDB node memory utilization across cluster exceeded 70% for 10 minutes | Consider increasing the node number or node size for TiDB to reduce the memory usage percentage of the current workload. |
Total TiKV node memory utilization across cluster exceeded 70% for 10 minutes | Consider increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. |
Total TiFlash node memory utilization across cluster exceeded 70% for 10 minutes | Consider increasing the node number or node size for TiFlash to reduce the memory usage percentage of the current workload. |
Total TiDB node CPU utilization exceeded 80% for 10 minutes | Consider increasing the node number or node size for TiDB to reduce the CPU usage percentage of the current workload. |
Total TiKV node CPU utilization exceeded 80% for 10 minutes | Consider increasing the node number or node size for TiKV to reduce the CPU usage percentage of the current workload. |
Total TiFlash node CPU utilization exceeded 80% for 10 minutes | Consider increasing the node number or node size for TiFlash to reduce the CPU usage percentage of the current workload. |
TiKV storage utilization exceeds 80% | Consider increasing the node number or node storage size for TiKV to increase your storage capacity. |
TiFlash storage utilization exceeds 80% | Consider increasing the node number or node storage size for TiFlash to increase your storage capacity. |
Max memory utilization across TiDB nodes exceeded 70% for 10 minutes | Consider checking if there is any hotspot in the cluster or increasing the node number or node size for TiDB to reduce the memory usage percentage of the current workload. |
Max memory utilization across TiKV nodes exceeded 70% for 10 minutes | Consider checking if there is any hotspot in the cluster or increasing the node number or node size for TiKV to reduce the memory usage percentage of the current workload. |
Max CPU utilization across TiDB nodes exceeded 80% for 10 minutes | Consider checking if there is any hotspot in the cluster or increasing the node number or node size for TiDB to reduce the CPU usage percentage of the current workload. |
Max CPU utilization across TiKV nodes exceeded 80% for 10 minutes | Consider checking if there is any hotspot in the cluster or increasing the node number or node size for TiKV to reduce the CPU usage percentage of the current workload. |
Data migration alerts
Condition | Recommended Action |
---|---|
Data migration job met error during data export | Check the error and see Troubleshoot data migration for help. |
Data migration job met error during data import | Check the error and see Troubleshoot data migration for help. |
Data migration job met error during incremental migration | Check the error and see Troubleshoot data migration for help. |
Data migration job has been paused for more than 6 hours during incremental migration | Data migration job has been paused for more than 6 hours during data incremental migration. The binlog in the upstream database might be purged (depending on your database binlog purge strategy) and might cause incremental migration to fail. See Troubleshoot data migration for help. |
Replication lag is larger than 10 minutes and still increasing for more than 20 minutes | See Troubleshoot data migration for help. |
Changefeed alerts
Condition | Recommended Action |
---|---|
Changefeed processor checkpoint delay more than 600 seconds | Check if the downstream system and network configuration are functioning normally, and rule out the possibility of an indexed table. |