Skip to content

Commit 3b3656b

Browse files
committed
Add potential_problem detection feature documentation
1 parent 9bfc0c0 commit 3b3656b

File tree

4 files changed

+115
-0
lines changed

4 files changed

+115
-0
lines changed
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
title: 🔍 Potential Problem Detection
3+
description: Proactive detection of system issues that could affect EMQX performance
4+
---
5+
6+
# Potential Problem Detection
7+
8+
EMQX Guard Pro continuously monitors your system for potential issues that could affect the performance and stability of your EMQX cluster. By detecting these problems early, you can address them before they impact your production environment.
9+
10+
## System Time Synchronization
11+
12+
### The Problem
13+
14+
When the system time on your host machine differs significantly from the Erlang VM's internal time, it can cause various scheduling and timing issues. This commonly happens when:
15+
16+
- Your machine goes to sleep for an extended period (especially in containerized environments)
17+
- Time synchronization services are not properly configured
18+
- Virtual machines experience time drift
19+
20+
### What Guard Pro Does
21+
22+
Guard Pro monitors the difference between the OS system time and the Erlang VM's system time. If the difference exceeds 30 seconds, it will:
23+
24+
1. Send a notification with details about the time discrepancy
25+
2. Include both timestamps and the difference between them
26+
3. Automatically suppress repeated notifications for the same issue
27+
28+
:::note
29+
A single time discrepancy notification is usually not a concern as the system automatically corrects itself. However, if you receive this notification repeatedly, it indicates a persistent issue with your system's time synchronization that should be addressed.
30+
:::
31+
32+
### Recommended Action
33+
34+
If you receive repeated time synchronization warnings:
35+
36+
- Ensure NTP (Network Time Protocol) is properly configured on your host system
37+
- Avoid allowing machines running EMQX to enter sleep states in production environments
38+
- For containerized deployments, ensure proper time synchronization between the host and containers
39+
40+
## Retained Message Monitoring
41+
42+
### The Problem
43+
44+
MQTT's retained message feature allows the broker to store the last message published to each topic. While useful, having too many retained messages can:
45+
46+
- Consume excessive memory
47+
- Slow down broker operations
48+
- Increase the time it takes new clients to receive all their subscribed topics' retained messages
49+
50+
### What Guard Pro Does
51+
52+
Guard Pro monitors the number of retained messages stored in the EMQX broker. If the count exceeds 100,000 messages, it will send a notification alerting you to the high number of retained messages.
53+
54+
### Recommended Action
55+
56+
If you receive a notification about too many retained messages:
57+
58+
- Review your application's use of retained messages
59+
- Consider implementing a retention policy to automatically clean up old retained messages
60+
- Use the EMQX Dashboard or CLI to manually clean up unnecessary retained messages
61+
- Consider using alternative patterns like Last Will and Testament (LWT) messages where appropriate
62+
63+
## Swap Memory Usage
64+
65+
### The Problem
66+
67+
Erlang applications like EMQX are designed to manage memory within the Erlang VM. When the system starts using swap memory:
68+
69+
- It can introduce unpredictable latency as memory pages are swapped in and out
70+
- The Erlang VM's memory management becomes less effective
71+
- Overall system performance can degrade significantly
72+
73+
### What Guard Pro Does
74+
75+
Guard Pro monitors the system's swap memory usage. If more than 128MB of swap memory is being used, it will send a notification with details about:
76+
77+
- Total swap memory available
78+
- Free swap memory
79+
- Amount of swap memory currently in use
80+
81+
### Recommended Action
82+
83+
If you receive a notification about swap memory usage:
84+
85+
- Increase the physical RAM available to the system
86+
- Adjust the system's swappiness parameter to reduce the likelihood of using swap
87+
- Review the EMQX memory settings to ensure they're appropriate for your hardware
88+
- Check for other processes that might be consuming excessive memory
89+
90+
## Configuration
91+
92+
No additional configuration is required for the potential problem detection feature. It runs automatically as part of EMQX Guard Pro's monitoring system.
93+
94+
## Notification Example
95+
96+
When a potential problem is detected, you'll receive a notification through your configured notification channels with details about the specific issue:
97+
98+
![potential-problem-notification](/img/potential_problem_notification.png)
99+
100+
By addressing these issues promptly, you can maintain optimal performance and reliability of your EMQX deployment.

docs/EMQX-Guard-Pro/index.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,20 @@ Learn how to use key features of Guard, such as monitoring, alerting, and securi
143143
</div>
144144
</div>
145145
</a>
146+
147+
<a className="homepage-card" href="/EMQX-Guard-Pro/features/potential_problem">
148+
<div className="icon">
149+
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
150+
<path d="M12 8V12M12 16H12.01M22 12C22 17.5228 17.5228 22 12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12Z" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round"/>
151+
</svg>
152+
</div>
153+
<div className="card-content">
154+
<div className="title">Potential Problem Detection</div>
155+
<div className="description">
156+
Proactive detection of system issues affecting performance
157+
</div>
158+
</div>
159+
</a>
146160
</div>
147161

148162
## Configuration

sidebars.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ const sidebars: SidebarsConfig = {
5656
'EMQX-Guard-Pro/features/inconsistent_configuration',
5757
'EMQX-Guard-Pro/features/auto_backup',
5858
'EMQX-Guard-Pro/features/conn_congestion_trace',
59+
'EMQX-Guard-Pro/features/potential_problem',
5960
],
6061
},
6162
{
182 KB
Loading

0 commit comments

Comments
 (0)