|
| 1 | +--- |
| 2 | +title: 🔍 Potential Problem Detection |
| 3 | +description: Proactive detection of system issues that could affect EMQX performance |
| 4 | +--- |
| 5 | + |
| 6 | +# Potential Problem Detection |
| 7 | + |
| 8 | +EMQX Guard Pro continuously monitors your system for potential issues that could affect the performance and stability of your EMQX cluster. By detecting these problems early, you can address them before they impact your production environment. |
| 9 | + |
| 10 | +## System Time Synchronization |
| 11 | + |
| 12 | +### The Problem |
| 13 | + |
| 14 | +When the system time on your host machine differs significantly from the Erlang VM's internal time, it can cause various scheduling and timing issues. This commonly happens when: |
| 15 | + |
| 16 | +- Your machine goes to sleep for an extended period (especially in containerized environments) |
| 17 | +- Time synchronization services are not properly configured |
| 18 | +- Virtual machines experience time drift |
| 19 | + |
| 20 | +### What Guard Pro Does |
| 21 | + |
| 22 | +Guard Pro monitors the difference between the OS system time and the Erlang VM's system time. If the difference exceeds 30 seconds, it will: |
| 23 | + |
| 24 | +1. Send a notification with details about the time discrepancy |
| 25 | +2. Include both timestamps and the difference between them |
| 26 | +3. Automatically suppress repeated notifications for the same issue |
| 27 | + |
| 28 | +:::note |
| 29 | +A single time discrepancy notification is usually not a concern as the system automatically corrects itself. However, if you receive this notification repeatedly, it indicates a persistent issue with your system's time synchronization that should be addressed. |
| 30 | +::: |
| 31 | + |
| 32 | +### Recommended Action |
| 33 | + |
| 34 | +If you receive repeated time synchronization warnings: |
| 35 | + |
| 36 | +- Ensure NTP (Network Time Protocol) is properly configured on your host system |
| 37 | +- Avoid allowing machines running EMQX to enter sleep states in production environments |
| 38 | +- For containerized deployments, ensure proper time synchronization between the host and containers |
| 39 | + |
| 40 | +## Retained Message Monitoring |
| 41 | + |
| 42 | +### The Problem |
| 43 | + |
| 44 | +MQTT's retained message feature allows the broker to store the last message published to each topic. While useful, having too many retained messages can: |
| 45 | + |
| 46 | +- Consume excessive memory |
| 47 | +- Slow down broker operations |
| 48 | +- Increase the time it takes new clients to receive all their subscribed topics' retained messages |
| 49 | + |
| 50 | +### What Guard Pro Does |
| 51 | + |
| 52 | +Guard Pro monitors the number of retained messages stored in the EMQX broker. If the count exceeds 100,000 messages, it will send a notification alerting you to the high number of retained messages. |
| 53 | + |
| 54 | +### Recommended Action |
| 55 | + |
| 56 | +If you receive a notification about too many retained messages: |
| 57 | + |
| 58 | +- Review your application's use of retained messages |
| 59 | +- Consider implementing a retention policy to automatically clean up old retained messages |
| 60 | +- Use the EMQX Dashboard or CLI to manually clean up unnecessary retained messages |
| 61 | +- Consider using alternative patterns like Last Will and Testament (LWT) messages where appropriate |
| 62 | + |
| 63 | +## Swap Memory Usage |
| 64 | + |
| 65 | +### The Problem |
| 66 | + |
| 67 | +Erlang applications like EMQX are designed to manage memory within the Erlang VM. When the system starts using swap memory: |
| 68 | + |
| 69 | +- It can introduce unpredictable latency as memory pages are swapped in and out |
| 70 | +- The Erlang VM's memory management becomes less effective |
| 71 | +- Overall system performance can degrade significantly |
| 72 | + |
| 73 | +### What Guard Pro Does |
| 74 | + |
| 75 | +Guard Pro monitors the system's swap memory usage. If more than 128MB of swap memory is being used, it will send a notification with details about: |
| 76 | + |
| 77 | +- Total swap memory available |
| 78 | +- Free swap memory |
| 79 | +- Amount of swap memory currently in use |
| 80 | + |
| 81 | +### Recommended Action |
| 82 | + |
| 83 | +If you receive a notification about swap memory usage: |
| 84 | + |
| 85 | +- Increase the physical RAM available to the system |
| 86 | +- Adjust the system's swappiness parameter to reduce the likelihood of using swap |
| 87 | +- Review the EMQX memory settings to ensure they're appropriate for your hardware |
| 88 | +- Check for other processes that might be consuming excessive memory |
| 89 | + |
| 90 | +## Configuration |
| 91 | + |
| 92 | +No additional configuration is required for the potential problem detection feature. It runs automatically as part of EMQX Guard Pro's monitoring system. |
| 93 | + |
| 94 | +## Notification Example |
| 95 | + |
| 96 | +When a potential problem is detected, you'll receive a notification through your configured notification channels with details about the specific issue: |
| 97 | + |
| 98 | + |
| 99 | + |
| 100 | +By addressing these issues promptly, you can maintain optimal performance and reliability of your EMQX deployment. |
0 commit comments