Debug found that carbon process not running
it says "carbon-cache (instance a) is already running with pid 36"
it work by check pid in this file
which content is 36. Check
cat /proc/36/cmdline , found it is another process!
check the modify time of
carbon-cache-a.pid, which is very old. For some reason, it is not updated or deleted properly, which prevent new process to start, and lead to deadlock.
carbon-cache-a.pid, restart grafana, it work again.