rabbitmq伺服器程序突然掛掉問題

crash dump was written to: erl_crash.dump eheap_alloc: cannot

allocate 762886488 bytes of memory (of type 「heap」). aborted (core

dumped)

[2019-03-13 07:13:57,717: warning/mainprocess] consumer: connection to

broker lost. trying to re-establish the connection… traceback (most

recent call last): file

「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/celery/worker/consumer.py」,

line 279, in start

blueprint.start(self) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/celery/bootsteps.py」,

line 123, in start

step.start(parent) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/celery/worker/consumer.py」,

line 838, in start

c.loop(*c.loop_args()) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/celery/worker/loops.py」,

line 76, in asynloop

next(loop) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/kombu/async/hub.py」,

line 340, in create_loop

cb(*cbargs) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/kombu/transport/base.py」,

line 164, in on_readable

reader(loop) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/kombu/transport/base.py」,

line 146, in _read

drain_events(timeout=0) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/amqp/connection.py」,

line 303, in drain_events

chanmap, none, timeout=timeout, file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/amqp/connection.py」,

line 366, in _wait_multiple

channel, method_sig, args, content = read_timeout(timeout) file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/amqp/connection.py」,

line 337, in read_timeout

return self.method_reader.read_method() file 「/home/deploy/virtualenv/mid-system/local/lib/python2.7/site-packages/amqp/method_framing.py」,

line 189, in read_method

raise m error: [errno 104] connection reset by peer [2019-03-13 07:13:57,772: error/mainprocess] consumer: cannot connect to

amqp://guest:**@172.16.2.10:5672//: [errno 111] connection refused.

trying again in 2.00 seconds…

重啟rabbitmq伺服器

錯誤分析：

從錯誤原因eheap_alloc: cannot allocate 762886488 bytes of memory (of type 「heap」).看，有可能是作業系統已經分不出足夠的記憶體給erlang，看起來也就需要700多m，然而伺服器就是分配不起這個級別的記憶體？好像還真分配不起。。。，

可以看下當前伺服器的實時記憶體

伺服器的的最大記憶體約為8g，當前可用記憶體約為800多m，那麼也是差不多剛剛好給上面的問題分配記憶體，不過這個時候還沒考慮交換記憶體的問題

現在查一下崩潰日誌檔案，使用 locate erl_crash.dump找到崩潰日誌檔案路徑:/var/lib/rabbitmq

然後使用網上提供的分析工具對erl_crash.dump進行分析

分析結果如下：

analyzing origin_erl_crash.dump, generated on: wed mar 13 07:13:57

2019

slogan: eheap_alloc: cannot allocate 762886488 bytes of memory (of

type 「heap」).

memory:

=== processes: 764 mb processes_used: 764 mb system: 641 mb atom: 1 mb atom_used: 1 mb binary: 475 mb code: 15 mb ets: 101

mb — total: 1406 mb

different message queue lengths (5 largest different):

=== 2245 0

error logger queue length:

=== 0

file descriptors open:

=== udp: 0 tcp: 196 files: 43 — total: 239

number of processes:

=== 2245

1 95360811

3 28690
2 17731
2 10958
49 6772

1 28690

10 17731
19 10958
83 6772
28 4185

1 garbing

1 running    2243 waiting

以後的解決辦法：

1，增加伺服器效能

2，減少伺服器記憶體負擔，可以將伺服器不相關的程序遷移到其他伺服器，

3，開啟swap，不過用交換記憶體實際也不是很好，可能有負面影響

4，公升級rabbitmq

更進一步的分析可以參考這篇

docker 安裝 rabbitMQ伺服器

拉取rabbitmq映象檔案,後邊要接上 management 表名是拉取帶有web管理端的映象,有web介面方便管理.docker run d p 5672 5672 p 15672 15672 name rabbitmq rabbitmq management p 埠對映 name 容器例項名稱...

關於RabbitMQ伺服器安裝

rabbitmq安裝部署 安裝 otp win64 19.3 erlang 再安裝 rabbitmq server 3.6.9 開啟c erl8.3 bin的 werl otp安裝目錄 win r 開啟 cmd 命令 輸入 cd 開啟 c rabbitmq server rabbitmq serve...

多程序伺服器

基於tcp實現多程序伺服器 伺服器端 1 建立套接字 include include int socket int domain,int type,int protocol domain 乙個位址描述。目前僅支援af inet格式，也就是說arpa internet位址格式。type 指定socke...