Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Issues

Go to all issues
Select view

Select search mode

 
50 of 790

mongodb 3.6.17 primary node crash

Won't Do

Description

2020-08-12T11:16:23.005+0800 F - [worker-5] Invariant failure asio::buffer_size(buffers) >= 4 src/mongo/transport/session_asio.h 187

2020-08-12T11:16:23.005+0800 F - [worker-5]

 

***aborting after invariant() failure

 

 

2020-08-12T11:16:23.024+0800 F - [worker-5] Got signal: 6 (Aborted).

 

0x7f7a8546c741 0x7f7a8546b789 0x7f7a8546bc6d 0x7f7a811e0500 0x7f7a80e708a5 0x7f7a80e72085 0x7f7a83acde90 0x7f7a84f514e8 0x7f7a84f5e3b4 0x7f7a84f6500f 0x7f7a84f7adf3 0x7

f7a84f7b46a 0x7f7a84d1d0d0 0x7f7a84d1b9b3 0x7f7a8531e1c4 0x7f7a811d8851 0x7f7a80f2611d

----- BEGIN BACKTRACE -----

 

{"backtrace":[\\{"b":"7F7A82FB0000","o":"24BC741","s":"_ZN5mongo15printStackTraceERSo"},\\{"b":"7F7A82FB0000","o":"24BB789"},
{"b":"7F7A82FB0000","o":"24BBC6D"},{"b":"7F7A811

D1000","o":"F500"},\\{"b":"7F7A80E3E000","o":"328A5","s":"gsignal"},
{"b":"7F7A80E3E000","o":"34085","s":"abort"},{"b":"7F7A82FB0000","o":"B1DE90","s":"_ZN5mongo22invariant

FailedWithMsgEPKcS1_S1_j"},
{"b":"7F7A82FB0000","o":"1FA14E8"},{"b":"7F7A82FB0000","o":"1FAE3B4","s":"_ZN5mongo9transport18TransportLayerASIO16ASIOSourceTicket15_headerCa

llbackERKNS_6StatusEm"},
{"b":"7F7A82FB0000","o":"1FB500F"},{"b":"7F7A82FB0000","o":"1FCADF3","s":"_ZN4asio6detail9scheduler11do_wait_oneERNS0_27conditionally_enabled_mut

ex11scoped_lockERNS0_21scheduler_thread_infoElRKSt10error_code"},
{"b":"7F7A82FB0000","o":"1FCB46A","s":"_ZN4asio6detail9scheduler8wait_oneElRSt10error_code"},{"b":"7F7A8

2FB0000","o":"1D6D0D0","s":"_ZN4asio10io_context7run_forIlSt5ratioILl1ELl1000000000EEEEmRKNSt6chrono8durationIT_T0_EE"},{"b":"7F7A82FB0000","o":"1D6B9B3","s":"_ZN5mongo9

transport23ServiceExecutorAdaptive20_workerThreadRoutineEiSt14_List_iteratorINS1_11ThreadStateEE"},\\{"b":"7F7A82FB0000","o":"236E1C4"},
{"b":"7F7A811D1000","o":"7851"},{"b

":"7F7A80E3E000","o":"E811D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.17-4.0", "gitVersion" : "96e9c7218eebc2995dd847d6185f20f102a86055", "compiledModules"

: [], "uname" :
{ "sysname" : "Linux", "release" : "2.6.32-279.23.1.mi5.el6.x86_64", "version" : "#1 SMP Mon Sep 2 12:00:41 CST 2013", "machine" : "x86_64" }, "somap" :

[
{ "b" : "7F7A82FB0000", "elfType" : 3, "buildId" : "FE708F115F5A9827FEA3B4E717A5729249FB442E" }, { "b" : "7FFF7BCFF000", "elfType" : 3, "buildId" : "A6539A72BE0493D910

90D15510B7866310CBEA50" },
{ "b" : "7F4038D78000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D0ABBCCAC542E41D33A638138FEC450AC08A1CF2" }, { "b" : "7F4036F

67000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "732F8FD5054C4FA43CF0CD4CC8C5FF02CEA3CC54" },
{ "b" : "7F402F514000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "0E587BA105E8193808F87D22D0B246FA1BE3F5B1" },
{ "b" : "7F40302FA000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "C447C77E41A336BA9AEE12D08FBF5D15948D7468" },
{ "b" : "7F40370E0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "31545FEA4F1F72061992E79A1DF461EC719942E8" },
{ "b" : "7F4035900000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3A8D65B9A373C0AFAF106F3A979835B16DBEFF1A" },
{ "b" : "7F4032A94000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "318EAB33420B000D542F09B91B716BACAB1AD546" },
{ "b" : "7F4038890000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "15B0822C819020F18BBF0E0C0286373155E03BE2" },
{ "b" : "7F4037288000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "5E9DDD9EE40AD0D4DDD032CE1086E402B7FA955A" },
{ "b" : "7F4037404000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "A26BC945B5765B1258DB01FFEFB0C4F53F3961D7" },
{ "b" : "7F40351EE000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "CE152B8676517F23E7F54AD6408330979BE41443" },
{ "b" : "7F40377D1000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "14853815DD64F2B830B8DCCB3A958A3804E13EFC" },
{ "b" : "7F403783E000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "F2BBDD778ABFECFBA0C59BBCBA94D1151DDF96E4" },
{ "b" : "7F4039F8E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D6BD776B36DAC438642CF84B282956738727901D" },
{ "b" : "7F402E20C000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "417A6ED67D904CD799091B56BDFADA0D594BB094" },
{ "b" : "7F402C7C1000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "28C7A6398A7051F6C3BF095A17EE5AD94D24D025" },
{ "b" : "7F403217D000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "76A3DEEB6876CBED69A57D3EBC1E2AFBCA84EC76" },
{ "b" : "7F4032297000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "605701A8AE551604303523B4F0D3A7E98CF9E153" },
{ "b" : "7F403246B000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "190D45F6743DEF9DF8169D65801D4575B01825BD" },
{ "b" : "7F4033667000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "4623A78918C882770E81AE7B5EE9DDF8DD2B6674" },
{ "b" : "7F402F02B000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "357DC97878A5DA1245F7A417864F192DA0F75EA1" },
{ "b" : "7F402F1FF000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "50997920C78BE240D8732DAC3C8795A32261F7AE" },
{ "b" : "7F4030AC4000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "E1F29C95A349AE5236EA1DBF853EBD1F2F30B2A7" },
{ "b" : "7F402F09E000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "7D8C67CD016553D4A3128A0D8291EF98E99A9D57" },
{ "b" : "7F402FA9A000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "4CE42AAA1D0281A833286F4AC1542E763578FB7B" },
{ "b" : "7F402F495000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "F743CDD819CFB0EBD28E8B0789FC1D6110745AD0" },
{ "b" : "7F402EE58000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "902962F62870840162B3EBAB532C37EA13D092B6" },
{ "b" : "7F402B431000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "F8EAFB6F5862E627EFF7F374852C121116771475" },
{ "b" : "7F4031DFA000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "5F5EB7F30B61E0DAF6BBF8A367C388A54B7010EA" },
{ "b" : "7F4032BEB000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "2FE020B6D9C5794700EE18696F240201E5E6976F" },
{ "b" : "7F4030DE0000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "DAE2A7E4E8B37D43EF6839FF5D8E012AFCF21A69" },
{ "b" : "7F402FBDD000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8A8734DC37305D8CC2EF8F8C3E5EA03171DB07EC" },
{ "b" : "7F403117B000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "68195872ECFB188389D29AAF01031A976FD18168" },
{ "b" : "7F403375C000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A287DC6B86A9823038F057105CE64671E0B392EC" },
{ "b" : "7F7A7DF56000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "2AF353E693CA8FBEE27C4E64AFAE5878264793EA" },
{ "b" : "7F402E5E2000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "DCB2DD56F9A628D3EAD83F5AAAAD197E0A2346E3" },
{ "b" : "7F7A7D9DC000", "path" : "/usr/lib64/sasl2/libcrammd5.so", "elfType" : 3, "buildId" : "1D8442D8C6CD2BCE4225A132A4A2510D9988A332" },
{ "b" : "7F7A7D7D7000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "9691E0170713DEF33D8D58D817B445135242DEE2" },
{ "b" : "7F7A7D5CA000", "path" : "/usr/lib64/sasl2/libdigestmd5.so", "elfType" : 3, "buildId" : "BD7C14AF17BF2EA8E155FA5A078E65C864D2E268" },
{ "b" : "7F7A7D3C5000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "EDF30C4BF72B5DC37E0A6ACCB62E53985F65276A" },
{ "b" : "7F7A7D1C0000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "358A33D0A8A76B939B28291406CD36994A7AFB21" } ] }}

mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f7a8546c741]

mongod(+0x24BB789) [0x7f7a8546b789]

mongod(+0x24BBC6D) [0x7f7a8546bc6d]

 

libpthread.so.0(+0xF500) [0x7f7a811e0500]

libc.so.6(gsignal+0x35) [0x7f7a80e708a5]

libc.so.6(abort+0x175) [0x7f7a80e72085]

mongod(_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j+0x0) [0x7f7a83acde90]

mongod(+0x1FA14E8) [0x7f7a84f514e8]

mongod(_ZN5mongo9transport18TransportLayerASIO16ASIOSourceTicket15_headerCallbackERKNS_6StatusEm+0xD4) [0x7f7a84f5e3b4]

mongod(+0x1FB500F) [0x7f7a84f6500f]

mongod(_ZN4asio6detail9scheduler11do_wait_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoElRKSt10error_code+0xC3) [0x7f7a84f7adf3]

mongod(_ZN4asio6detail9scheduler8wait_oneElRSt10error_code+0xCA) [0x7f7a84f7b46a]

mongod(_ZN4asio10io_context7run_forIlSt5ratioILl1ELl1000000000EEEEmRKNSt6chrono8durationIT_T0_EE+0xC0) [0x7f7a84d1d0d0]

mongod(_ZN5mongo9transport23ServiceExecutorAdaptive20_workerThreadRoutineEiSt14_List_iteratorINS1_11ThreadStateEE+0x423) [0x7f7a84d1b9b3]

mongod(+0x236E1C4) [0x7f7a8531e1c4]

libpthread.so.0(+0x7851) [0x7f7a811d8851]

libc.so.6(clone+0x6D) [0x7f7a80f2611d]

----- END BACKTRACE -----

Environment

os version: centos 6.3

 

Smart Checklist

Details

Assignee

Reporter

Components

Affects versions

Priority

Smart Checklist

Created August 13, 2020 at 11:29 AM
Updated March 6, 2024 at 4:48 PM
Resolved March 25, 2022 at 11:31 AM

Activity

Show:

Igor SolodovnikovMarch 25, 2022 at 11:31 AM
Edited

Closing this thicket because v3.6 is EOL.
Feel free to reopen it if you think the issue still exists in any alive version.

Akira KuroganeAugust 13, 2020 at 1:15 PM

Hi Weipengfei.

That's interesting. It seems like the function below was reached.

Firstly this code couldn't be intentionally for a HTTP request because the HTTP REST interface which used to be in mongod, which showed diagnostic info, was deprecated and removed by the time v3.6 was released.

I'm not sure how the code was reached, even allowing for mistaken TCP communication, because I currently see only two paths to checkForHTTPRequest() and both ensure a buffer of certain MSGHEADER::View struct (or larger) exists.

One way to it is a general use TransportLayerASIO::ASIOSourceTicket:: function and another specifically for a SSL Handshake. The backtrace in the output you shared shows it's first.

Do you happen to know any special sort of circumstance that corresponds in time with this crash?

Akira

Flag notifications