Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Medium
Fix Version/s: None
Affects Version/s: None
Component/s: e2
Labels:
None

Hello, during testing, we encountered crash issues in the E2T's Prometheus metric Increment function. We found two locations where the crash can occur:

https://github.com/o-ran-sc/ric-plt-e2/blob/i-release/RIC-E2-TERMINATION/sctpThread.cpp#L2180
https://github.com/o-ran-sc/ric-plt-e2/blob/i-release/RIC-E2-TERMINATION/sctpThread.cpp#L2451

Below are part of the logs corresponding to the crash, including debug messages and a gdb backtrace:

{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP 16 fd"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP fd 16 stream 0 "}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Finish Read from SCTP 16 fd message length = 581"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP 16 fd"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP fd 16 stream 0 "}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Finish Read from SCTP 16 fd message length = 18"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Before Encoding E2AP PDU for : , Read time is : 0 seconds, 34406 nanoseconds"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"PDU buffer length = 18, data =  : 0003000e0000020031"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Before Encoding E2AP PDU for : , Read time is : 0 seconds, 131485 nanoseconds"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"PDU buffer length = 581, data =  : 0001008240000004003100020001000300090037343738b4c67788000a0081ff0000080081f900000081e920c04f52414e2d4532534d2d4b504d0000054f494431323305004b504d206d6f6e69746f72084e86e8a164799800600001010700506572696f646963207265706f727401051401011d004f2d4455204d6561737572656d656e7420436f6e7461696e657220666f72207468652035474320636f6e6e6563746564206465706c6f796d656e74010101010001021d004f2d4455204d6561737572656d656e7420436f6e7461696e657220666f72207468652045504320636f6e6e6563746564206465706c6f796d656e74010101010001031e804f2d43552d4350204d6561737572656d656e7420436f6e7461696e657220666f72207468652"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"After Encoding E2AP PDU for : , Read time is : 0 seconds, 15257 nanoseconds"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Encoding E2AP PDU past : InitiatingMessage ::= {     procedureCode: 3     criticality: 0 (reject)     value: ResetRequest ::= {         protocolIEs: ProtocolIE-Container ::= {             ResetRequestIEs ::= {                 id: 49                 criticality: 1 (ignore)                 value: 1             }             ResetRequestIEs ::= {                 id: 1                 criticality: 1 (ignore)                 value: 0 (e2node-component-unknown)             }         }     } } "}
{"ts":1705853425590,"crit":"INFO","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"in asnInitiatingRequest"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Initiating message 3 "}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737169561344","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Got Reset "}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"After Encoding E2AP PDU for : , Read time is : 0 seconds, 41276 nanoseconds"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Encoding E2AP PDU past : InitiatingMessage ::= {     procedureCode: 1     criticality: 0 (reject)     value: E2setupRequest ::= {         protocolIEs: ProtocolIE-Container ::= {             E2setupRequestIEs ::= {                 id: 49                 criticality: 0 (reject)                 value: 1             }             E2setupRequestIEs ::= {                 id: 3                 criticality: 0 (reject)                 value: GlobalE2node-gNB-ID ::= {                     global-gNB-ID: GlobalgNB-ID ::= {                         plmn-id: 37 34 37                         gnb-id: B4 C6 77 88 (3 bits unused)                     }                 }             }             E2setupRequestIEs ::= {                 id: 10                 criticality: 0 (reject)                 value: RANfunctions-List ::= {                     ProtocolIE-SingleContainer ::= {                         id: 8                         criticality: 0 (reject)                         value: RANfunction-Item ::= {                             ranFunctionID: 0                             ranFunctionDefinition:                                  20 C0 4F 52 41 4E 2D 45 32 53 4D 2D 4B 50 4D 00                                  00 05 4F 49 44 31 32 33 05 00 4B 50 4D 20 6D 6F                                  6E 69 74 6F 72 08 4E 86 E8 A1 64 79 98 00 60 00                                  01 01 07 00 50 65 72 69 6F 64 69 63 20 72 65 70                                  6F 72 74 01 05 14 01 01 1D 00 4F 2D 44 55 20 4D                                  65 61 73 75 72 65 6D 65 6E 74 20 43 6F 6E 74 61                                  69 6E 65 72 20 66 6F 72 20 74 68 65 20 35 47 43                                  20 63 6F 6E 6E 65 63 74 65 64 20 64 65 70 6C 6F                                  79 6D 65 6E 74 01 01 01 01 00 01 02 1D 00 4F 2D                                  44 55 20 4D 65 61 73 75 72 65 6D 65 6E 74 20 43                                  6F 6E 74 61 69 6E 65 72 20 66 6F 72 20 74 68 65                                  20 45 50 43 20 63 6F 6E 6E 65 63 74 65 64 20 64                                  65 70 6C 6F 79 6D 65 6E 74 01 01 01 01 00 01 03                                  1E 80 4F 2D 43 55 2D 43 50 20 4D 65 61 73 75 72                                  65 6D 65 6E 74 20 43 6F 6E 74 61 69 6E 65 72 20                                  66 6F 72 20 74 68 65 20 35 47 43 20 63 6F 6E 6E                                  65 63 74 65 64 20 64 65 70 6C 6F 79 6D 65 6E 74                                  01 01 01 01 00 01 04 1E 80 4F 2D 43 55 2D 43 50                                  20 4D 65 61 73 75 72 65 6D 65 6E 74 20 43 6F 6E                                  74 61 69 6E 65 72 20 66 6F 72 20 74 68 65 20 45                                  50 43 20 63 6F 6E 6E 65 63 74 65 64 20 64 65 70                                  6C 6F 79 6D 65 6E 74 01 01 01 01 00 01 05 1E 80                                  4F 2D 43 55 2D 55 50 20 4D 65 61 73 75 72 65 6D                                  65 6E 74 20 43 6F 6E 74 61 69 6E 65 72 20 66 6F                                  72 20 74 68 65 20 35 47 43 20 63 6F 6E 6E 65 63                                  74 65 64 20 64 65 70 6C 6F 79 6D 65 6E 74 01 01                                  01 01 00 01 06 1E 80 4F 2D 43 55 2D 55 50 20 4D                                  65 61 73 75 72 65 6D 65 6E 74 20 43 6F 6E 74 61                                  69 6E 65 72 20 66 6F 72 20 74 68 65 20 45 50 43                                  20 63 6F 6E 6E 65 63 74 65 64 20 64 65 70 6C 6F                                  79 6D 65 6E 74 01 01 01 01                             ranFunctionRevision: 2                             ranFunctionOID: OID123                         }                     }                 }             }             E2setupRequestIEs ::[truncated]"}
{"ts":1705853425590,"crit":"INFO","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"in asnInitiatingRequest"}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Initiating message 1 "}
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Got E2setup"}
{"ts":1705853425590,"crit":"INFO","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"in collectSetupRequestData"}gNB_CU_UP_ID and gNB_DU_ID is null
cuupid =-1
duid =-1
ranName =gnb_734_373_1698cef1
{"ts":1705853425590,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start from SCTP 16 fd"}
{"ts":1705853425591,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"SCTP status assoc id 88 instrms 2 outstrms 2"}
{"ts":1705853425591,"crit":"INFO","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"in buildPrometheusList"}
{"ts":1705853425591,"crit":"INFO","id":"E2Terminator","mdc":{"PID":"140737186346752","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-c6mmm","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"in buildAndSendSetupRequest"}Thread 10 "e2" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffecff9700 (LWP 37)]
0x0000555555776124 in std::atomic<double>::load(std::memory_order) const ()
#0  0x0000555555776124 in std::atomic<double>::load(std::memory_order) const ()
#1  0x0000555555775ee9 in prometheus::Gauge::Change(double) ()
#2  0x0000555555775e1c in prometheus::Gauge::Increment(double) ()
#3  0x0000555555775dee in prometheus::Gauge::Increment() ()
#4  0x000055555575dcd0 in prometheus::Counter::Increment() ()
#5  0x0000555555624c6b in asnInitiatingRequest (pdu=0x7fffdc002770, sctpMap=0x55555597a330, message=..., rmrMessageBuffer=..., streamId=0) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:2276
#6  0x000055555561253a in receiveDataFromSctp (events=0x7fffdc000e40, sctpMap=0x55555597a330, numOfMessages=@0x7fffecd77e60: 3, rmrMessageBuffer=..., ts=...) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1572
#7  0x000055555560f3bc in listener (params=0x7fffffff9000) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1057
#8  0x00005555556fdf34 in std::__invoke_impl<void, void (*)(sctp_params*), sctp_params*> (__f=@0x55555597a570: 0x55555560d5dc <listener(sctp_params*)>) at /usr/include/c++/9/bits/invoke.h:60
#9  0x00005555556fd849 in std::__invoke<void (*)(sctp_params*), sctp_params*> (__fn=@0x55555597a570: 0x55555560d5dc <listener(sctp_params*)>) at /usr/include/c++/9/bits/invoke.h:95
#10 0x00005555556fc64f in std::thread::_Invoker<std::tuple<void (*)(sctp_params*), sctp_params*> >::_M_invoke<0ul, 1ul> (this=0x55555597a568) at /usr/include/c++/9/thread:244
#11 0x00005555556fb29a in std::thread::_Invoker<std::tuple<void (*)(sctp_params*), sctp_params*> >::operator() (this=0x55555597a568) at /usr/include/c++/9/thread:251
#12 0x00005555556f9650 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(sctp_params*), sctp_params*> > >::_M_run (this=0x55555597a560) at /usr/include/c++/9/thread:195
#13 0x00007ffff750edf4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007ffff7963609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#15 0x00007ffff7888353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I will also attach the full log below. We could not identify the root cause of the crash, but any ideas would be appreciated. Thanks!

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

e2t_increment.log
291 kB
09/Feb/24 4:15 AM

- Issue Only
- Show All Reviews
- Show Open Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Assignee:: Himanshu Purohit

Reporter:: Tianchang Yang

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 09/Feb/24 4:15 AM

Updated:: 19/Apr/24 10:37 AM

Details

Description

Attachments

Attachments

Gerrit Reviews

Activity

People

Dates