Uploaded image for project: 'Near Realtime RAN Intelligent Controller'
  1. Near Realtime RAN Intelligent Controller
  2. RIC-1043

Crash in E2T buildPrometheusList Function

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • I
    • e2
    • None

      Hello, during my testing of the E2T, I encountered the following crash in the E2T. The entire log is quite large as it contains non-related traffic, but the relevant part is provided below. I set the log level to "Debug" and used gdb to trace the root cause for the error.

       

      {"ts":1706046263587,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start EPOLL Wait. Timeout = -1"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"handling epoll event 1 out of 1"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"new message from SCTP, epoll flags are : 1"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP 19 fd"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start Read from SCTP fd 19 stream 0 "}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Finish Read from SCTP 19 fd message length = 7"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Before Encoding E2AP PDU for : gnb_734_373_16b8cef1, Read time is : 0 seconds, 29556 nanoseconds"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"PDU buffer length = 7, data =  : 0001400"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"After Encoding E2AP PDU for : gnb_734_373_16b8cef1, Read time is : 0 seconds, 14486 nanoseconds"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Encoding E2AP PDU past : InitiatingMessage ::= {     procedureCode: 1     criticality: 1 (ignore)     value: E2setupRequest ::= {         protocolIEs: ProtocolIE-Container ::= {         }     } } "}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Initiating message 1 "}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Got E2setup"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"Start from SCTP 19 fd"}
{"ts":1706046263784,"crit":"DEBUG","id":"E2Terminator","mdc":{"PID":"140737236219648","POD_NAME":"deployment-ricplt-e2term-alpha-98ff49f7d-z78rk","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"tianchang-ubuntu","SYSTEM_NAME":"SEP"},"msg":"SCTP status assoc id 116 instrms 2 outstrms 2"}
terminate called after throwing an instance of 'std::invalid_argument'
  what():  Invalid label nameThread 8 "e2" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff0f8b700 (LWP 40)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7dd3859 in __GI_abort () at abort.c:79
#2  0x00007ffff71a08d1 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff71ac37c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff71ac3e7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff71ac699 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x000000000053fcd8 in prometheus::Family<prometheus::Counter>::Add(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unique_ptr<prometheus::Counter, std::default_delete<prometheus::Counter> >) ()
#7  0x0000000000442afb in prometheus::Family<prometheus::Counter>::Add<>(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) (this=0x65b910, labels=Python Exception <class 'AttributeError'> 'NoneType' object has no attribute 'pointer': 
std::map with 2 elements) at /opt/e2/RIC-E2-TERMINATION/3rdparty/prometheus-cpp/core/include/prometheus/family.h:113
#8  0x0000000000429c03 in buildPrometheusList (peerInfo=0x7fffdc002b60, prometheusFamily=0x65b910) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1998
#9  0x00000000004282f8 in asnInitiatingRequest (pdu=0x7fffdc04e390, sctpMap=0x656700, message=..., rmrMessageBuffer=..., streamId=0) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:2231
#10 0x000000000042594e in receiveDataFromSctp (events=0x7fffdc000e40, sctpMap=0x656700, numOfMessages=<optimized out>, rmrMessageBuffer=..., ts=...) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1618
#11 0x000000000041f509 in listener (params=0x7fffffff8fb8) at /opt/e2/RIC-E2-TERMINATION/sctpThread.cpp:1110
#12 0x00007ffff71d8df4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff72ec609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#14 0x00007ffff7ed0353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

      The crash seems to stem from the buildPrometheusList function, particularly at the following line: 

       

       

      peerInfo->counters[IN_INITI][MSG_COUNTER][(ProcedureCode_id_E2setup)] = &prometheusFamily->Add({{peerInfo->enodbName, "IN"}, {"SetupRequest", "Messages"}});

      As a temporary fix, I implemented a non-null check and wrapped the metric collection code in a try-catch block:

       

      void buildPrometheusList(ConnectedCU_t *peerInfo, Family<Counter> *prometheusFamily) {
    if (peerInfo == nullptr || prometheusFamily == nullptr) {
        return;
    }    if (peerInfo->enodbName[0] == '\0') {
        return;
    }    

    try {
        peerInfo->counters[IN_INITI][MSG_COUNTER][(ProcedureCode_id_E2setup)] = &prometheusFamily->Add({{peerInfo->enodbName, "IN"}, {"SetupRequest", "Messages"}});
        // rest of the metric collecting code
    } catch (const std::exception& e) {
        mdclog_write(MDCLOG_ERR, "Exception caught in %s: %s", __FUNCTION__, e.what());
    }
}

      Please let me know if you need any additional information, or if you have any idea on what might be the root cause for the crash. I can also provide the complete log (16.3MB) upon request. Thanks for your help!
       

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            himanshup Himanshu Purohit
            tchyang Tianchang Yang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: