Uploaded image for project: 'Near Realtime RAN Intelligent Controller'
  1. Near Realtime RAN Intelligent Controller
  2. RIC-436

e2Mgr panic when processing E2 setup

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Highest Highest
    • Bronze-R4
    • None
    • e2mgr
    • None
    • RICP_T4 Sprint B.D6

      When testing O-RAN Bronze E2 setup from O-DU, the E2Mgr went into panic while processing E2 Setup message. Problem repeatable. Artifact versions: E2Mgr ric-plt-e2mgr:4.4.10, and E2T ric-plt-e2:4.0.10.

      Setup: deploy E2Mgr; deploy E2T and seeing E2_TERM_INIT message processed by E2Mgr; deploy O-DU.
      Issue: seeing O-DU setup SCTP conn w. E2T; O-Du sends E2 Setup to. E2T; E2T forwards to RMR; E2Mgr receives msg over RMR, parses msg, handles, then panic due to index out of range.

      The E2 setup msg from O-DU contains below info.
      <E2AP-PDU>
      <initiatingMessage>
      <procedureCode>1</procedureCode>
      <criticality><reject/></criticality>
      <value>
      <E2setupRequest>
      <protocolIEs>
      <E2setupRequestIEs>
      <id>3</id>
      <criticality><reject/></criticality>
      <value>
      <GlobalE2node-ID>
      <gNB>
      <global-gNB-ID>
      <plmn-id>13 F1 84</plmn-id>
      <gnb-id>
      <gnb-ID>
      00000000000000010000001000000001
      </gnb-ID>
      </gnb-id>
      </global-gNB-ID>
      </gNB>
      </GlobalE2node-ID>
      </value>
      </E2setupRequestIEs>
      </protocolIEs>
      </E2setupRequest>
      </value>
      </initiatingMessage>
      </E2AP-PDU>

      And below is the log dump from e2mgr:

      root@bronze-int-ric:~/dep-0526# kubectl logs -f -n ricplt deployment-ricplt-e2mgr-57f78fb687-rq6tz
      {"crit":"INFO","ts":1590981273284,"id":"E2Manager","msg":"#app.main - Configuration {logging.logLevel: info, http.port: 3800, rmr:

      { port: 3801, maxMsgSize: 65536}

      , routingManager.baseUrl: http://service-ricplt-rtmgr-http:3800/ric/v1/handles/, notificationResponseBuffer: 100, bigRedButtonTimeoutSec: 5, maxRnibConnectionAttempts: 3, rnibRetryIntervalMs: 10, keepAliveResponseTimeoutMs: 360000, keepAliveDelayMs: 120000, e2tInstanceDeletionTimeoutMs: 0, globalRicId:

      { plmnId: 131014, ricNearRtId: 556670}

      ","mdc":{"time":"2020-06-01 03:14:33.284"}}
      redis: 2020/06/01 03:14:33 sentinel.go:375: sentinel: discovered new sentinel="f30f04016169cfda8ed4f2dfd63753b624de37a1" for master="dbaasmaster"
      redis: 2020/06/01 03:14:33 sentinel.go:375: sentinel: discovered new sentinel="c6aff85a518c6f298707f466a1a6720e35795bea" for master="dbaasmaster"
      redis: 2020/06/01 03:14:33 sentinel.go:328: sentinel: new master="dbaasmaster" addr="10.244.0.127:6379"
      1590981273 6/RMR [INFO] ric message routing library on SI95/g mv=3 flg=00 (d762b36 4.0.2 built: Apr 27 2020)
      {"crit":"INFO","ts":1590981274293,"id":"E2Manager","msg":"#rmrCgoApi.Init - RMR router has been initiated","mdc":{"time":"2020-06-01 03:14:34.293"}}
      {"crit":"INFO","ts":1590981274294,"id":"E2Manager","msg":"#RnibDataService.GetE2TAddresses - addresses: [10.108.252.128:38000]","mdc":{"time":"2020-06-01 03:14:34.294"}}
      {"crit":"INFO","ts":1590981274294,"id":"E2Manager","msg":"#RnibDataService.GetE2TInstances - addresses: [10.108.252.128:38000]","mdc":{"time":"2020-06-01 03:14:34.294"}}
      {"crit":"INFO","ts":1590981274294,"id":"E2Manager","msg":"#RnibDataService.SaveE2TInstance - E2T instance address: 10.108.252.128:38000, podName: e2term, state: ACTIVE, associated RANs count: 0, keep Alive ts: 1590981274294917383","mdc":{"time":"2020-06-01 03:14:34.294"}}
      {"crit":"INFO","ts":1590981274295,"id":"E2Manager","msg":"E2TInstancesManager.ResetKeepAliveTimestampForAllE2TInstances - Done with reset","mdc":{"time":"2020-06-01 03:14:34.295"}}
      {"crit":"INFO","ts":1590981274295,"id":"E2Manager","msg":"#E2TKeepAliveWorker.Execute - keep alive started","mdc":{"time":"2020-06-01 03:14:34.295"}}
      1590981274 6/RMR [INFO] sends: ts=1590981274 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-rsm-rmr.ricplt:4801 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981274 6/RMR [INFO] sends: ts=1590981274 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-a1mediator-rmr.ricplt:4562 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981274 6/RMR [INFO] sends: ts=1590981274 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-e2term-rmr-alpha.ricplt:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
      {"crit":"INFO","ts":1590981277330,"id":"E2Manager","msg":"[RMR -> E2 Manager] #rmrCgoApi.RecvMsg - message

      { MType: 1100, Len: 81, Meid: \"\", Xaction: &\" 846273070077\", Payload: [&7b2261646472657373223a202231302e3130382e3235322e3132383a3338303030222c226671646e223a202231302e3130382e3235322e313238222c22706f645f6e616d65223a202265327465726d227d] }

      has been received for transaction id: 846273070077","mdc":{"time":"2020-06-01 03:14:37.330"}}
      {"crit":"INFO","ts":1590981277330,"id":"E2Manager","msg":"#E2TermInitNotificationHandler.Handle - E2T payload:

      {10.108.252.128:38000 10.108.252.128 e2term}

      - handling E2_TERM_INIT","mdc":{"time":"2020-06-01 03:14:37.330"}}
      {"crit":"INFO","ts":1590981277330,"id":"E2Manager","msg":"#RnibDataService.GetE2TInstance - E2T instance address: 10.108.252.128:38000, state: ACTIVE, associated RANs count: 0, keep Alive ts: 1590981274294917383","mdc":{"time":"2020-06-01 03:14:37.330"}}
      {"crit":"INFO","ts":1590981277330,"id":"E2Manager","msg":"#E2TermInitNotificationHandler.Handle - E2T Address: 10.108.252.128:38000 - E2T instance has no associated RANs","mdc":{"time":"2020-06-01 03:14:37.330"}}
      1590981305 6/RMR [INFO] sends: ts=1590981305 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-rsm-rmr.ricplt:4801 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981305 6/RMR [INFO] sends: ts=1590981305 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-a1mediator-rmr.ricplt:4562 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981305 6/RMR [INFO] sends: ts=1590981305 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-e2term-rmr-alpha.ricplt:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981336 6/RMR [INFO] sends: ts=1590981336 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-rsm-rmr.ricplt:4801 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981336 6/RMR [INFO] sends: ts=1590981336 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-a1mediator-rmr.ricplt:4562 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981336 6/RMR [INFO] sends: ts=1590981336 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-e2term-rmr-alpha.ricplt:38000 open=0 succ=0 fail=0 (hard=0 soft=0)

      1590981367 6/RMR [INFO] sends: ts=1590981367 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-rsm-rmr.ricplt:4801 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981367 6/RMR [INFO] sends: ts=1590981367 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-a1mediator-rmr.ricplt:4562 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981367 6/RMR [INFO] sends: ts=1590981367 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-submgr-rmr.ricplt:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981367 6/RMR [INFO] sends: ts=1590981367 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=service-ricplt-e2term-rmr-alpha.ricplt:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
      1590981367 6/RMR [INFO] sends: ts=1590981367 src=service-ricplt-e2mgr-rmr.ricplt:3801 target=10.108.252.128:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
      {"crit":"INFO","ts":1590981371120,"id":"E2Manager","msg":"[RMR -> E2 Manager] #rmrCgoApi.RecvMsg - message

      { MType: 12001, Len: 1204, Meid: \"gnb:311-048-00010201\", Xaction: &\" 846273070078\", Payload: [&31302e3130382e3235322e3132383a33383030307c3c453241502d5044553e0a093c696e6974696174696e674d6573736167653e0a09093c70726f636564757265436f64653e313c2f70726f636564757265436f64653e0a09093c637269746963616c6974793e0a0909093c72656a656374202f3e0a09093c2f637269746963616c6974793e0a09093c76616c75653e0a0909093c45327365747570526571756573743e0a090909093c70726f746f636f6c4945733e0a09090909093c45327365747570526571756573744945733e0a0909090909093c69643e333c2f69643e0a0909090909093c637269746963616c6974793e0a090909090909093c72656a656374202f3e0a0909090909093c2f637269746963616c6974793e0a0909090909093c76616c75653e0a090909090909093c476c6f62616c45326e6f64652d49443e0a09090909090909093c674e423e0a0909090909090909093c676c6f62616c2d674e422d49443e0a090909090909090909093c706c6d6e2d69643e31332046312038343c2f706c6d6e2d69643e0a090909090909090909093c676e622d69643e0a09090909090909090909093c676e622d49443e0a20202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202030303030303030303030303030303031303030303030313030303030303030310a20202020202020202020202020202020202020202020202020202020202020202020202020202020202020203c2f676e622d49443e0a090909090909090909093c2f676e622d69643e0a0909090909090909093c2f676c6f62616c2d674e422d49443e0a09090909090909093c2f674e423e0a090909090909093c2f476c6f62616c45326e6f64652d49443e0a0909090909093c2f76616c75653e0a09090909093c2f45327365747570526571756573744945733e0a090909093c2f70726f746f636f6c4945733e0a0909093c2f45327365747570526571756573743e0a09093c2f76616c75653e0a093c2f696e6974696174696e674d6573736167653e0a3c2f453241502d5044553e0a20202020202020202020202020202020202020202020202020202020203c2f676e622d49443e0a202020202020202020202020202020202020202020202020202020202020202020202020202020203c2f676e622d69643e0a2020202020202020202020202020202020202020202020202020202020202020202020203c2f676c6f62616c2d674e422d49443e0a20202020202020202020202020202020202020202020202020202020202020203c2f674e423e0a202020202020202020202020202020202020202020202020202020203c2f476c6f62616c45326e6f64652d49443e0a2020202020202020202020202020202020202020202020203c2f76616c75653e0a20202020202020202020202020202020202020203c2f45327365747570526571756573744945733e0a202020202020202020202020202020203c2f70726f746f636f6c4945733e0a2020202020202020202020203c2f45327365747570526571756573743e0a20202020202020203c2f76616c75653e0a202020203c2f696e6974696174696e674d6573736167653e0a3c2f453241502d5044553e0a] }

      has been received for transaction id: 846273070078","mdc":{"time":"2020-06-01 03:16:11.120"}}
      {"crit":"INFO","ts":1590981371121,"id":"E2Manager","msg":"#E2SetupRequestNotificationHandler.Handle - RAN name: gnb:311-048-00010201 - received E2_SETUP_REQUEST. Payload: 31302e3130382e3235322e3132383a33383030307c3c453241502d5044553e0a093c696e6974696174696e674d6573736167653e0a09093c70726f636564757265436f64653e313c2f70726f636564757265436f64653e0a09093c637269746963616c6974793e0a0909093c72656a656374202f3e0a09093c2f637269746963616c6974793e0a09093c76616c75653e0a0909093c45327365747570526571756573743e0a090909093c70726f746f636f6c4945733e0a09090909093c45327365747570526571756573744945733e0a0909090909093c69643e333c2f69643e0a0909090909093c637269746963616c6974793e0a090909090909093c72656a656374202f3e0a0909090909093c2f637269746963616c6974793e0a0909090909093c76616c75653e0a090909090909093c476c6f62616c45326e6f64652d49443e0a09090909090909093c674e423e0a0909090909090909093c676c6f62616c2d674e422d49443e0a090909090909090909093c706c6d6e2d69643e31332046312038343c2f706c6d6e2d69643e0a090909090909090909093c676e622d69643e0a09090909090909090909093c676e622d49443e0a20202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020202030303030303030303030303030303031303030303030313030303030303030310a20202020202020202020202020202020202020202020202020202020202020202020202020202020202020203c2f676e622d49443e0a090909090909090909093c2f676e622d69643e0a0909090909090909093c2f676c6f62616c2d674e422d49443e0a09090909090909093c2f674e423e0a090909090909093c2f476c6f62616c45326e6f64652d49443e0a0909090909093c2f76616c75653e0a09090909093c2f45327365747570526571756573744945733e0a090909093c2f70726f746f636f6c4945733e0a0909093c2f45327365747570526571756573743e0a09093c2f76616c75653e0a093c2f696e6974696174696e674d6573736167653e0a3c2f453241502d5044553e0a20202020202020202020202020202020202020202020202020202020203c2f676e622d49443e0a202020202020202020202020202020202020202020202020202020202020202020202020202020203c2f676e622d69643e0a2020202020202020202020202020202020202020202020202020202020202020202020203c2f676c6f62616c2d674e422d49443e0a20202020202020202020202020202020202020202020202020202020202020203c2f674e423e0a202020202020202020202020202020202020202020202020202020203c2f476c6f62616c45326e6f64652d49443e0a2020202020202020202020202020202020202020202020203c2f76616c75653e0a20202020202020202020202020202020202020203c2f45327365747570526571756573744945733e0a202020202020202020202020202020203c2f70726f746f636f6c4945733e0a2020202020202020202020203c2f45327365747570526571756573743e0a20202020202020203c2f76616c75653e0a202020203c2f696e6974696174696e674d6573736167653e0a3c2f453241502d5044553e0a","mdc":{"time":"2020-06-01 03:16:11.121"}}
      {"crit":"INFO","ts":1590981371122,"id":"E2Manager","msg":"#E2SetupRequestNotificationHandler.parseSetupRequest - payload: <E2AP-PDU>\n\t<initiatingMessage>\n\t\t<procedureCode>1</procedureCode>\n\t\t<criticality>\n\t\t\t<reject />\n\t\t</criticality>\n\t\t<value>\n\t\t\t<E2setupRequest>\n\t\t\t\t<protocolIEs>\n\t\t\t\t\t<E2setupRequestIEs>\n\t\t\t\t\t\t<id>3</id>\n\t\t\t\t\t\t<criticality>\n\t\t\t\t\t\t\t<reject />\n\t\t\t\t\t\t</criticality>\n\t\t\t\t\t\t<value>\n\t\t\t\t\t\t\t<GlobalE2node-ID>\n\t\t\t\t\t\t\t\t<gNB>\n\t\t\t\t\t\t\t\t\t<global-gNB-ID>\n\t\t\t\t\t\t\t\t\t\t<plmn-id>13 F1 84</plmn-id>\n\t\t\t\t\t\t\t\t\t\t<gnb-id>\n\t\t\t\t\t\t\t\t\t\t\t<gnb-ID>\n 00000000000000010000001000000001\n </gnb-ID>\n\t\t\t\t\t\t\t\t\t\t</gnb-id>\n\t\t\t\t\t\t\t\t\t</global-gNB-ID>\n\t\t\t\t\t\t\t\t</gNB>\n\t\t\t\t\t\t\t</GlobalE2node-ID>\n\t\t\t\t\t\t</value>\n\t\t\t\t\t</E2setupRequestIEs>\n\t\t\t\t</protocolIEs>\n\t\t\t</E2setupRequest>\n\t\t</value>\n\t</initiatingMessage>\n</E2AP-PDU>\n </gnb-ID>\n </gnb-id>\n </global-gNB-ID>\n </gNB>\n </GlobalE2node-ID>\n </value>\n </E2setupRequestIEs>\n </protocolIEs>\n </E2setupRequest>\n </value>\n </initiatingMessage>\n</E2AP-PDU>\n","mdc":{"time":"2020-06-01 03:16:11.122"}}
      {"crit":"INFO","ts":1590981371123,"id":"E2Manager","msg":"#E2SetupRequestNotificationHandler.Handle - E2T Address: 10.108.252.128:38000 - handling E2_SETUP_REQUEST","mdc":{"time":"2020-06-01 03:16:11.123"}}
      {"crit":"INFO","ts":1590981371124,"id":"E2Manager","msg":"#RnibDataService.GetE2TInstance - E2T instance address: 10.108.252.128:38000, state: ACTIVE, associated RANs count: 0, keep Alive ts: 1590981274294917383","mdc":{"time":"2020-06-01 03:16:11.124"}}
      panic: runtime error: index out of range

      goroutine 26 [running]:
      e2mgr/models.(*E2SetupRequestMessage).ExtractRanFunctionsList(0xc0001940d0, 0x0, 0xc0002555c0, 0xc000389cc8, 0x125105b, 0x14ec660)
      /opt/E2Manager/models/e2_setup_request_message.go:160 +0x47b
      e2mgr/handlers/rmrmsghandlers.E2SetupRequestNotificationHandler.setGnbFunctions(0xc0000107f8, 0xc0002db4d0, 0x1800200, 0xc00040c360, 0xc00040c300, 0x1808340, 0xc0004a9c00, 0xc000404380, 0xc00012d0e0, 0xc0001940d0, ...)
      /opt/E2Manager/handlers/rmrmsghandlers/e2_setup_request_notification_handler.go:136 +0x32
      e2mgr/handlers/rmrmsghandlers.E2SetupRequestNotificationHandler.buildNodebInfo(0xc0000107f8, 0xc0002db4d0, 0x1800200, 0xc00040c360, 0xc00040c300, 0x1808340, 0xc0004a9c00, 0xc000404380, 0xc00041c060, 0x14, ...)
      /opt/E2Manager/handlers/rmrmsghandlers/e2_setup_request_notification_handler.go:254 +0x19c
      e2mgr/handlers/rmrmsghandlers.E2SetupRequestNotificationHandler.handleNewRan(0xc0000107f8, 0xc0002db4d0, 0x1800200, 0xc00040c360, 0xc00040c300, 0x1808340, 0xc0004a9c00, 0xc000404380, 0xc00041c060, 0x14, ...)
      /opt/E2Manager/handlers/rmrmsghandlers/e2_setup_request_notification_handler.go:117 +0xa8
      e2mgr/handlers/rmrmsghandlers.E2SetupRequestNotificationHandler.Handle(0xc0000107f8, 0xc0002db4d0, 0x1800200, 0xc00040c360, 0xc00040c300, 0x1808340, 0xc0004a9c00, 0xc000404380, 0xc000130070)
      /opt/E2Manager/handlers/rmrmsghandlers/e2_setup_request_notification_handler.go:91 +0x42f
      created by e2mgr/managers/notificationmanager.NotificationManager.HandleMessage
      /opt/E2Manager/managers/notificationmanager/notification_manager.go:53 +0x2d9

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            idanshal Idan Shalom
            wrider Lusheng Ji
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: