-
Bug
-
Resolution: Done
-
Medium
-
None
-
None
-
minikube on Ubuntu 22.04 . Kubernetes version -
code:bash
{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.InfoKustomize Version: v4.5.4
{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:51:25Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.InfoWARNING: version difference between client (1.24) and server (1.26) exceeds the supported minor version skew of +/-1
code
minikube on Ubuntu 22.04 . Kubernetes version - code:bash $ kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info {Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info {Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:51:25Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.24) and server (1.26) exceeds the supported minor version skew of +/-1 code
In the code the number of CPUs is hard coded to 3. And listener function is started in each of these threads which takes a pointer sctp_params.
num_cpus = 3;
std::vector<std::thread> threads(num_cpus);
for (unsigned int i = 0; i < num_cpus; i++) {
threads[i] = std::thread(listener, &sctpParams);
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(i, &cpuset);
int rc = pthread_setaffinity_np(threads[i].native_handle(), sizeof(cpu_set_t), &cpuset);
if (rc != 0) {
mdclog_write(MDCLOG_ERR, "Error calling pthread_setaffinity_np: %d", rc);
}
}
There are parts of listener function that can get called from multiple threads without proper lock/protection.
For example in each of the threads following code could get called simultaneously
void listener(sctp_params_t *params) { ... } else if (params->rmrListenFd == events[i].data.fd) { // got message from XAPP //num_of_XAPP_messages.fetch_add(1, std::memory_order_release); num_of_messages.fetch_add(1, std::memory_order_release); if (mdclog_level_get() >= MDCLOG_DEBUG) { mdclog_write(MDCLOG_DEBUG, "new RMR message"); } if (receiveXappMessages(params->sctpMap, rmrMessageBuffer, message.message.time) != 0) { mdclog_write(MDCLOG_ERR, "Error handling Xapp message"); } } else if (params->inotifyFD == events[i].data.fd) { ... }
This is a potential race condition.receiveXappMessage takes as input unprotected members from sctp_params. Most likely so far we have not observed this is mainly because if the code runs on a single CPU, this problem won't be observed. Also, we are already using epoll_ctl to handle multiple events so it is suffice to call this function in a single thread. We should be only running a single listener with epoll.
