• What is Longhorn?

• What is NFS server?

• How to enable RWX access mode in Longhorn.

Issue Description:

NFS server itself has much problems with some versions of Linux kernels. In this link of official Longhorn docs you can find which kernel versions are not suggested to use longhorn on them. But apart of confusing with kernels, NFS may have some issues those cause disconnecting Longhorn from NFS server. And since most of this kind of problems are so difficult to solve the Devops operator may be confused about finding the root of problem.

Recently I needed to replace the entire of my Kubernetes cluster so I should took two types of backups:
1. Velero full cluster backup to store resources of cluster and,
2. Longhorn volumes backup to keep the persistent data safe.

When I restored the Longhorn volumes in the new cluster and Velero resources next, I realized there is a problem with volumes those access mode is rwx: The pods cant read or write on their volumes and a simple “ls” command would cause the pod hang out and freeze.
I checked the entire Longhorn system but I couldn’t find any problem with Longhorn share manager pods or any other related resources, and everything was completely OK. All the share manager pods were running and connected to NFS server (based on their logs).

After lots of searching and checking the pods I realized that not all pods have this problem. About 3 pods from 58 total pods are OK and can access to their rwx volumes; so the issue was not pervasive.
If I hadn’t notice that, I might thought that the problem is with kernel and downgrade my cluster kernel to earlier version.

Solution:

So i guessed there is kind of limitations with NFS servers. On the other hand the output of dmesg | grep nfs showed that some IP’s tried connect to the NFS server but timeout error occurred:

[334579.017116] nfs: server 10.233.14.62 not responding, timed out
[334579.528673] nfs: server 10.233.58.28 not responding, timed out
[334579.528825] nfs: server 10.233.48.185 not responding, timed out
[334579.528894] nfs: server 10.233.22.75 not responding, timed out
[334583.112813] nfs: server 10.233.46.135 not responding, timed out
...

All systemd services showed that NFS server is correctly run. After lots of research and surfing in Longhorn related documentations finally I noticed that NFS server has a max threads count limitation that causes this problem and prevents of connecting other pods to the NFS server.
The default value of max threads is 8. So I immediately increased that to 128 by doing such this:

$ sudo systemctl edit nfs-server

# Find this line and add 128 to end of the line.
...
[Service]
ExecStart=/usr/sbin/rpc.nfsd 128
...

# Save and exit.
$ sudo systemctl daemon-reexec
$ sudo systemctl restart nfs-server

By applying above changes, your NFS server max threads limitation will increase to 128 temporary and will be reverted after system reboot.
To increase the max threads count permanently, do this:

$ echo 128 > /proc/fs/nfsd/threads

At the end, restart all problematic pods and you can see that issue’s gone.

First of all, let me talk a little about race conditions. What are they, how they happen and what should we do about them.

Race conditions

Suppose a DB row that holds data of a random user of your website. you wanna add some budgets to its balance. In normal situation, you may do something like this: (Note that this code snippet is just for example. Using magic tricks and clean coding are not goals of this content.)

user = models.UserProfile.objects.get(username='Adam')
user.budget += 1000
user.save()

In the first look, everything looks normal. Now we wanna add some more complex processes right before saving the new budget of user. For example checking the available budget with user’s bank account balance through an API. Then if there was any problem with verifying user’s balance, the budget amount that was supposed to be added, will change. OK?

For handling such processes we have to do some actions in our web-app that takes a bit of time.
Now what will be happen if during verifying the user’s budget by bank API, another process override the content of the budget field?

In this situation two processes write on a specific field and race condition error occurs.
Race condition error may cause critical problems in such web-app those a great probability of overriding a row data by two parallel processes exists in them. Like bank apps, financials or etc.

In a bank transaction, the transaction must not cost any balance of user account until the transaction is over completely successful. That if it is not so, a user can exchange a single amount of money to other accounts more than one time.

The process of exchanging is as follows: When the bank received the exchange request from user, it checks the balance of account and if there were enough amounts of money, then submits the request.
Now exactly after the balance check process and before submitting the request, user can apply more requests to exchange money, and since no money has been deducted from balance yet (Because the process is not submitted yet and balance will deducted after the process has been submitted), the bank program will verify other request also. (If the bank doesn’t care about race conditions).

Ways to prevent

Now, that we now what is race condition and how it may occur, It’s time to know what can we do against these critical errors.

1. Using DB transactions

   The first step and must important one, is using Database transaction. although it does not prevent RC errors singly, but it’s the foundation of building such apps like bank applications that have parallel processes.

2. Row blocking

   The main task to do is designing a logic that blocks the DB row when we start to Updating or changing that.
   You can design this logic yourself. But frameworks like Django, provide enhanced solutions to apply this logic when you need.

Django select_for_update() method

Like other powerful web frameworks, Django provides lots of functionalities to apply complex database related logics in your app. One of them is select_for_update() method.
In short, when you use this method inside of an atomic transaction, It blocks the row data until the transaction closes.(Committed or rolled back)
To do this, we can just calling it from objects manager of a Model:

from django.db import transaction

with transaction.atomic():
    row = models.SomeModel.objects.select_for_update().get(pk=some_pk)

In above code, the row of some_model table will be blocked until the transaction context manager ends.

At the end, you can also implement this logic yourself. You can define a new field in your model named is_locked = models.BooleanField(default=False) or something like that and when you want to start editing the object you’ll make it True. (Don’t forget to change the is_blocked back to False after your changes committed, Otherwise the row will stay blocked forever!)

Conclusion

In summary, race conditions in web applications, especially those involving database transactions, can lead to severe issues, such as data inconsistency or even financial mistakes. Understanding how race conditions occur is key to preventing them, particularly in applications where critical processes, such as financial transactions, are involved.

By employing techniques like database transactions and row locking, such as Django's select_for_update(), you can prevent database inconsistencies.

We have two variables like below:
int a = 10; int b = 12;

Swap variable’s content and print the result.

At first, you think, wow! what a simple question. Let’s get free scores. But suddenly you notice something at the bottom of the question:

(Note: you can not use a any third variable!)

Wait! What? Without any third variable? How it’s even possible?

Here is how:

You may know that in compiled language like C++ everything will be converted to binary (Zero and One).
So you can convert the variable’s content to binaries also. By doing that and replacing the content of the variables with the binary one and printing them to output, You’ll see something like this:

...
std::cout<<"A contains: "<<a<<endl;
str::cout<<"B contains: "<<b;
...

Output:
A contains: 1010
B contains: 1100

And looks normal.

Step 2:

Now it’s time to play the magic trick. If we apply the XOR operation three times in each digits column of the binaries content, we could see that the variables just swapped automatically!
Don’t you believe it? Let’s give it a try!

“XOR” an abbreviation for “Exclusively-OR.” The simplest XOR gate is a two-input digital circuit that outputs a logical “1” if the two input values differ, i.e., its output is a logical “1” if either of its inputs are 1, but not at the same time (exclusively).

The first column of digits are two number ones. If we apply XOR on them, it will be 0. right?
Now we’ll put the operation result in place of the first A’s digit and A looks like 0010. Great. Then we need do the same process two more times and each time put the operation result in the opposite variable name (we put the result of XOR between A & B in A, So in the same way B & A in B. The order is important).

A: 1010 // Zero step.
B: 1100
A: 0010 // First step for the first colume of digits.
B: 1100 // Second step for the first colume of digits.
A: 1010 // Third step for the first colume of digits.

Step 3 (Final):

Great! now it’s enough to just do the same process for the 3 remaining columns of digits. (As done below)
Note that in the below codes, the process were applied at a same time for all columns to prevent boring calculations:

A: 1010 // Zero step. definition.
B: 1100
A: 0110 // First step
B: 1010 // Second step
A: 1100 // Last step. just look at the result!

Did you see that? At the end of the third step, the content of our two variables are completely swapped!
The strange thing is that the whole process will work for any binary values and there are no any exceptions!
You can try it by your self with different values and see that it works fine with any binary value.

One more note:

At the end, if you want to test on any other datatype apart from int datatype, It will work fine. BUT the important note is that you should convert your data into binary before you start the process, and after swapping the values convert them back to your considered datatype.

Conclusion:

In this article, we demonstrated a clever way to swap two variables without using a third variable in C++ by applying the XOR bitwise operation. This technique works by manipulating the binary representation of the variables through a series of XOR steps, resulting in the values being swapped.
Here, we used C++ language to explain our subject. But it works fine with all programming languages those you’re able to convert datatypes into binaries by them; Because as you know, binary isn’t depend on programming languages.