Assigning specific CPU cores to system processes (using cpuset systemfile)

[camilo2]

Hello everyone.

I’m using the Jetson Orin AGX 32GB 12 Cores, and I need to assign only 2 cores (10 and 11) to all the system processes, so the other ones are free for my application.

Currently I’m doing it like this:

• Go to the cpuset file system:

cd /sys/fs/cgroup/cpuset

• Create a group and assign resources:

mkdir -p sys
echo 10,11 > sys/cpuset.cpus
echo 0 > sys/cpuset.mems
echo 0 > sys/cpuset.cpu_exclusive

• Move all processes running in the default group to the new “sys” group:

for T in `cat tasks`; do echo "Moving " $T; /bin/echo $T > sys/tasks; done; true

Even though not all the processed can be moved (something related to be kernel processes, or something like that that I’ve read in google).

So my new idea was to change the default group to only allow cores 10 and 11. When I check the default assigntaion, it’s:

# cat /sys/fs/cgroup/cpuset/cpuset.cpus 
0-11

But when I try to override it I get:

# echo 10,11 > /sys/fs/cgroup/cpuset/cpuset.cpus
bash: echo: write error: Permission denied

Even being sudo.

I wonder if there is a way to “unlock” the default cpuset.cpus file, or if you know another way to assign CPUs 10 and 11 to all the processes of the system (and new processes launched in the default group).

[DaneLLL]

Hi,
We suggest set isolcpus in extlinux.conf to not to schedule tasks on the cores. Here is an example:
How to re-enable CPU Cores after isolcpus

And call taskset(1) - Linux manual page to assign processes to the cores.

[camilo2]

Hey!, thanks for the quick response.

Effectively if I add isolcpus=2-11 to APPEND in the file /boot/extlinux/extlinux.conf and restart, I see that even if I launch a commands that use 12 threads it only uses 0 and 1. For example, if I run:

stress -c 12

I get:

And if I use taskset I can assign one cpu to run my command, like:

taskset -c 8 stress -c 12

I get:

But now another problem rised. I can not set more than one core with taskset, for example using -c 8,9,10 or -c 8-10. If I run:

taskset -c 8,9,10 stress -c 12

I get:

Aparently it randomly take one of the selected cores instead of all of them.

This is what I get when I run the same command without the isolcpus piece of code:

[ShaneCCC]

I got below from stress-ng

taskset -c 2-5 stress -c 0
stress-ng: info:  [9465] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [9465] dispatching hogs: 8 cpu

[camilo2]

What is your configuration with isolcpus?
Can you share your /boot/extlinux/extlinux.conf?
And how does htop look when you don’t add taskset?

[ShaneCCC]

I didn’t modify the extlinux.conf
The “stress-ng --cpu 0” shows all cpu full run.

[camilo2]

It makes sense, the same here.

But try to do this: set isolcpus=2-7, restart, then run:

stress -c 8

Yo should see is running on cpus 0 and 1.

Then stop that command and run:

taskset -c 2-5 stress -c 8

And check what you see in htop.

[ShaneCCC]

Maybe set CONFIG_CPU_ISOLATION in kernel config to verify.

[camilo2]

Did you get the same results?

And what should I do about that CONFIG_CPU_ISOLATION? Where to modify it, which file and whico procedure?

Btw I replicated the problem in another two Jetson Orin AGX, and got exactly the same problem.

[ShaneCCC]

Yes, I get the same result after adding the isolcpus to extlinux.
You can add the CONFIG_CPU_ISOLATION to tegra_defconfig at …/kernel/kernel-5.10/arch/arm64/configs

[camilo2]

The unique directory where I see that structure is:

/usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10/arch/arm64

And this is the content:

# ls /usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10/arch/arm64
boot  crypto  include  Kbuild  Kconfig  Kconfig.debug  Kconfig.platforms  kernel  kvm  lib  Makefile  mm  net  xen

Is it the Kconfig file? If yes, should I just add config CONFIG_CPU_ISOLATION or something like that? I’d appreciate a more detailed explanation of how to add that configuration.

In addition and just to make sure, it’s something that I can just change and restart, right? Or do I need to re build the kernel.

[ShaneCCC]

It’s in the kernel source instead of the kernel header file.
This document tells how to download and build it.

Suppose need to rebuild the Image to replace it at /boot/Image

BTW, do you have any test on any others platform to reference the behavior?