Layered Performance Profiles with Tuned, Systemd, and Ansible

Although Tuned's website is looking pretty dated these days, I still believe tuned is useful for creating and managing performance profiles in a modern Linux system. Red Hat is still keeping it updated and it is still packaged in Fedora, Debian, and OpenSUSE. This article is about combining tuned, systemd, and ansible to simplify performance tuning on Linux servers.

So what is tuned, anyway?

As its website says, "TuneD is a system tuning service for Linux." To me, Tuned's killer feature is its "support (for) various types of configuration like sysctl, sysfs, or kernel boot command line parameters, which are integrated in a plug-in architecture."¹

Can't I just do that with config management?

I'm currently using Ansible to manage tuned profiles. Before tuned, I simply managed the performance settings directly. Here's why I switched to tuned:

Visibility from the host: With tuned, you can see exactly what profiles are loaded, and in what order. Here's an example from my homelab:

root@dreamcast:~# tuned-adm active
Current active profile: 00-riichi-base 01-riichi-baremetal 10-riichi-incus 99-riichi-dreamcast

If I'm not sure which values belong to which profiles, I can fish around in the /etc/tuned/profiles directories:

root@dreamcast:/etc/tuned/profiles# tree .
.
├── 00-riichi-base <- all servers, applied by base role
│   └── tuned.conf
├── 01-riichi-baremetal <- baremetal servers, applied by base role
│   └── tuned.conf
├── 10-riichi-incus <- incus hosts, applied by incus role
│   └── tuned.conf
└── 99-riichi-dreamcast <- this host only, applied by base role
    └── tuned.conf

Running multiple profiles is a design choice on my part. You can also include profiles in other profiles, but that hampers visibility.

Speaking of visibility, it gets even worse without tuned. Is that vm.swappiness sysctl set by a role? Was it a base setting? Do I actually understand Ansible or Puppet's merge behavior? The less I have to think about this kinda stuff, the more sanity I can preserve ;).

One place for multiple configs: Chances are, your current "tuning" Ansible or Puppet role is setting sysctl values, sysfs values, I/O schedulers, kernel boot arguments, etc. Each one of these tunables has its own syntax and its own associated services, which translates into writing handlers and learning syntax for each one.

With tuned, you can use the same syntax for everything, You don't need to write a bunch of separate handlers to run udevadm, grub-mkconfig, etc because tuned will manage that for you. Here's an example baremetal tuned profile:

# noop I/O scheduler for NVMEs
[disk_nvme0n1]
type=disk
devices=nvme0n1
elevator=none

# BFQ I/O scheduler for rotational disks
[disk_sda]
type=disk
devices=sda
elevator=bfq

[variables]
cpus=0,1,2,3,4,5,6,7

# Enable receive packet steering (RPS) to improve performance on single-queue NIC
[sysfs]
/sys/class/net/eno1/queues/rx-0/rps_cpus = ${f:cpulist2hex:${cpus}}
/sys/class/net/eno1/queues/rx-0/rps_flow_cnt = 4096

[sysctl]
rps_default_mask = ${f:cpulist2hex:${cpus}}
# encourage some swapping
vm.swappiness = 30

As you can see, we are able to set multiple tunables in a single file, using a single, simple config format. But what about that variables and cpulist2hex business? We'll talk about that in a later section.

Easy apply/rollback: When you're benchmarking (something I am admittedly pretty terrible at), you might want to toggle some settings on and off. Without tuned, you are again writing extra config management code to remove things and verify they're gone.

Tuned variables and functions

Tuned has a number of built-in functions.

In the above example, we're setting a variable based on the number of CPUs. Here's what it the template looks like before Ansible processes it:

[variables]
cpus={{ range(0, ansible_processor_vcpus | int) | join(',') }}

Then I'm using tuned's built-in cpu2listhex function, because RPS requires setting CPU masks.² Again, you calculate the value with config management, but it's clearer and more convenient to use the function.

Systemd service for applying profiles

In the unit file below, I invoke tuned-adm profile with multiple profiles. Per tuned-adm's man page: if more than one profile is given, the profiles are merged (in case of conflicting settings, the setting from the last profile is used) and the resulting profile is applied.

# /etc/systemd/system/set-tuned-profiles.service
[Unit]
Description=Apply all tuned profiles when tuned.service starts
After=tuned.service
Requires=tuned.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/tuned-adm profile {% for profile in tuned_profiles %}{{ profile }} {% endfor %}

[Install]
WantedBy=multi-user.target

In Conclusion

If you're managing Linux servers and you need a convenient way to organize your tunables, definitely check out tuned!

References

¹ https://tuned-project.org/

² https://support.scc.suse.com/s/kb/Improving-network-performance-using-Receive-Packet-Steering-RPS-1583239411093?language=en_US

closedwontfix.it