Speed up your QGIS build times

There are lots of ways of speeding your QGIS build times, some of them described in the INSTALL file, other on blog posts like Nathan's or on the QGIS Developers Mailing list.

But everyone wants more speed, right? So, I'm going to present one solution that I haven't encountered until two days ago so I thought of letting everybody know. Maybe all of you do it and I'm the last one to the party :).

I was planning to switch to the new quad core mobile processors (either Intel or AMD) and digging around I settled on a Thinkpad T480s. As I like to do my homework, the finest article I could find is from a laptop review site that I can wholeheartedly recommend: notebookcheck.net

From the article in the link above, I found out that I can do some fine tuning and utilize the full performance potential of my CPU by keeping the core at much higher frequencies for prolongued periods of time. You never get this out of the box because all the chips are shipped from the factory with a safety margin in their default operating voltage. This is all because there is a level of variation in every chip although their naming is the same (your i5-8250U is different then mine).

Knowing this, you can fine tune your chip and do undervolting with some specific tools. As a result, you can either:

• get some performance benefits without sacrificing temperature or power consumption
• get even more performance at the expense of some increase in temperature and in power consumption
• decrease your battery consumption and the chip temperature.

Your choice!

If your laptop cooling solution is good (the T480s has a nifty cooling system, as it was setup to support also a dedicated NVIDIA card) you can go with the "far more performance" mode as the increase in temperature is noticeable only when you look at the temperature metrics.
Does it bother you if you consume 10 W more while you laptop is plugged in? I am surely not bothered!

Out of the box, my QGIS building time under Linux was 39 minutes 12 s, with each core running stable at 2400MHz after few minutes.

Now I get 29 minutes 15 seconds, each core stabilizing at around 3200MHz, so I shaved around 10 minutes of my building time, which is quite a lot!

Look at my poor MacBook Pro from 2016.. running Linux of course.
Now I'm more than 50 Minutes faster. Can't believe it!

If you want to compare your build time, I used the following parameters under Debian Buster:

cmake -G Ninja -D CMAKE_BUILD_TYPE=Debug -D CMAKE_INSTALL_PREFIX=/home/web/qgis-master -D WITH_ASTYLE=ON -D WITH_APIDOC=ON -D WITH_CUSTOM_WIDGETS=ON -D WITH_QWTPOLAR=ON -D WITH_INTERNAL_QWTPOLAR=ON -D WITH_SERVER=ON -D SERVER_SKIP_ECW=ON -D ENABLE_TESTS=ON -D ENABLE_PGTEST=ON -D WITH_3D=False ..

My ThinkPad T480s with its 15W TDP i5-8250U even surpasses some units with the 45 W chip Core i7-7700HQ. Now that's really cool, ain't it?

To quote from the article above:

Undervolting: We managed to reduce the core voltage of our processor by 125 mV, which stabilizes the performance and reduce both the temperature as well as the consumption at the same time.

If you want another nice article of undervolting the T480s and the X1 Carbon 6th generation you can check this one.

How did I specifically did it?

I installed Windows on a stick, and with XTU I determined the stable undervolting parameters for my CPU:

• 'CPU' -90 mV (fate has it that my CPU was not as good as the one the guys from notebookcheck got as I didn't manage to go to 125mV like they did)
• 'CPU Cache' -90 mV
• 'GPU' -90 mV

I immediately noticed benchmark improvements under Windows which I won't post here as my target was and is to have the improvements under Linux.

For my Linux machine (Debian) things are not as easy to do as on Windows. Those bad boys from Intel didn't do a XTU for Linux.

I found out of this reverse engineered project and I'm using it despite the Disclaimer:

This tool may damage your hardware since it uses reverse engineered methods of MSR usage. Use it on your own risk.

My /etc/intel-undervolt.conf config looks like this:

# CPU Undervolting
# Usage: apply ${index} ${display_name} ${undervolt_value}
# Example: apply 2 'CPU Cache' -25.84

apply 0 'CPU' -90
apply 1 'GPU' -90
apply 2 'CPU Cache' -90
apply 3 'System Agent' 0
apply 4 'Analog I/O' 0

# TDP Alteration
# Usage: tdp ${short_term} ${long_term}
# Example: tdp 45 35
tdp 45 45
# Critical Temperature Offset Alteration
# Usage: tjoffset ${temperature_offset}
# Example: tjoffset -20
tjoffset -15

Notice that I made the temperature offset to -15. This means that on my specific chip with 100 degrees C limit I enforced the upper temperature limit at 85 degrees as I like to keep my notebook surface cool enough so it doesn't bother me.

There is a trick (learned after quite some time of trial and error) in that it won't work until you set your CPU scaling Governor in performance mode.

You can use cpupower to do it after installing it with apt-get install linux-cpupower.

Check first what's your current situation with cpupower frequency-info which should output something like:

analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 3.40 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 400 MHz and 3.40 GHz.
          The governor "powersave" may decide which speed to use within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 1.52 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes

Notice the The governor "powersave" may decide which speed to use within this range. part which tells you that you're in powersave mode.

By doing cpupower frequency-set -g performance you can put it in performace mode and your CPU will fly over the performance realms.

I've used the sensors utility by doing watch -n 1 sensors plus the watch -n 1 grep \"cpu MHz\" /proc/cpuinfo command to check the improvements and to see the CPU temperature.

The commands output:

Every 1.0s: sensors          thinkpad: Fri May 25 12:23:18 2018

iwlwifi-virtual-0
Adapter: Virtual device
temp1:            N/A  

pch_skylake-virtual-0
Adapter: Virtual device
temp1:        +66.0°C  

acpitz-virtual-0
Adapter: Virtual device
temp1:        +84.0°C  (crit = +128.0°C)

thinkpad-isa-0000
Adapter: ISA adapter
fan1:        4291 RPM

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +85.0°C  (high = +100.0°C, crit = +100.0°C)
Core 0:        +85.0°C  (high = +100.0°C, crit = +100.0°C)
Core 1:        +84.0°C  (high = +100.0°C, crit = +100.0°C)
Core 2:        +81.0°C  (high = +100.0°C, crit = +100.0°C)
Core 3:        +83.0°C  (high = +100.0°C, crit = +100.0°C)

Every 1.0s: grep "cpu ...  thinkpad: Fri May 25 12:22:57 2018

cpu MHz         : 3217.363
cpu MHz         : 3217.202
cpu MHz         : 3217.075
cpu MHz         : 3216.951
cpu MHz         : 3216.628
cpu MHz         : 3216.522
cpu MHz         : 3216.501
cpu MHz         : 3216.220

As I usually keep my notebook plugged in I created a systemd service to set it in performance mode when it boots with:

cat << EOF | tee /etc/systemd/system/cpupower.service
[Unit]
Description=CPU powersave

[Service]
Type=oneshot
ExecStart=/usr/bin/cpupower -c all frequency-set -g performance

[Install]
WantedBy=multi-user.target
EOF

systemctl enable cpupower.service

When I'm not plugged in, I simply do a cpupower frequency-set -g powersave to prolong the battery life.

Or.. better yet, you could install acpi with apt-get install acpi -y, add this to a bash script /root/switch_governor.sh with the following content:

plugged_in=$(acpi -a | cut -d' ' -f3 | cut -d- -f1)

if [ "$plugged_in" = "on" ]; then
    cpupower frequency-set -g performance
else
    cpupower frequency-set -g powersave
fi

Make it executable with chmod +x /root/switch_governor.sh and add to your root crontab (crontab -e -u root):

*/1 * * * * /root/switch_governor.sh

This checks each minute if your laptop is plugged in and sets the governor appropiately.