• AMD Athlon 64 X2 6000+ 3.0GHz 125W
• 8GB DDR3

• AMD Ryzen 1600 3.2GHz 65W
• 32GB DDR4 2400
• APC BN700MC Back-UPS 700 VA (420W)
• ATI Radeon X600 ($4.99 cheap PCIe graphics card)

• Kernel version 4.10
• gcc version 6.3.0

• Kernel version 4.11.9
• gcc version 7.1.1

Command:

bonnie++ -s 3g

Executed on a VM with 768MB of RAM.

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
hathor           3G   748  98 85183  23 63873  27  2543  96 295344  57  1971 195
Latency             23852us    9710ms    6711ms    9268us   10251us   43389us
Version  1.97       ------Sequential Create------ --------Random Create--------
hathor              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             15314us     591us     778us    3577us     913us     217us
1.97,1.97,hathor,1,1499995713,3G,,748,98,85183,23,63873,27,2543,96,295344,57,1971,195,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,23852us,9710ms,6711ms,9268us,10251us,43389us,15314us,591us,778us,3577us,913us,217us

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
hathor           3G   752  97 100045  19 97967  23  2584  98 737193  70  3204  57
Latency             30917us     656ms     235ms    9763us    4836us    7188us
Version  1.97       ------Sequential Create------ --------Random Create--------
hathor              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              9398us     595us     647us    1329us      48us     226us
1.97,1.97,hathor,1,1499994609,3G,,752,97,100045,19,97967,23,2584,98,737193,70,3204,57,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,30917us,656ms,235ms,9763us,4836us,7188us,9398us,595us,647us,1329us,48us,226us

Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
hathor           3G  1591  99 195864   9 192590  10 +++++ +++ 2287940  56 +++++ +++
Latency             13792us    3537us    3623us    3168us    1356us    1846us
Version  1.97       ------Sequential Create------ --------Random Create--------
hathor              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              2731us     267us     257us     122us      25us     122us
1.97,1.97,hathor,1,1500205275,3G,,1591,99,195864,9,192590,10,+++++,+++,2287940,56,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,13792us,3537us,3623us,3168us,1356us,1846us,2731us,267us,257us,122us,25us,122us

• Install base F26 server.
• dnf update

• dnf install \
    vim \
    qemu-system-x86 \
    qemu-kvm \
    apcupsd \
    nfs-utils \
    libvirt-daemon-qemu \
    libvirt-client \
    ssmtp

• Edit /etc/default/grub:
  • Remove rhgb
  • Decrease timeout
• grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
• Add /etc/sysconfig/network-scripts/ifcfg-br0:

  DEVICE=br0
  TYPE=Bridge
  DELAY=0
  NM_CONTROLLED=no
  ONBOOT=yes
  BOOTPROTO=static
  IPADDR=192.168.16.20
  NETMASK=255.255.255.0
  GATEWAY=192.168.16.1

• In /etc/sysconfig/network-scripts/ifcfg-enp7s0, change BOOTPROTO to none, and add BRIDGE=br0 and NM_CONTROLLED=no
• chkconfig network on
• edit /etc/apcupsd/apcupsd.conf and /etc/apcupsd/apccontrol
• systemctl enable apcupsd.service
• systemctl enable libvirt-guests
• systemctl enable nfs-server.service
• systemctl enable rpcbind
• firewall-cmd –add-service=nfs –permanent
• Do fixes in https://bugzilla.redhat.com/show_bug.cgi?id=1472062 (disable SELinux and reduce stop timeout of apcupsd service to 10s) to allow apcupsd to work properly
• Configure ssmtp in /etc/ssmtp/ssmtp.conf

I think this is a Fedora / systemd problem. apcupsd will detect the AC power failure and initiate a shutdown (shutdown -h -H now via apccontrol). But apccontrol killpower does not seem to be called. Also, during shutdown, systemd reports “A stop job is running for APC UPS Power Control Daemon for Linux”. Not sure if this is because the apcupsd service was what kicked off the shutdown command, so it wasn't able to be stopped.

Bug report filed: https://bugzilla.redhat.com/show_bug.cgi?id=1472062 Problem of not cutting UPS power seems to be due to /etc/apcupsd/powerfail file not getting created which might be because of SELinux. Other part of the problem (hanging on stopping the apcupsd service) still remains but is worked around by changing the service timeout to 10s.

Three times now my server has stopped responding and shows a blank console. After resetting and going into the UEFI setup the Western Digital M.2 drive no longer shows up. After a poweroff and cold boot, the drive reappears in UEFI setup and I have to reset my boot settings to select it as the default boot entry. The system appears to run properly after this until the next time.

The problem I'm observing seems very similar to this: https://superuser.com/questions/1194478/ssd-suddenly-becomes-unreadable-how-to-diagnose

After the third time this happened, on 2017-12-13, I updated the UEFI firmware on the ASRock motherboard to version 3.30. The upgrade was successful, and after resetting my options (particularly re-enabling SVM and power on after AC loss), the system appears to be running properly again now.

On 2017-12-15 the system froze again. I disabled “C6 Mode” in the UEFI setup and started up again.

2017-12-17: I have not observed the problem since disabling “C6 Mode”, but I have a feeling it will still come back and could be related to NVMe APST modes. Similar problem here: https://bbs.archlinux.org/viewtopic.php?id=232692. I added kernel parameter:

nvme_core.default_ps_max_latency_us=0

Now nvme get-feature -f 0x0c -H /dev/nvme0n1 shows APST is disabled.

Turned “C6 Mode” back on but have not observed the M.2 drive disappearing problem again with APST disabled.

A few times after fixing the M.2 drive disappearing issue, my server has frozen. One of the times I caught the console output which listed several messages like “watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [worker:14788]”, and one “INFO: rcu_sched detected stalls on CPUs/tasks:” … “rcu_sched kthread starved for 19150377 jiffies!”. I'm not sure what is causing this.

On 2018-01-08, I upgraded from kernel 4.14.4 to 4.14.11 and added “consoleblank=0” to the kernel command line so if this happens again hopefully I will not lose console output.

2018-01-09: Igor seems to be experiencing the same bug and pointed me to https://bugzilla.kernel.org/show_bug.cgi?id=196683. I will probably disable C6 in UEFI setup again.

2018-01-15: Got the freeze again after disabling C6 mode in setup. I looked deeper in setup options and found the buried “Global C-State Control” option that Igor and Mike had disabled so I disabled that as well. No freezes since then.