Patching CentOS 7 (and overcoming problems)

So I was working on patching some of my Icinga infrastructure at work, and it seems that sometimes libyajl breaks things, as illustrated below

root@icingasatellite ~]# yum update
Loaded plugins: fastestmirror, rhnplugin
This system is receiving updates from RHN Classic or Red Hat Satellite.
Loading mirror speeds from cached hostfile

  • epel:
    Resolving Dependencies
    --> Running transaction check
    ---> Package icinga2.x86_64 0:2.10.4-1.el7.icinga will be updated
    ---> Package icinga2.x86_64 0:2.10.5-1.el7.icinga will be an update
    ---> Package icinga2-bin.x86_64 0:2.10.4-1.el7.icinga will be updated
    ---> Package icinga2-bin.x86_64 0:2.10.5-1.el7.icinga will be an update
    --> Processing Dependency: for package: icinga2-bin-2.10.5-1.el7.icinga.x86_64
    Traceback (most recent call last):
    File "/bin/yum", line 29, in
    yummain.user_main(sys.argv[1:], exit_code=True)
    File "/usr/share/yum-cli/", line 375, in user_main
    errcode = main(args)
    File "/usr/share/yum-cli/", line 239, in main
    (result, resultmsgs) = base.buildTransaction()
    File "/usr/lib/python2.7/site-packages/yum/", line 1198, in buildTransaction
    (rescode, restring) = self.resolveDeps()
    File "/usr/lib/python2.7/site-packages/yum/", line 893, in resolveDeps
    CheckDeps, checkinstalls, checkremoves, missing = self._resolveRequires(errors)
    File "/usr/lib/python2.7/site-packages/yum/", line 1025, in _resolveRequires
    (checkdep, missing, errormsgs) = self._processReq(po, dep)
    File "/usr/lib/python2.7/site-packages/yum/", line 350, in _processReq
    CheckDeps, missingdep = self._requiringFromTransaction(po, requirement, errormsgs)
    File "/usr/lib/python2.7/site-packages/yum/", line 680, in _requiringFromTransaction
    File "/usr/lib/python2.7/site-packages/yum/", line 5280, in update
    availpkgs = self._compare_providers(availpkgs, requiringPo)
    File "/usr/lib/python2.7/site-packages/yum/", line 1648, in _compare_providers
    bestnum = max(pkgresults.values())
    ValueError: max() arg is an empty sequence

Turns out the secret is simply to install yaljl and yajal-devel and then I can patch successfully, really surprised nobody else out there has run into this yet but its the second time in a month I have had it happen when patching.

Simple CI with Chef

So I needed to work out a way to make a script I wrote recently be deployed across a whole host of systems, turns out the only option is Chef so I had to dive into it and read a bunch of stuff.  Also had to try a bunch of things and ended up with my own Chef server in the lab to test against.  Several hours of clicking and clacking later and I have my task worked out, so here it is.

First we need to create a new cookbook and drop a pretty simple default recipe in, all it does is make sure git is installed then clone a repo to /opt/nhlapi.

# Cookbook:: repo
# Recipe:: default
# Copyright:: 2018, The Authors, All Rights Reserved.
package 'git' do
  action :install

git '/opt/nhlapi' do
  repository 'git://'
  revision 'master'
  action :sync
default.rb (END)

Once we have the recipe we need a role to tell it what to do.

   "name": "repo-update",
   "description": "update chef from time to time",
   "json_class": "Chef::Role",
   "default_attributes": {
     "chef_client": {
       "interval": 1800,
       "splay": 60
   "override_attributes": {
   "chef_type": "role",
   "run_list": ["recipe[chef-client::default]",
   "env_run_lists": {

Create the role with # knife role from file repo-update.json  (or whatever you named the file to create the role from).

Now all that is left is to assign the role to the node so use #knife node edit itsj-cheftest.itscum.local  and assign the role and repo to the node we want

  "name": "itsj-cheftest.itscum.local",
  "chef_environment": "_default",
  "normal": {
    "tags": [

  "policy_name": null,
  "policy_group": null,
  "run_list": [


That is enough to get it working, you can kick back and watch it with # while :; do knife status ‘role:repo-update’ –run-list; sleep 120; done and wait to see it run in about 30 minutes based on the interval and splay values.  Speaking of which Interval is pretty self explanatory, but Splay not-so-much; Splay is used keep a bunch of nodes from all running at once basically so it doesn’t overwhelm a system that they might be checking into or otherwise digitally assaulting.

Simple Icinga2 Plugin

I’ve seen bits and pieces of the process of creating an Icinga2 (or Nagios) plugin, so here are my notes dumped straight from my brain.

First and foremost we need a script to call from Icinga, in this case I created a very simple Python script to simply get the version of LibreNMS running on my monitoring system.

import argparse
import requests
import json
import sys

parser = argparse.ArgumentParser(description='Process some integers.')

parser.add_argument('-H', action="store",dest="host", help='name of host to check')

#parser.add_argument('token', metavar='token', help='API token')
token = 'yourAPItokenGOEShere'
args = parser.parse_args()

host_check = 'http://''/api/v0/system'
headers = {'X-Auth-Token': token }
r = requests.get(host_check, headers=headers,verify=False)


json_string = r.text
parsed_json = json.loads(json_string)

system_status = parsed_json['status']
system_ver = parsed_json['system'][0]['local_ver']

if system_status == 'ok':
	ret = "status: "+system_status+" version:"+system_ver
elif system_status != 'ok':
	ret = "status: "+system_status+" version:"+system_ver

This is a pretty simple script, you could call it with ./ -H to see how it works.  With the script working the next portion is done in the command line, first create the directory that will later be referenced as CustomPluginDir

# mkdir -p /opt/monitoring/plugins

Now we need to tell Icinga2 about the directory, this is done in a few different places

in /etc/icinga2/constants.conf add the following

const CustomPluginDir = “/opt/monitoring/plugins”

and in /etc/icinga2/conf.d/commands.conf we add the following block

object CheckCommand "check-lnms" {
    command = [ CustomPluginDir + "/" ]

    arguments = {
        "-H" = "$address$"

The block above defines the custom command, specifies the script we created first and also passes the correct flags.  Now its time to add the check into the hosts.conf file, so place the following block into /etc/icinga2/conf.d/hosts.conf

object Host "itsj-lnms" {
        address = ""
        check_command = "check-lnms"

And with that we wait for the next polling cycle and should see something like the screenshot below

This is a highly simplistic example, but figuring it out was necessary for me because I had to port some existing code from Ruby to Python so I wanted to know exactly how a plugin was created to understand what values were returned and how it all fits together.

Homelab: Synology failure post-mortem

I take my homelab very seriously, its modeled after several production environments I have worked on over the years. What follows is my recap of events over a few weeks leading up to the total failure of my central storage system, my beloved Synology DS1515 hosting 5.5TB of redundant network storage. The first signs of problems cropped up on May 31st and culminated over the last week in June.

Read moreHomelab: Synology failure post-mortem

A cautionary tale of Git and Virtualbox

I have been trying to keep the code chops sharp since work doesn’t require it very often anymore, usually by working on polishing up projects I’ve started in the past but let fall by the wayside.  Last night was a bugfix session on imgur2pdf which I have been neglecting for a while now and specifically working on the resizing logic which was hosed up and created some ugly PDFs.  All told I think I spent about two and a half hours working on the code testing it over and over with galleries to make sure sizes were right and it made changes as appropriate to large dimension images.  Once I got things right to where I wanted them I pushed my commit up to the repo and took a break for a while, grabbing some tea and having a look at why the virtualbox VM I was working on wasn’t letting me copy/paste between it and my parent OS.

Some quick poking around and I realized I hadn’t installed the Guest Additions software so I loaded that up and rebooted the VM only to be graced with a solid black screen that was unresponsive. I rebuild the VM since it was mostly empty and try it a second time and the exact same thing happens when rebooting after installing.  Asked around on IRC and a buddy pointed out to me that this is kind of a known issue and there is a potential fix out for it, so I guess I know what I am doing with my evening tonight.

Thinking about this today made me realize that the old way of doing code would have probably cost me several hours of work and resulted in a great deal of profanity had I wiped out everything.  So let this be a lesson to anybody just getting started with Git, commit/push often as it beats the alternative of losing hours of work

Curl within a salt-state

So I have been looking all over for how to make this happen and finally figured it out, preserving it for anybody else who wants to kick off a curl in a salt state to say add something into monitoring or begin another process via an API call

# this will perform a curl on the target minion
    - name: >-
        curl -X POST -d '{"hostname":"","version":"v2c","community":"public"}' -H 'X-Auth-Token: 286755fad04869ca523320acce0dc6a4' ""

Right now this is just using testing data from my lab, but as long as you enclose all the salient data in ‘ or ” it should be fine

Strange behavior from Postman

I was working through changing my Saltstack configuration to work with LibreNMS and was working through adding devices via the API as opposed to using auto discovery and realized that basically the same query in curl works fine, but when I tried it with Postman it doesn’t work and acts like I never passed some of the values, observe!

as opposed to when done in curl

dword@DESKTOP:~$ curl -X POST -d '{"hostname":"","version":"v2c","community":"public"}' -H 'X-Auth-Token: 286755fad04869ca523320acce0dc6a4'
    "status": "ok",
    "message": "Device (18) has been added successfully"
dword@DESKTOP:~$ man curl

The only possible thing I can figure that is going since this is such an absurdly simple API query is that Postman does some kind of magic thats not plainly visible that changes how the data is received by the API.  This is moderately troubling because it gets me wondering what else they are doing with data and if there is some kind underhanded snooping going on, not that I’m working on anything too terribly sensitive other than helping myself become more lazy in the lab.  If I was tossing in a pile of headers I could see where the room for mistakes exists but with only three key/value pairs passed in data and the X-Auth-Token passed in headers I can’t really see any possible place I have messed up but sure enough we get the error about not specifying the version of SNMP for the add device call, so something definitely is hosed up somewhere.

Nginx permalinks and search fixed!

So I realized the other day I couldn’t have nice permalinks for some reason so I set out to try to figure out why and/or fix it as it had all worked fine before I migrated from Apache to Nginx a month or so ago.  Unfortunately it seems like there were plenty of sites talking about a fix of changing the location block around a bit, however I quickly realized that while the suggested fixes would indeed let me change to non-default permalink styles it invariably broke the ability to search the website which kind of inhibits me using it for storing thoughts and fixed and things and being able to recall them with a quick search.  Finally I stumbled across the golden bullet for this problem and I will provide it so that hopefully nobody else spends an hour and a half of their Sunday messing with this when there are QSOs to be made!

Original location block

location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules

and now the updated one

location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                #try_files $uri $uri/ =404;
                try_files $uri $uri/ /index.php?q=$uri&$args;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules


And with a quick restart of nginx (that took a few tries because I’m too used to systemctl over service already) and ta-da I can search again and have the fancy pants permalinks that mean search engines can better index my ramblings.  Oddly enough as I was testing this I saw a spider doing its thing while I was looking at the access logs for the server, pretty interesting that a spider would be active on a sunday at like 1145 EST.

Network Manager and OpenVPN

It blows my mind that Network Manager is still as bad as ever, I just finished up getting my new phone aimed at the home VPN when I remembered that the laptop lost all the old settings in my switch to Fedora so I figured I would give it a spin and see if somehow NM had been fixed.  A few minutes and some profanity later and it seems it STILL is unable to properly load up .ovpn profiles and parse out the various bits into the fields they need to go.  Even when I manually split up the keys and certs and all that it only worked halfway, I could connect to the VPN but was unable to browse the internet over it or even access resources local to the VPN server itself.  Fortunately the command line comes to the rescue again, all I had to do was tell openvpn itself where the config was and it did all the legwork that the abomination known as Network Manager failed to do.  For those who might care the proper way to invoke it is as follows

sudo openvpn --config /home/user/openvpn.ovpn

Now I just have to make a handy way to suppress the output, give me a status indicator and kill off the connection when I am done with it…

Successful Upgrade is Successful

I would say I can’t believe I’m typing this from a successful full upgrade from Fedora 23 to 24 but I’m not since I am at work and they frown upon me pecking away on my personal laptop, but I am still amazed that it was an absolutely painless process to upgrade from 23 to 24 with dnf.  In prior years it was almost always advisable to reinstall rather than attempt an upgrade from one major release to the next but the fine folks over at Fedora seem to have hit a home run on this one.  Sure it took a while to apply everything but the moment of truth (or reboot) came and passed and all I got was my normal login screen, no fancy explosions of failed video drivers, no corrupted profiles or missing files; it went so smooth I almost didn’t think it worked until I checked the redhat-release file and verified that it was in fact on the 24 release.

Close Bitnami banner