Remote Check

So I’ve been working on setting up a way to monitor a server at work to prevent my “assistant” from screwing up too badly and I’ve come up with a temporary solution until I can work out a much more robust script.

There are a few required elements, like the rc.head and rc.tail files which basically are the top and bottom of the email it sends out. Everything else is created and removed while the script is running making for a mostly clean experience. There needs to be at least (and ideally only) a one-way SSH trust from the testing machine to the target machine. I setup everything this way because the machine actually running the script is locked down to keep my boss and assistant from touching (see also: breaking) anything. Even if some fool removes the vixie-cron package this script continues to function until either a loss of network connectivity occurs or someone breaks the ssh trust (both of which in later versions will cause alert emails to go out)


# Script requires rc.head and rc.tail to be at least present, handy places
# to put some generic information to be included in each alert email

> errors.txt # this should zero out the file


# function: checksvc
# argument: service name to check
# returns: string statement 

SV="service $SVC status"
V=`ssh $HOST "$SV"`
        touch errors.txt

if echo $V | grep -q 'is running'
        #echo "$1 is running"
        echo ""
        echo "$1 is NOT running on host $HOST" >> errors.txt

checksvc httpd
checksvc memcached
checksvc mysqld
checksvc crond
checksvc postfix

if [ -s errors.txt ]
        # errors present
        cat rc.head > mail.msg
        cat errors.txt >> mail.msg
        echo "Scan run at $DATE" >> mail.msg
        cat rc.tail >> mail.msg
        mail -s '[ALERT]Services NOT running' -a errors.txt first.last@host.tld < mail.msg
        # no errors present
        echo ""

# lets clean some junk up
rm mail.msg

Git as a Backup System

So I have a co-worker on a coding project who has all but refused to use any kind of version control  ( and in general prefers to do most of his work in production ) so I’ve been kicking around ways to either force him to do the work, or at least make changes get committed to to a repository without his interaction at all. Took a little reading around to figure out the flags necessary but I managed to get everything working with shared ssh keys added into my bit-bucket settings as well as a cron job that runs at set intervals (in my case at about 10 am when the co-worker’s shift has ended) and pushes all changes including new files to the repository so that I can review what work he has done and the things he has broken. Unfortunately as I got all of this working it seems the web proxy between the server and the outside world shit the bed and I can’t push commits out to my external (and not under the control of the co-worker) repository.

For whatever reason by default bash doesn’t add timestamps to bash.history, so here is how to make that happen.

Add the following line to /etc/bashrc

export HISTTIMEFORMAT="%d/%m/%y %T "

and now when you use the history command you get nice clean timestamps, just keep in mind this is not retroactive so it will only work going forward.  If you have just done this you might need to back out of your current session log back in or create another term window before it will start showing in the new format.

552 11/11/12 17:22:27 vi bashrc
553 11/11/12 17:22:27 man history
554 11/11/12 17:22:27 help history

Account Info Script

Had to do a quick crontab job at work to grab user details out of /etc/passwd and email the details in a report, here is the resulting script

> account-report.txt
date > account-report.txt
echo -e "=== Account Report ===n" >> account-report.txt
hName=( $(awk '{print $1}' $Nlist) );
for name in "${hName[@]}"; do

        users=$(cat <<EOL
                cat /etc/passwd | grep "/bin/bash" | grep "[1-9][0-9][0-9]" | cut -d: --fields=1,5

        echo "Host: $name" >> account-report.txt
        ssh -o StrictHostKeyChecking=no ${name} "$users" >> account-report.txt
        echo -e "End of $name accountsnn" >> account-report.txt

mailx -s "Monthly Account Report" user@host.tld < account-report.txt

It grabs a list of hostnames from nodelist.txt and utilizes a ssh key-trust to connect to the hosts without supplying a password.  We lucked out bigtime that there was a trust already setup otherwise the whole process would have been overly complicated requesting permission to setup a trust in the first place.

Home Labs

Today’s random thought is on the concept of the home lab (from the IT worker perspective, not the Walter-White-Meth-Lab view).

I know several people at work who have home networks setup specifically for testing and learning, some are network labs with stacks of switches and routers while others have full blown SAN environments for use with ESX virtualization.  In my case my lab is neither elegant nor particularly impressive, it consists of two machines: a jumphost running Ubuntu and a KVM virtualization server built on CentOS.  This got me thinking about how some of my co-workers and others outside of the office that I have spoken to have no lab equipment at home what so ever, in fact some don’t even have a home computer to speak of.

How do people keep their skills sharp and more importantly learn new things if they don’t even have lab equipment at home.  It doesn’t even have to be fancy (like the previously mentioned SAN setup) but just something to get by doing light testing or even practicing for certification exams, yet there are people who don’t have anything.  I wonder if its due to having been in the field for far too long that they have decided they just don’t care to stay competitive anymore?  My home setup has even been useful for issues directly relating to work; currently its helped me build out a new and improved remediation script that will hopefully see production in the not-so-distant future.

Youtube Ripping Made Easy

12-11-2014 – Updated bash script syntax to reflect usage of –youtube-skip-dash-manifest to keep errors down when doing bulk processed lists of videos.

There are plenty of annoying windows tools that claim to be able to properly rip audio from Youtube, however I don’t trust most of them any further than I can throw a car.  Here is a dirt simple guide to ripping audio on Linux, you could even accomplish it on a headless server if you really felt like it.

The center piece of this process is youtube-dl which is just a python script which takes care of all the work of pulling down and converting the files to mp3.  Go ahead and pull down the script, and drop it somewhere that you can access (check your $PATH variable depending on your distro of choice) and don’t forget to chmod +x the file so that its executable without first invoking the Python interpreter.  At this point you also want to make sure you have ffmpeg installed as that’s what does all the leg work for youtube-dl.  Once you have that out of the way lets look at a sample use of this program to figure out how to streamline things so you don’t have to type out a pile of arguments every time you want to pull down a file.

youtube-dl -l --prefer-free-formats --extract-audio --audio-format mp3

Fortunately the developers of this app didn’t go ape with cryptic command line arguments, pretty much everything there is self explanatory except for -l, which to save everyone the time of checking the –help output is to give it a litteral file name instead of the pseudo-random one that it derives from the youtube id string.  Now to simplify things even more, lets make a bash script called ‘ytdl’ in the same location you placed youtube-dl earlier.


/usr/local/bin/youtube-dl $1 --youtube-skip-dash-manifest -q -o %(title)s.%(ext)s --prefer-free-formats --extract-audio --audio-format mp3

Make the script executable and now all you have to do is invoke ytdl <youtubeurl> and it will kick off the process.  Background the process for bulk conversions or better yet if you really want to make things slick create a php front end for the whole thing (but make sure to password protect it) so you can pull down mp3s from youtube.  The one caveat is some videos don’t convert over to mp3 but mp4 or other strange formats that won’t work with this, at this point I don’t have a real fix for this but I’m sure one is in the works somewhere.

OwnCloud bug

Not going to make this a long and complicated post, just wanted to toss this out there in case any body else finds themselves in my situation.  I was playing around with OwnCloud and enabled the Subphonic add-on and suddenly my install quits working completely citing some bogus Apache nonsense.  Not to be deterred I got on ssh and hit up the box and immediately checked /var/log/apache2/error.log and here is what I found

[Sun Jun 03 09:17:30 2012] [error] [client] PHP Fatal error: Class 'OCP\App' not found in /var/www/apps/subphonic/appinfo/app.php on line 10
[Sun Jun 03 09:18:55 2012] [error] [client] PHP Fatal error: Class 'OCP\App' not found in /var/www/apps/subphonic/appinfo/app.php on line 10

Just as I figured Subphonic was the culprit, however not being able to login to OwnCloud to disable it I was left with no choice but to either re-install the whole thing (which I didn’t want to do) or manually disable it in the database where all the config settings live.  Took maybe a minute or two of poking around to figure out the structure of the database and locate where the setting was stored, for simplicity sake all you need to do to disable it is connect to the mysql database and run the following query

UPDATE oc_appconfig SET configvalue = 'no' WHERE appid = 'subphonic' AND configkey = 'enabled';

And obviously you can change this to disable whatever random thing you happen to have enabled which breaks OwnCloud by changing the appid to the necessary name, hopefully this helps someone out there who may not be overly comfortable with command line SQL manipulation.


Fresh SCADA hacks /

Jesus H Christ on a pogo-stick, why are these things still being left attached to the damn internet?  People have been ragging on the utility industry to get their shit in gear and lock down their SCADA equipment for years now, especially Bruce Schneier.  I can’t even fathom how this has been left alone for so long and nobody has managed to take out the power and water to a major city like say Chicago.  I just hope those who do tread upon the various SCADA systems out there continue to be like pr0f and not some anti-social miscreant who thinks its a great idea to plunge a city into darkness for shits and giggles; at least until we finally realize this is a very real threat and address its accordingly.

Easy Pickin’s

A while back I decided it would be kind of neat to learn how to pick locks, I mean think of how sexy it looks in the movies and on TV when some hero takes a paperclip and a hair pin and gets into some restricted area in less than a minute with no light and hands covered in blood and sweat?  Sure it looks pretty awesome, but I can safely say that it will piss you right the hell off to get all but one pin to line up right then readjust your hand and reset the whole damn process, or drop your tension bar, or just spend tons of time trying to open a lock and have no damn progress at all.

So lets get down to the business of the reality of consumer grade locks and how well they resist compromise by a serious amateur at lock picking.  The first candidate was a Fortress brand padlock that I picked up at the local hardware store. My roommate initially had far more success with this lock than I did, however once I got over the urge to throw it across the room I was able to open it fairly fast with a gentle touch and a little effort.  Most disturbing about being able to open this lock was that it’s an off brand made by Master whom would love everyone to believe their locks are invincible.

Having one lock successfully defeated I decided it was time to give my Brinks lock box a try since I had actually lost the key for it a while back and wished to retrieve the contents of it without the aid of a crow bar or other brute instrument.  I was all prepared for a long fight with this device however after one pick change I managed to open it; repeated attempts with the pick yielded even faster results.

By this point I was quite pleased with myself and the efforts so far, so I decided to up the stakes and try my fireproof safe made by Sentry.  After removing all the contents of the safe I went to work on it trying to get it open, but about three to five minutes in I was getting a little frustrated with having to stand in a weird position to assault the lock so I turned the tools over to the roommate so I could rest.  No sooner had I sat down to let my back un-kink he turns around with a shit-eating grin and an open safe; the trick we figured out was that you didn’t have to get the lock to rotate completely to disengage but rather just enough so that it would clear the catch so in all likely hood not all the pins were protecting the contents.

Now we are just tickled pink with the fact that we can get into all these locks; its almost as sexy as in the movies except for the soundtrack was a little lacking and there was no attractive females present to be amazed at such lock-decimating prowess. Naturally we want to find another one to open so I find one of my unused trigger locks that came with one of my guns.  Immediately I had my apprehension since the word Master was stamped on the lock-side of the mechanism; I mean if their locks can be shot and not unlock then surely it must be a beast to pick requiring a lot of time right?  Fast-forward about 10 minutes of quality time sweating my ass off on the back deck and suddenly the cylinder rotates 90 degrees and the two halves of the trigger lock slide apart. I wasn’t quite sure I had just managed to open a Master brand lock being an extreme beginner to the world of lock picking, but sure enough I was able to replicate the process repeatedly with improvements to speed and reductions to wiggling necessary in order to align the pins and defeat the mechanism.

The whole afternoon of picking made me realize a few things, first of which is this is a fun and almost addictive hobby; each new lock defeated gives you the urge to run over to the nearest hardware store or big-box like Walmart and look for another lock to buy and work on picking.  More importantly than the fun of it was the fact that many of the commonly available devices that people rely on to keep things safe (like the Sentry brand fire safe that I mentioned earlier) were total jokes as far as the lock integrity is concerned.  If an unskilled hack such as myself can pick the lock in under 10 minutes (or in this case my equally unskilled roommate) then a serious criminal with years of experience is going to make even shorter work of it.

If you are in the market to buy a safe to protect something I suggest asking yourself this question: Which is more likely to happen, someone attacking the safe or the building its stored in burning to the ground around it?  In the case of fire I suspect any UL rated box will do the job of keeping your belongings safe provided you get one suited to the type of building that (hopefully wont) would be burning down with the safe nestled away inside.  However in case you happen to be siting on piles of drugs, conflict diamonds, the real Shroud of Turin or anything else you value put some time into researching suitable safes and their ratings. If you don’t feel like taking the short time to do that just look for a TL-15 rated safe or better if you are truly concerned about someone assailing it to gain access to its contents; a TL 15 is certified to stand up to 15 minutes of net working time on the lock with various tools and techniques before giving up the goodies inside.