Pictures of SysAdminDay Dinner 2014 are up!

Hi Everybodeee! Every year, the last Friday in July is System Administrator Appreciation Day. A SysAdmin is the person who keeps your servers serving, your network working, and your backups, um, backed up. Most people only deal with their SysAdmin when things go wrong, but on the last Friday in July they shower their SysAdmins with gifts, chocolate cake and ice cream.

If you’re a SysAdmin, aspire to be one, are friends with or married to one, or just want to see what SysAdmins look like, come to this special Ubuntu Hour and celebrate with us.

Sysadminus Windowservus

Sysadminus Emailservus

Sysadminus Databasus

Sysadminus Linuxservus

What SysAdmins look like

Usually, we celebrate SysAdminDay in Kitchener-Waterloo with Egg Rolls and Guy Ding at the Egg Roll King Restaurant, but this year Tony and his family will be on vacation so we have to find another venue. I’ve received some good suggestions already; let me know of any others in the comments.

CrankyOldBugger writes:

I know of a perfect place in St. Jacobs (Harvest Moon), seating-wise, but
no wi-fi.

What about the Williams at University Plaza? I’m just tossing out names
here…

This might be a bit out of the ordinary, but maybe we could use my house in
St. Jacobs. If it’s nice out, we could have a pool party. The pizza joint
around the corner is makes good stuff. Just a thought….

Tim Laurence suggests:

If Chinese food is in order I am a big fan of Lia Lia.

Otherwise we could grab some space at the Rum Runner. Their rooms are just
perfect for groups.

Nathan Fish offers:

Kam Yin is an excellent Chinese restaurant, family-run I believe. They don’t have a party room, though. How many are we expecting?

I don’t know, Nathan… So, Everybodeee, please register for the SysAdminDay Ubuntu Hour in Kitchener-Waterloo event on the Ubuntu LoCo Team Portal, or let me know you’re coming in the comments.

–Bob.

WordPress Hacked!

WordPress upgrades are famous for their ease of installation. Surprise! After upgrading the first installation most of the plugins were missing, and the theme was broken. A quick look at a directory listing showed that the plugins and themes were still installed. A quick look with a text editor showed some peculiar PHP code at the top of every .php file in the plugins folders. Surprise! This WordPress installation had been hacked! Fortunately, of the five instances of WordPress on this server, only two appeared to be affected. This Blog Is Not For Reading was not one of them.

Each .php file started with something like this:

<?php $zend_framework="\x63\162\x65(…)\x6e"; 
@error_reporting(0); 
zend_framework("", "\x7d\7(…)

Searching the Internet for “wordpress plugin invalid header zend_framework” I found a reference that makes me think this may have been possible because of a flaw in an earlier version of the WordPress code that handles comments. Most likely one of the comment fields (user name, e-mail, web address or the comment text itself) wasn’t properly sanitized, and allowed some kind of code injection (probably PHP injection, not a MySQL injection; the contents of the databases appeared to be untouched).

From the backups of the server it appeared that the breach occurred in or before August — either just before the release of WordPress 3.6 on 1 August 2013 or just before the release of WordPress 3.6.1 on 11 September 2013. If I had not been slack in upgrading to WP v3.6.1 then this breach might have been identified much sooner.

The upgrade to WordPress identified the modified files because the injected code preceded (and corrupted) the WP headers, and so WP v3.7 disabled any affected plugins and themes.

The Fix Is In

I renamed the directory containing the WordPress code, installed a fresh copy of WP3.7, cleaned and copied the wp-config.php and .htaccess files, uploaded a small image to create the wp-content/uploads hierarchy, then copied the upload folder (which didn’t contain any .php files), and then re-installed and re-configured the themes and plugins directly from the WordPress site.

Aside from the additional PHP code, there didn’t appear to be any other damage to the system. So I used the original wp-config.php (but cleaned, and with the “Authentication Unique Keys and Salts” section refreshed), and the new installation just used the existing databases. If there’s any malcode in the databases then that could re-infect the system, so I’m keeping an eye on it.

I have no idea what the malcode was intended to do. It didn’t corrupt the databases or anything else, but it’s possible it was acting as a keylogger or phoning home some other way. If I feel inclined I might try to de-obfuscate the injected code, but right now I don’t really feel like doing forensics.

Someone suggested using AppArmor to make the WordPress directories read-only. I’m not sure that locking down the WP directory is a good idea. The big new feature in WordPress 3.7 is its automatic update feature. If the WordPress directories are locked down then future security updates won’t be applied automatically. If there is an exploit and WordPress issues a new release to fix it, then a locked-down site will experience a delay in upgrading until the SysAdmin notices and upgrades manually (which is what used to happen before v3.7, but it seems a bad idea to delay upgrades when that’s no longer necessary). Also, the plugin and themes directories would be locked down, and they still require fairly frequent manual upgrades.

I sent the users on the affected sites this message:

While doing upgrades on WordPress yesterday I saw that your blog had been hacked sometime during or before August. I’ve fixed it (re-installed the code, copied your media library, re-installed themes and plugins). I don’t think any damage was done beyond the insertion of malicious code in some of the WordPress files. I don’t know what the action of that code was intended to be, but you should change your WordPress password just in case the bad guys captured it. You can change your password on the “Users, Your Profile page” once you’ve logged in.

After spending some time on Saturday fixing the two hacked WordPress sites I’m a little paranoid, and making sure to implement updates quickly. But a little paranoia is good — it’ll ensure I won’t become complacent again.

–Bob.

WordPress Hacks by Rafael Poveda is used under a Creative Commons — Attribution-NonCommercial-ShareAlike — CC BY-NC-SA license.

They say on the Internet Pictures, or it didn’t happen. So I just want to give you my assurances that the System Administrator’s Appreciation Day Dinner did actually happen!

27 July 2012, Egg Roll King Restaurant

System Administrator Appreciation Day Dinner, 27 July 2012 at the Egg Roll King Restaurant. Clockwise from left: Willem Jonkman, Bob Jonkman, Becky Nguyen, Wael Khawaja, Hamin (sp?), Rodney Martin, Sergiane Nascimento, Tim Laurence, Henrique Nascimento, Gui Nascimento.

Willem and Bob

Bob and Becky

Wael and Hamin

Rodney and Tim

Henrique and Gui

System Administrators out for dinner. Clockwise from left: Sergiane Nascimento, Tim Laurence, Henrique Nascimento, Gui Nascimento, Laurel L. Russwurm, Willem Jonkman, Bob Jonkman, Becky Nguyen, Wael Khawaja.

Cheese Wontons for dessert!

Bob Jonkman, System Administrator

–Bob.

Visit the official System Administrator Appreciation Day web site.

All pictures courtesy of Laurel L. Russwurm and released under a CC-BY-SA license.

Novell Netware 6.5

Swap Screens

ALT+ESC

List of screens

CTRL+ESC

Alternate console

CTRL+SHIFT+ALT+ESC

Shutdown prompt or the “Hung Console Menu”

CTRL+ALT+ESC

Entering the Netware Debugger Monitor

LSHIFT+RSHIFT+ALT+ESC

• he — Help on expressions
• hb — Help on breakpoints
• h — Help
• .h — Help on ‘dot’ commands
• .a — Display abend reason
• .c — Core Dump
• .d — Display page entry map for current domain
• .d address — Display page directory map for current domain
• .l offset — Display linear address at page map offset
• .lx — Display page offset and values
• .m — Display names and addresses of NLMs
• .p — Display process names and address
• .p address — Display address as a process control block
• .r — Display running process control block
• .s — Display screen names and addresses
• .s address — Display address as a screen structure
• .sem — List all semaphores with waiting processes
• .sem address — Display detailed semaphore inforation at address
• .t — Toggle ‘Developer Option’ on or off
• .v — Display version
• b — Display breakpoints
• bc number — Clear breakpoint number
• bca — Clear all breakpoints
• b= address — Set breakpoint at address
• br= address — Set read or write breakpoint at address
• bw= address — Set write breakpoint at address
• c — Change memory
• c entrypoint=hexdigits — Change bytes at entrypoint
• d — Display memory at the current stack pointer
• d address [length] — Display bytes at address for length bytes
• dl [+offset] address [length] — Display linked list for length nodes with next node address at offset
• f flag=value — Change flag to value. Valid flags: CF, AF, ZF, SF, IF, TF, PF, DF, or OF
• g — Go (back to Operating System)
• g breakaddress — Go, end at breakaddress
• i[b|w|d] port — Input Byte, Word, or Doubleword from port
• o[b|w|d] port=value — Output Byte, Word or Doubleword with value to port
• n — Display symbol names and NLMs
• n symbolname value — Create new symbolname with value
• n-symbolname — Remove symbolname defined with n
• p — Step through program code, skip calls
• s or t — Step through program code, enter calls
• q — Quit to DOS
• r — Display registers and flags
• u entrypoint [length] — See assembly code at entrypoint for length bytes
• v — View screens
• x — Exchange processor stack frames
• z expression — Evaluate expression
• ? entrypoint — Display reference to entrypoint
• register= entrypoint — Set register to entrypoint. Example: EIP=CSleepUntilInterrupt will interrupt the last NLM called. Valid registers: EAX, EBX, ECX, EDX, ESI, EDI, EBP, EIP, and EFL

See also the Novell Application Note The NetWare Internal Debugger. There’s more good information in the Novell TID 3193476: How to troubleshoot … an abended, unresponsive or crashed server.

Bob Jonkman <bjonkman@sobac.com>         http://sobac.com/sobac/
SOBAC Microcomputer Services              Phone: +1-519-669-0388
6 James Street, Elmira ON Canada  N3B 1L5  Cell: +1-519-635-9413
Software   ---   Office & Business Automation   ---   Consulting

Pictures are up!

System Administrators install servers

System Administrator Appreciation Day falls on 27 July this year, a global celebration of System Administrators around the world. To see what System Administrators do, have a look at some of the systems they have to administer.

After the KWLUG meeting in July, Becky asked me where System Administrator Appreciation Day is being held. “It’s a global event”, I said, but Becky was hoping for something a bit more local. So we decided to have a System Administrator Appreciation Day Dinner. What better way for System Administrators to celebrate than by treating ourselves to dinner?

System Administrators maintain web sites

So, if you’re a System Administrator, married to one, or good friends come join us!

Friday, 27 July 2012 from 6:00pm to 9:00pm iCal
Egg Roll King Restaurant (map)
85 Courtland Avenue East 
Kitchener ON

System Administrators make backups

If you’re planning on coming, let me know by e-mail or in the comments by Wednesday 25 July so that I can reserve enough seats and egg rolls for everyone.

–Bob & Becky

Event URL for SysAdminDay Dinner: https://bob.jonkman.ca/blogs/2012/07/10/system-administrator-appreciation-day-dinner/

SysAdminDay banners used with permission granted on SysAdminDay.com Banners page.