This Blog Is Not For Reading

The last Friday in July is System Administrator Appreciation Day, and SysAdmins from Kitchener-Waterloo went to the Abe Erb Restaurant and Brewery for dinner.

Abe Erb Brewing Company

The Whole Gang (almost)

Infinite Beer

Having a laugh

Beer Valves

Jean and Laurel

Beer Console

Beer Admin

More SysAdmins

Beer Vat

Empties

Ooh, The Shiny!

Something is funny

Acidulated Malt

A beer thing

Tech Talk

Wild Goose

Such shiny things!

Having dinner

In The Brewery

Twilight

SysAdminDay

There are more pictures in the SysAdminDay gallery.

Pictures taken at the System Administrator Appreciation Day Dinner at the Abe Erb Restaurant and Brewery on Friday, 28 July 2017.

Pictures by Laurel L. Russwurm, used under a CC BY 4.0 license.

It’s July again, and System Administrator Appreciation Day is always celebrated on the last Friday in July. Although the SysAdminDay website indicates that SysAdmins are the happy recipients of cake and ice cream, for the last few years SysAdmins in Kitchener-Waterloo have been celebrating SysAdminDay by taking themselves to dinner, along with spouses, friends, and co-workers.

This year we’re teaming up with the Kitchener-Waterloo VoIP Users Group. While KWVoIP meetings are usually scheduled for the fourth Thursday in July, this month we’re combining the KWVoIP meeting with SysAdminDay. There’s an overlap between the KWVoiP members and SysAdmins — VoIP systems need administration too! As a bonus, we’ll get a presentation from Brian Bentley about his roaming experiments in the U.S.

Tony, the Egg Roll King himself, is a versatile host. Not only are the egg rolls and spring rolls the best in KW, there are vegetarian meals available, and ERK has some of the best fried chicken, fish & chips, and poutine in town! Friday nights are busy at the Egg Roll King Restaurant, so leave a comment to let me know you’re coming and I can make a reservation.

Date: Friday, 29 July 2016 from 6:00pm to 9:00pm
Location: Egg Roll King Restaurant, 85 Courtland Avenue East, Kitchener, Ontario Map
iCalendar: kwvoip-2016-07-29.ics

KWVoIP Topic: Brian Bentley – Roaming Profile Experiments

Brian Bentley has returned from a short trip to the United States. Unfortunately, he wasn’t able to present last month, so he’s catching up this month.

In order to stay in touch telephonically, Brian used Roam Mobility for Data services. He tested Fongo (Canadian number) and TextNow (US number) for VoIP services. While in the US he tried to sign up for a Google Voice number. What worked? What was cheapest? Was it worth the bother? Come to dinner to hear Brian’s answers.

Cheese Wontons for dessert!

Oh look! Pictures!
Oh look! More pictures in the gallery!

System Administrator Appreciation Day falls on the last Friday of July every year, and is allegedly celebrated by users gifting their SysAdmins chocolate cake and ice cream. Hands up, those of you who have actually experienced that? Hmmm? I thought so…

Nobody appreciates System Administrators more than other System Administrators, so for the last few years I’ve hosted a Systems Administrator Appreciation Day Dinner in Kitchener-Waterloo. Last year we went to LaiLai’s, and previously we’ve gone to Egg Roll King.

SysAdminDay is on Friday, 31 July 2015, just over two weeks away. Where would you like to go this year? Potential venues must offer vegetarian fare, and be physically accessible. Leave suggestions in the comments or send me e-mail, then next week we can vote. If there’s more than one candidate on the list we’ll rank choices from 3 points (most favoured) to 1 point (least favoured), and I’ll add them up, post the results here, and we’ll all meet for dinner.

SysAdminDay Dinner is open to everyone, whether you’re a System Administrator, a SysAdmin Student, or a former SysAdmin who’s been lured to the dark side. And also their friends, family, and end-users. OK, maybe not the end-users. Unless they’re friends or family.

See you on 31 July!
–Bob.

Proposed Venues

A smattering of local sysadmins

The System Administrators of Kitchener-Waterloo got together for dinner on System Administrator Appreciation Day:

A smattering of local sysadmins

One System Administrator: John

Another System Administrator: Bob

Two System Administrators, Jeff and Jean

Three System Administrators: Tim, Laurel, Rigas

Systems Administrators discuss Android Systems Administration

The Bills

More pictures in the gallery

Pictures by Laurel L. Russwurm from Waterloo Region is Awesome and by Bob Jonkman, used under a CC BY 4.0 license.

WordPress Hacked!

WordPress upgrades are famous for their ease of installation. Surprise! After upgrading the first installation most of the plugins were missing, and the theme was broken. A quick look at a directory listing showed that the plugins and themes were still installed. A quick look with a text editor showed some peculiar PHP code at the top of every .php file in the plugins folders. Surprise! This WordPress installation had been hacked! Fortunately, of the five instances of WordPress on this server, only two appeared to be affected. This Blog Is Not For Reading was not one of them.

Each .php file started with something like this:

<?php $zend_framework="\x63\162\x65(…)\x6e"; 
@error_reporting(0); 
zend_framework("", "\x7d\7(…)

Searching the Internet for “wordpress plugin invalid header zend_framework” I found a reference that makes me think this may have been possible because of a flaw in an earlier version of the WordPress code that handles comments. Most likely one of the comment fields (user name, e-mail, web address or the comment text itself) wasn’t properly sanitized, and allowed some kind of code injection (probably PHP injection, not a MySQL injection; the contents of the databases appeared to be untouched).

From the backups of the server it appeared that the breach occurred in or before August — either just before the release of WordPress 3.6 on 1 August 2013 or just before the release of WordPress 3.6.1 on 11 September 2013. If I had not been slack in upgrading to WP v3.6.1 then this breach might have been identified much sooner.

The upgrade to WordPress identified the modified files because the injected code preceded (and corrupted) the WP headers, and so WP v3.7 disabled any affected plugins and themes.

The Fix Is In

I renamed the directory containing the WordPress code, installed a fresh copy of WP3.7, cleaned and copied the wp-config.php and .htaccess files, uploaded a small image to create the wp-content/uploads hierarchy, then copied the upload folder (which didn’t contain any .php files), and then re-installed and re-configured the themes and plugins directly from the WordPress site.

Aside from the additional PHP code, there didn’t appear to be any other damage to the system. So I used the original wp-config.php (but cleaned, and with the “Authentication Unique Keys and Salts” section refreshed), and the new installation just used the existing databases. If there’s any malcode in the databases then that could re-infect the system, so I’m keeping an eye on it.

I have no idea what the malcode was intended to do. It didn’t corrupt the databases or anything else, but it’s possible it was acting as a keylogger or phoning home some other way. If I feel inclined I might try to de-obfuscate the injected code, but right now I don’t really feel like doing forensics.

Someone suggested using AppArmor to make the WordPress directories read-only. I’m not sure that locking down the WP directory is a good idea. The big new feature in WordPress 3.7 is its automatic update feature. If the WordPress directories are locked down then future security updates won’t be applied automatically. If there is an exploit and WordPress issues a new release to fix it, then a locked-down site will experience a delay in upgrading until the SysAdmin notices and upgrades manually (which is what used to happen before v3.7, but it seems a bad idea to delay upgrades when that’s no longer necessary). Also, the plugin and themes directories would be locked down, and they still require fairly frequent manual upgrades.

I sent the users on the affected sites this message:

While doing upgrades on WordPress yesterday I saw that your blog had been hacked sometime during or before August. I’ve fixed it (re-installed the code, copied your media library, re-installed themes and plugins). I don’t think any damage was done beyond the insertion of malicious code in some of the WordPress files. I don’t know what the action of that code was intended to be, but you should change your WordPress password just in case the bad guys captured it. You can change your password on the “Users, Your Profile page” once you’ve logged in.

After spending some time on Saturday fixing the two hacked WordPress sites I’m a little paranoid, and making sure to implement updates quickly. But a little paranoia is good — it’ll ensure I won’t become complacent again.

–Bob.

WordPress Hacks by Rafael Poveda is used under a Creative Commons — Attribution-NonCommercial-ShareAlike — CC BY-NC-SA license.

They say on the Internet Pictures, or it didn’t happen. So I just want to give you my assurances that the System Administrator’s Appreciation Day Dinner did actually happen!

27 July 2012, Egg Roll King Restaurant

System Administrator Appreciation Day Dinner, 27 July 2012 at the Egg Roll King Restaurant. Clockwise from left: Willem Jonkman, Bob Jonkman, Becky Nguyen, Wael Khawaja, Hamin (sp?), Rodney Martin, Sergiane Nascimento, Tim Laurence, Henrique Nascimento, Gui Nascimento.

Willem and Bob

Bob and Becky

Wael and Hamin

Rodney and Tim

Henrique and Gui

System Administrators out for dinner. Clockwise from left: Sergiane Nascimento, Tim Laurence, Henrique Nascimento, Gui Nascimento, Laurel L. Russwurm, Willem Jonkman, Bob Jonkman, Becky Nguyen, Wael Khawaja.

Cheese Wontons for dessert!

Bob Jonkman, System Administrator

–Bob.

Visit the official System Administrator Appreciation Day web site.

All pictures courtesy of Laurel L. Russwurm and released under a CC-BY-SA license.

Novell Netware 6.5

Swap Screens

ALT+ESC

List of screens

CTRL+ESC

Alternate console

CTRL+SHIFT+ALT+ESC

Shutdown prompt or the “Hung Console Menu”

CTRL+ALT+ESC

Entering the Netware Debugger Monitor

LSHIFT+RSHIFT+ALT+ESC

• he — Help on expressions
• hb — Help on breakpoints
• h — Help
• .h — Help on ‘dot’ commands
• .a — Display abend reason
• .c — Core Dump
• .d — Display page entry map for current domain
• .d address — Display page directory map for current domain
• .l offset — Display linear address at page map offset
• .lx — Display page offset and values
• .m — Display names and addresses of NLMs
• .p — Display process names and address
• .p address — Display address as a process control block
• .r — Display running process control block
• .s — Display screen names and addresses
• .s address — Display address as a screen structure
• .sem — List all semaphores with waiting processes
• .sem address — Display detailed semaphore inforation at address
• .t — Toggle ‘Developer Option’ on or off
• .v — Display version
• b — Display breakpoints
• bc number — Clear breakpoint number
• bca — Clear all breakpoints
• b= address — Set breakpoint at address
• br= address — Set read or write breakpoint at address
• bw= address — Set write breakpoint at address
• c — Change memory
• c entrypoint=hexdigits — Change bytes at entrypoint
• d — Display memory at the current stack pointer
• d address [length] — Display bytes at address for length bytes
• dl [+offset] address [length] — Display linked list for length nodes with next node address at offset
• f flag=value — Change flag to value. Valid flags: CF, AF, ZF, SF, IF, TF, PF, DF, or OF
• g — Go (back to Operating System)
• g breakaddress — Go, end at breakaddress
• i[b|w|d] port — Input Byte, Word, or Doubleword from port
• o[b|w|d] port=value — Output Byte, Word or Doubleword with value to port
• n — Display symbol names and NLMs
• n symbolname value — Create new symbolname with value
• n-symbolname — Remove symbolname defined with n
• p — Step through program code, skip calls
• s or t — Step through program code, enter calls
• q — Quit to DOS
• r — Display registers and flags
• u entrypoint [length] — See assembly code at entrypoint for length bytes
• v — View screens
• x — Exchange processor stack frames
• z expression — Evaluate expression
• ? entrypoint — Display reference to entrypoint
• register= entrypoint — Set register to entrypoint. Example: EIP=CSleepUntilInterrupt will interrupt the last NLM called. Valid registers: EAX, EBX, ECX, EDX, ESI, EDI, EBP, EIP, and EFL

See also the Novell Application Note The NetWare Internal Debugger. There’s more good information in the Novell TID 3193476: How to troubleshoot … an abended, unresponsive or crashed server.

Bob Jonkman <bjonkman@sobac.com>         http://sobac.com/sobac/
SOBAC Microcomputer Services              Phone: +1-519-669-0388
6 James Street, Elmira ON Canada  N3B 1L5  Cell: +1-519-635-9413
Software   ---   Office & Business Automation   ---   Consulting

Pictures are up!

System Administrators install servers

System Administrator Appreciation Day falls on 27 July this year, a global celebration of System Administrators around the world. To see what System Administrators do, have a look at some of the systems they have to administer.

After the KWLUG meeting in July, Becky asked me where System Administrator Appreciation Day is being held. “It’s a global event”, I said, but Becky was hoping for something a bit more local. So we decided to have a System Administrator Appreciation Day Dinner. What better way for System Administrators to celebrate than by treating ourselves to dinner?

System Administrators maintain web sites

So, if you’re a System Administrator, married to one, or good friends come join us!

Friday, 27 July 2012 from 6:00pm to 9:00pm iCal
Egg Roll King Restaurant (map)
85 Courtland Avenue East 
Kitchener ON

System Administrators make backups

If you’re planning on coming, let me know by e-mail or in the comments by Wednesday 25 July so that I can reserve enough seats and egg rolls for everyone.

–Bob & Becky

Event URL for SysAdminDay Dinner: https://bob.jonkman.ca/blogs/2012/07/10/system-administrator-appreciation-day-dinner/

SysAdminDay banners used with permission granted on SysAdminDay.com Banners page.