Once In A Blue Moon

By slpowers on July 31, 2015 • ( 0 )

Once in a blue moon, we pull our engineers away from their clients long enough to coax a blog post out of them. In honor of today’s blue moon, Senior Engineers Josh Spranger and Chris Fettig weigh in with two takes on “Once in a blue moon . . .”

By Josh Spranger

As technology professionals we are asked on an almost daily basis to review other clients’ networks. This is usually in response to some data breach, management shakeup, or to address a specific issue that is going on that the local IT staff is unprepared to handle. In 99.9% of those instances we find at least one major issue with the network we are tasked to evaluate, and we then remediate that issue and leave the client happy.

In this ‘Blue Moon’ event I came across a situation where the client actually did everything ‘by the book’.

We were tasked to do a total review of the network of a nationally known Fortune 500 multimedia company after the hasty and unplanned removal of their CIO. The newly-appointed CIO did the wise thing in this situation and ordered a complete top-to-bottom third-party review of their national network. We spent the better part of a week onsite at the client’s home office and data center, with remote access to their New York IT headquarters.

We evaluated every single aspect of their network, from security, to backups, to routing and switching best practices. We looked at more than a hundred virtual servers and their virtual infrastructure, including multiple storage networks. We reviewed WAN and LAN efficiencies and overall network health. We then reviewed all their operational guidelines and procedures for both completeness and best practices. And finally we reviewed their DR plan.

During this time we interviewed all the IT staff, many of which were from the prior administration.

My review showed that at every turn they had established the correct procedure, applied the correct configuration, and put the correct monitoring plans into place to ensure network operation, security and backup/recovery. They had developed detailed plans for what happens during a DR event – and more impressive they actually TESTED all aspects of DR on a quarterly basis!

So in the end, this ‘Blue Moon’ moment was that we found an IT shop that did exactly what it was supposed to do and was ‘on their game’, which meant we had nothing to fix. And in my 35+ years of doing evaluations, that truly was a rare moment!

By Chris Fettig

Security reviews are usually done “once in a blue moon”, but we all know they should be a regular occurrence. Same for documentation reviews, but there’s not a lot of flash and pizazz there, which may be why they are so scarcely performed.

There is a general misunderstanding that a data network can become a “set it and forget it” project, instead of an evolving organism demanding ongoing support. The idea that performing any routine maintenance ‘once in a blue moon’ will be enough to keep entropy at bay is problematic thinking. (Heck, many folks have a failure to connect entropy as a concern for their network at all…)

Some specific examples:

1. UPS battery tests and/or generator cutover tests should be reviewed at least every couple years.

2. Group membership reviews. This can be accomplished as part of a larger security review, but just knowing which employees are in which distribution lists can be important. I recently had a client send an email to “EVERYONE” without being aware that the list had zero recipients. Other clients have sent sensitive information to a group, just to turn around and panic that the e-mail went to unintended recipients.

3. Backup/Recovery/Business Continuity plan tests and reviews.

4. Hardware refresh (servers, workstations, firewalls). Who HASN’T had a client that waits until something breaks before replacing it? Ironically, we’ve had just as many heave a big sigh after a network overhaul and ask why they hadn’t upgraded months sooner, now that they realize just how inefficient their old systems were.

5. Updating long-term IT goals and plans. I once had a 90-minute debate with an ex-employer over why an IT department should have a three-month goal. Needless to say my thoughts on 12- and 36-month goals were also an uphill battle.

6. Product / tool utilization review. We all get in a rut. I recently reminded a client Adobe Photoshop isn’t the ONLY photo editor in the world, and installed GIMP for them to prove the point. Rethinking what tools we use usually occurs when we don’t want to spend the money to replace a broken or worn out tool. (Or because Adobe is switching to monthly subscription only and this particular client dislikes the idea of paying for their photo editor in perpetuity.) Keeping pace with other options isn’t on most clients’ radar. We need better reasons to use the tools we do than “Because it’s what I started with.”

Josh and Chris both bring up good examples of when “once in a blue moon” is not sufficient to ensure solid network security. A lax approach to maintenance and review can put an organization at risk of infiltration, data loss, and other security perils.

Contact SLPowers today to start putting a strong multi-layered security plan in place, and leave the rare events to that big, blue moon in the sky.