Does Your Mac Patch Process Rely Solely on Hope and Prayer?

The Hope and Prayer of a Contractor

My family and I love to play pickup basketball games together in our backyard. We do however, live in Texas, and as you probably know, Texas summers are hot. Due to the heat, we decided it would be wise to install an overhead light on the basketball court so we could play hoops when the sun goes down. Since I’m not electrician, we contracted out the work in early February.

Well, it’s now the middle of August and we’re still waiting for the electrician to finish his job. Last week I was pressing the general contractor for a commitment for a specific day for the work to be done. Let’s just say his response was not exactly what I was expecting.

“He said next week for sure. Praying that happens.”

Now, I am a very religious individual and strongly believe in the power of prayer. I even believe prayer should be invoked on behalf of our businesses and livelihood.

However, I believe prayer is a tool to be used after we’ve done everything possible on our end to bring about our desired outcomes. I do not believe prayer should be a tool used as a crutch to overcome our own lack of effort and planning (James 2:20). A general contractor, in my opinion, should be able to effectively work with his team and his subcontractors to gain commitments and delivery dates without doing anything more than throwing it onto the wings of hope and prayer.

How Does this Relate?

So what does this story have anything to do with patching processes for Macs?

To that, I say, good question. In response, I’ll answer your question with a question. Are you secretly relying solely on hope and prayer that the Macs in your environment will simply be self-managed by the savvy users that employ them? After all, Mac users want to be treated differently, right?

I was on a phone call again on Friday and asked the IT administrator if they have just Windows devices employed in their environment, or if they have anything else such OS X devices and/or Linux devices that he would need to manage. His response is about as common as it gets.

“We have some Macs, the total number of which is growing, but it’s mainly just our marketing team and several of our executives. At this time, we don’t manage them.”

You don’t manage them? Wow! How often do we hear the idiom “a chain is only as strong as its weakest link?” Why do organizations invest so much time and money in their infrastructure when they’re only going to patch a percentage of the total attack vectors, leaving the remaining portion to chance…or was that to hope and prayer?

The Vulnerability Data

Common Vulnerabilities and Exposures is reporting that OS X has 178 new security vulnerabilities published in 2015 alone. To compare, that’s more than 2013 and 2014 combined and we still have a third of the year to go – this data only includes vulnerabilities released up until the first week of July. It doesn’t account for the latest zero-day vulnerabilities being reported this week here and here. Furthermore, of the 178 vulnerabilities, 25 of them allow the attacker to gain additional privileges on the compromised system. To put that into perspective, 25 new privilege vulnerabilities is greater than all previously reported exploits of the same type over the last eight years combined!

NumberofVulnerabilities GainPrivileges

Hoping and praying an attack doesn’t surface, through the management process of “look the other way”, is incredibly risky and could potentially cost you millions. Infosec Institute is reporting that in 2013 “the average annualized cost of cybercrime incurred per organization was $11.56 million, with a range of $1.3 million to $58 million.”

So What Do I Do?

Sounds like it’s time to quit relying on just hope and prayer and to put forth some action. So where do you start? First and foremost, obtain a tool that allows you to scan your managed machines against a known vulnerability list that you can control, and ensure your machines scan on a frequent basis and then centrally report up their results.

Just knowing where you stand against the 178 new vulnerabilities is a great start. It will also give you the information you need to make intelligent business decisions.

Notice that I mentioned you will now have the data to make intelligent business decisions. Blindly patching all 178 vulnerabilities without properly testing them, while better than completely ignoring them altogether, is not moving forward intelligently. If you were to blindly patch everything, there’s a good chance you could break a critical business process which in turn could potentially affect your organization to the tune of millions of dollars as well. So be careful and use caution.

Many security pundits will encourage you to test and evaluate new patches in phases. Apple doesn’t have a “Patch Tuesday” so you’ll need to be constantly vigilant about when Apple releases new security updates. Hopefully the tool you use to scan your machines can notify you of new content. If not, keep on an eye on Apple’s Security Updates page found here.

I personally like to break down my patch process into five distinct phases:

  • The lab test phase
  • The smaller, more controlled initial test pilot phase
  • The larger, greater reach secondary pilot phase
  • The mass rollout phase
  • The exception handling phase

The size of your organization and the resources available may force you to adapt, and that’s alright. It’s the checks and balances part that is critical.

The Lab Test Phase

Inside the lab test phase, you should have replica environments for the different types of configurations you have deployed in your environment. Keep them up to date with reality. Make sure when you update your critical business applications in production that you also update your test lab machines. Almost as important, make sure you downgrade the business critical applications to the current release version when testing new security content. Don’t think just because the Apple security update works with version 2.2.x of your business app that it will also work with version 2.1.x.

In order to advance out of the lab test phase, ensure that your exit criteria has been defined and that it meets your success metrics. These metrics may be as simple as defining a machine has been patched, it still can be logged into and that the critical business applications can still be opened. You may also decide to get much more granular, that’s up to you and your organization’s tolerance for risk. What’s important is that a set of defined exit criteria has been established. You can’t be successful if you don’t know what success looks like. Once you have success, document the status of your exit criteria with times and dates and move on.

In regards to timing, when new content comes out, you should be immediately testing against your lab machines.

The Smaller, More Controlled Initial Test Pilot Phase

Once you’ve validated the content against your lab machines and have met your successful exit criteria, move on to a focused pilot group. This is stating the obvious but members of this pilot group need to be using machines that match the type of devices deployed to your entire corporation. That may mean you’re going to need to find dozens of different types of users to match finance, engineering, marketing, etc. Understanding your different user types is critical. Furthermore, these individuals need to be flexible and must be willing to communicate when things don’t work. Build feedback channels into your process that are easy to use. If something breaks, the pilot test group user should know exactly how to contact you with the information you desire.

The next obvious step is to notify the pilot group that patches are coming so they know to be alert and aware.

Just like with the lab test phase, define your exit criteria and document it. Not everyone in your pilot group is going to be available 100 percent of the time. You may have to settle with a 100 percent approval rating from your test group, or up to seven business days. Define what success is, establish the process, and once you can quantify whether you’ve hit your metrics, move on. If you can’t meet your success metrics due to issues, single out the offender and proceed with the rest while you move the offender into the exception handling process.

Ideally, you should be able to wrap up this phase within three to 10 business days from the initial content publication date.

The Larger, Greater Reach Secondary Pilot Phase

The larger pilot phase is nearly identical to the first, only this time increasing the amount of participants. Maybe you move from 2 percent of your machines to five to 10 percent of your machines. Due to the greater number of participants, you may need to be more lax in your exit criteria. You may decide that if 75 percent of your participants approve, or if seven business days have transpired, you have sufficient information to move on. When things break, people complain. When it comes to patch, no news is often good news, that’s why we can put time down as a successful metric. It’s not perfect, there are always those applications that are run on a periodic basis, but it’s a start. Just document were you stand and move on.

Again, just make sure the proper notification is going out to your pilot group so they know to be aware and provide them with dates they need to respond by. Ensure the proper channels are in place so the pilot users can communicate the good and the bad.

This phase should put you two to three weeks out from the content’s publication date. Remember, time is not on your side. The key is to be disciplined, yet focused on getting the content released.

The Mass Rollout Phase

Here comes the fun part, let’s roll out the patch updates to the rest of the organization. While no process will prevent every potential bad outcome, enter into the mass rollout phase with confidence. It won’t hurt to hope and pray at this point either, but at least you’ve done your part.

The key to this portion is to capture the feedback and learn. If a certain individual or group is adversely affected, understand how he/she/them differ from your pilot test groups. Consider adding in this individual or a member from the group into your pilot phase for future testing. The outspoken, high-energy individuals are usually willing to participate in such programs, so bring them in.

Make sure you’re using your patch tool to track the progress. You should be able to hit a pretty high success criteria within days of release.

Where I see most companies struggle here is with off-network devices. Ideally your patch tool should be able to handle the delivery of a patch regardless of where it sits, whether on or off the network. If it doesn’t, you may want to re-evaluate your tool. Machines are always going to be offline or off network, sometimes for days, weeks, months or even indefinitely. Build your process around the exceptions and you’ll be successful.

The Exception Handling Phase

Use this phase to deal with all of the one-off scenarios. It may be a single system that causes you grief. It may be a patch that will not apply. It may be hardware that fails months after patches have been applied, making you vulnerable to items you thought you had patched months ago.

Every scenario is going to be unique and require a different set of steps to troubleshoot. Figure out what you can and then re-insert it back into your patch workflow when it’s ready to be re-evaluated. For me, what’s important, is to document, learn and use what you can to enhance your established patch process so that with each iteration it gets better and better.

Conclusion

Using your patch tool, you should be able track the trends and understand how quickly you can turn around a patch. What may take 35 days can be gradually improved to 30 days, then into the 20s, the 10s and maybe even into single digits. The size of an organization will have such an impact on turn around that comparing against others is not as important as comparing against yourself.

Now, after implemented your newly minted and customized patch process, continue to hope and pray what you’ve done is enough. Zero-day threats will always be with us, having 100 percent prevention is nigh impossible. But having done all that you can do and covering what is in your control, you can now use that prayer and hope  to overcome what is outside your control.

Frustrations in the Sky…Why So Many Plugins and Restrictions to Watch a Movie?

Flying is something I do a lot.  Having spent as much time on the road as I have, I’m accustomed to the frustrations that take place with all of the questions one goes through in order to prepare for travel.   Thankfully, I have the “rituals” in place to be able to travel without frustrations…at least for the frustrations that are in my control.  No longer do I have to mentally spend effort and time on:

  • When do I need to leave for the airport?
  • Which route should I take to avoid traffic?
  • Long term or short term parking?
  • Will the security line make me miss my flight? (Thanks TSA Pre√ )
  • Will the liquids in my bag flag me for an inspection?
  • Will this belt set off the metal detector?
  • What pre-flight food should I purchase that won’t cause indigestion?
  • Will my carryon bag fit in the overhead bins?
  • Will my seat offer adequate leg room?
  • How do I get to the rental car agency?

So while I fly often, 95% of the time it is for business travel.  When flying for business, while in the air traversing the skies, I’m typically working and preparing for the upcoming meeting or I’ll spend my time catching up from being out of the office.  This week, however, I took a personal flight after standard business hours and due to the craziness leading up to the flight, I was looking forward to pulling out my iPad to relax and watch a movie on Delta Studios. 

As soon as the airplane beeped indicating we’d reached 10,000 feet, I grabbed my iPad Mini 2 ready to browse my movie option list.  Unfortunately, I forgot to install Fly Delta prior to take off, a rookie mistake I think.  Not a huge deal, but on flights that are less than 3 hours, I know that timing is important as you don’t want to be in the last 10 minutes of movie during landing when the flight entertainment is shut off.  It is frustrating, I know.  GoGo did reimburse me on that occasion though, so props to their customer service. 

Knowing I can download it Fly Delta for free, without having to pay for Internet, I launch Safari and browse to video.gogoinflight.com. I’m quickly whisked away to a page letting me know that my browser is not supported. 

What, why I ask?  I’m using Safari on iOS.  Why would it not be supported?  I look through the supported browser list and the only supported browser for iOS is Safari 5.0 and up.  Well, the perils of being an early adopter bit me.  Apparently in inflight entertainment provided by GoGo or Delta Studios checks for a minimum and maximum version of the browser.  My iPad is on iOS 9 beta, and because of the coding restriction, it’s therefore not supported.

My frustration begins, however, I can see the logic in the decision.  Early adopters for Operating Systems could be purchasing movies and then complaining about their experience when using a browser the vendor has not yet been able to properly evaluate.  Refunds and support costs go up, service experience goes down, etc.  Whatever, I get it. I have options, no big deal. 

I then pull out my phone.  Not exactly the ideal video experience, but I don’t want to use my laptop.  Unfortunately for me, my phone battery was at 36%.  It had been a long day with a lot of phone use, not leaving me with enough battery to watch a movie, so outcomes my MacBook Pro.

Oh no!  Adobe Flash player required. 

GoGo video entertainment requiring flash

Oh no is right!  After the latest exploits in Flash, I purposely removed it from my MacBook

Turns out to watch a movie, Flash is required.  Ugh!  Furthermore, Chrome is not supported so i can’t use the embedded Flash support available via Google Chrome.  Now the frustrations are really starting to mount.  I’m on my third option and what was supposed to be a relaxing time has turned into a lot of troubleshooting and back and forth between devices. 

Reluctantly I decide I’ll install Flash and remove it when the flight is over. Success is imminent.

Or maybe success is not to be had at all!  I guess it’s a good thing I have not ordered popcorn from the flight attendant yet. In addition to needing to install Flash, I also need to install plugin for the GoGo video player.  Now I’m really reluctant to proceed.  With Flash, I understand the risks and I know how to remove it.  In regards to the Widevine Optimizer, I’m clueless and at 30,000 feet and Internet-less, so I can’t do a proper discovery. Widevine Plugin Required

Motivated now by competition and unwilling to let the situation get the best of me, I install a second plugin onto my machine.  My frustration level is at a peak and I’m ready to call the product manager over the inflight entertainment and ask him/her the below: 

  • Where’s the GoGo app to handle all of this for me? 
  • Is there a way to have a one click install to take care of what is needed? 
  • Why use technology that requires so many plugins when people are trying to use this when flying at 30,000 and may be without the Internet? 
  • What are the goals in the future to make the end-user experience more enjoyable?

Anyways, after one more browser restart, it looks as if I’ll be able to watch a movie.  By this point, I’m an hour into the flight, the flight attendant has come and gone which means know movie popcorn and I know full well I won’t be able to finish the movie. 

Why so many plugins and restrictions to watch a movie?

Force LANDESK Antivirus for Mac to Update its License File

I recently had to install LANDESK AV directly from the Mac Client package on a machine, but unfortunately, the machine was not connected to the Core server’s network nor the Internet.  I’m not sure if that is a required step in order to pull down the license file or if it was just circumstantial, nonetheless, I couldn’t get the client to pull down the license even after I connected it to the Core’s network and to the Internet.

As I always do in these types of situations, I pulled up my browser and hit the LANDESK Community site.  I quickly found an article titled How to refresh the Mac AV license key on client machines and was quite hopeful the answer would be found therein.  I was in luck!  The prescribed solution written by Bryce worked like a champ.

Within minutes I had a custom script written in the Manage Scripts portion of the LANDESK Console, a task created and targeted with a successful result returned.  The Mac I was working with was able to update to the latest pattern files letting me get back to the problem at hand.

Now, while the script did indeed work as described in the how to community article, the ldkahuna command to download files is old and outdated within the modern LANDESK Mac agent.  It works, it hasn’t been completely deprecated, but there is a better way and that better way is to use sdclient.

Using sdclient instead of ldkahuna has a number of advantages.  First off, it has access to the bandwidth controls built into the agent.  It also has a -dest switch allowing you to not only download but to specify the location of where you want the files to be placed.  Furthermore, rumor has it, that in the next LANDESK Management Suite release, sdclient will become peer aware like the Windows agent.

Who wants to use the old stuff anyway, when you can use the new stuff?  If Steve Goodrich uses it when he needs to download stuff, I’ll use it as well.

The best part, the switch is quite straight forward.  Let’s look at how we would write it with ldkahuna and how it would be written with sdclient.

Old Code:

REMEXEC01=ldkahuna http://%CUSTJOBHOSTIP%/ldlogon/avclient/install/key/ldav.key</pre>

New Code:

REMEXEC01=sdclient –noinstall –package http://%CUSTJOBHOSTIP%/ldlogon/avclient/install/key/ldav.key</pre>

Pretty simple change, right?  I thought so.  Now just schedule out your Managed Script and target those stubborn Macs that won’t update their license file.  In short order, you’ll be back in business.

How to Recover from “Redirect-AVScanner.com” or Other Malware Highjacking Safari on Mac OS X

Over the weekend I was troubleshooting a machine in which Safari was being highjacked by a visual pop up and audible voice indicating the machine had a MAC iOS Alert.  

The malware was completely taking over Safari and wouldn’t allow any other tab to open.  The message on the screen essentially said there was a security update to fix an SSL connection and said I just had to call the “Apple Support” number.  Even if you hit the OK button, you were stuck.  The only thing that could be done was a Force Quit on Safari.  

Red flag alert!  Don’t call the number. 

I know those of you who are LANDESK admins won’t do it, but for anyone else that may stumble upon this article, DO NOT call the number on the screen and DO NOT pay them any money.

For information on how to remove Mac Defender, I looked at Apple’s article on removing Mac Defender but it was for 10.6 and earlier and didn’t help much.  My machine was on 10.10 so I kept searching and ended up using this article on StackExchange. 

Basically, I deleted the following files:

  • ~/Library/Saved\ Application\ State/com.apple.Safari.savedState
  • ~/Library/Safari/LastSession.plist
  • ~/Library/Cookies (all cookies in the folder)

I then relaunched Safari and all was well, Safari was back to normal. 

If you’re a LANDESK customer and want to deploy a script to remediate more than one machine, copy the code below into a shell script.  Just set the execute permissions on it and copy it to your distribution repository.

#!/bin/sh
#&lt;span class="Apple-converted-space"&gt;  &lt;/span&gt;Redirect-AVScanner Malware Removal.sh
#August 3, 2015
rm -rf ~/Library/Saved\ Application\ State/com.apple.Safari.savedState
rm -rf ~/Library/Safari/LastSession.plist
rm -rfv ~/Library/Cookies/*

Since these folder locations are inside the user’s profile, creating a deploy package may be a bit more difficult if multiple profiles exist on a single machine.  It may be best to create an optional package and publish into Workspaces.

To create your Workspace package, use the following steps:

  1. Open the LANDESK Console
  2. Navigate to the top menu bar, select Tools > Distribution > Distribution Packages.
  3. In the lower left menu tree, highlight My Packages or Public Packages from within the Distribution Packages window
  4. On the Distribution menubar, press the New Package button and select New Macintosh Package.
  5. Give the package a name, I used #1 MacDefender Removal
  6. Provide a description if desired
  7. Set the primary file to the zip file you previously transferred to your software distribution folder
  8. Fill out the Metadata details if desired, specifically supplying a logo so it shows up properly in the portal
  9. Save the package

With the malware removal package created, just schedule a task for deployment.

  1. Right click on the malware removal package created and select Create Scheduled Task
  2. From the network view, select and drag the desired machine(s), user(s) or query(ies) and drop them onto the task
  3. Now, right click on the task and select properties
  4. Set the desired Task type under Task Settings as to whether you want a push, a policy or a hybrid of the two types in a policy-supported push.
  5. Set the radio button in the Portal Settings to either Recommended or Optional.  You’ll also want to check the box for “Allow users to run as desired (keep in portal after selected) so they can execute the script multiple times if need be.
  6. Change the Reboot Settings or Distribution and Patch settings if desired
  7. Set the schedule task settings with the appropriate start time
LANDESK Workspace MacDefenderRemoval Screenshot
LANDESK Workspace MacDefenderRemoval Screenshot