Does Your Mac Patch Process Rely Solely on Hope and Prayer?

The Hope and Prayer of a Contractor

My family and I love to play pickup basketball games together in our backyard. We do however, live in Texas, and as you probably know, Texas summers are hot. Due to the heat, we decided it would be wise to install an overhead light on the basketball court so we could play hoops when the sun goes down. Since I’m not electrician, we contracted out the work in early February.

Well, it’s now the middle of August and we’re still waiting for the electrician to finish his job. Last week I was pressing the general contractor for a commitment for a specific day for the work to be done. Let’s just say his response was not exactly what I was expecting.

“He said next week for sure. Praying that happens.”

Now, I am a very religious individual and strongly believe in the power of prayer. I even believe prayer should be invoked on behalf of our businesses and livelihood.

However, I believe prayer is a tool to be used after we’ve done everything possible on our end to bring about our desired outcomes. I do not believe prayer should be a tool used as a crutch to overcome our own lack of effort and planning (James 2:20). A general contractor, in my opinion, should be able to effectively work with his team and his subcontractors to gain commitments and delivery dates without doing anything more than throwing it onto the wings of hope and prayer.

How Does this Relate?

So what does this story have anything to do with patching processes for Macs?

To that, I say, good question. In response, I’ll answer your question with a question. Are you secretly relying solely on hope and prayer that the Macs in your environment will simply be self-managed by the savvy users that employ them? After all, Mac users want to be treated differently, right?

I was on a phone call again on Friday and asked the IT administrator if they have just Windows devices employed in their environment, or if they have anything else such OS X devices and/or Linux devices that he would need to manage. His response is about as common as it gets.

“We have some Macs, the total number of which is growing, but it’s mainly just our marketing team and several of our executives. At this time, we don’t manage them.”

You don’t manage them? Wow! How often do we hear the idiom “a chain is only as strong as its weakest link?” Why do organizations invest so much time and money in their infrastructure when they’re only going to patch a percentage of the total attack vectors, leaving the remaining portion to chance…or was that to hope and prayer?

The Vulnerability Data

Common Vulnerabilities and Exposures is reporting that OS X has 178 new security vulnerabilities published in 2015 alone. To compare, that’s more than 2013 and 2014 combined and we still have a third of the year to go – this data only includes vulnerabilities released up until the first week of July. It doesn’t account for the latest zero-day vulnerabilities being reported this week here and here. Furthermore, of the 178 vulnerabilities, 25 of them allow the attacker to gain additional privileges on the compromised system. To put that into perspective, 25 new privilege vulnerabilities is greater than all previously reported exploits of the same type over the last eight years combined!

NumberofVulnerabilities GainPrivileges

Hoping and praying an attack doesn’t surface, through the management process of “look the other way”, is incredibly risky and could potentially cost you millions. Infosec Institute is reporting that in 2013 “the average annualized cost of cybercrime incurred per organization was $11.56 million, with a range of $1.3 million to $58 million.”

So What Do I Do?

Sounds like it’s time to quit relying on just hope and prayer and to put forth some action. So where do you start? First and foremost, obtain a tool that allows you to scan your managed machines against a known vulnerability list that you can control, and ensure your machines scan on a frequent basis and then centrally report up their results.

Just knowing where you stand against the 178 new vulnerabilities is a great start. It will also give you the information you need to make intelligent business decisions.

Notice that I mentioned you will now have the data to make intelligent business decisions. Blindly patching all 178 vulnerabilities without properly testing them, while better than completely ignoring them altogether, is not moving forward intelligently. If you were to blindly patch everything, there’s a good chance you could break a critical business process which in turn could potentially affect your organization to the tune of millions of dollars as well. So be careful and use caution.

Many security pundits will encourage you to test and evaluate new patches in phases. Apple doesn’t have a “Patch Tuesday” so you’ll need to be constantly vigilant about when Apple releases new security updates. Hopefully the tool you use to scan your machines can notify you of new content. If not, keep on an eye on Apple’s Security Updates page found here.

I personally like to break down my patch process into five distinct phases:

  • The lab test phase
  • The smaller, more controlled initial test pilot phase
  • The larger, greater reach secondary pilot phase
  • The mass rollout phase
  • The exception handling phase

The size of your organization and the resources available may force you to adapt, and that’s alright. It’s the checks and balances part that is critical.

The Lab Test Phase

Inside the lab test phase, you should have replica environments for the different types of configurations you have deployed in your environment. Keep them up to date with reality. Make sure when you update your critical business applications in production that you also update your test lab machines. Almost as important, make sure you downgrade the business critical applications to the current release version when testing new security content. Don’t think just because the Apple security update works with version 2.2.x of your business app that it will also work with version 2.1.x.

In order to advance out of the lab test phase, ensure that your exit criteria has been defined and that it meets your success metrics. These metrics may be as simple as defining a machine has been patched, it still can be logged into and that the critical business applications can still be opened. You may also decide to get much more granular, that’s up to you and your organization’s tolerance for risk. What’s important is that a set of defined exit criteria has been established. You can’t be successful if you don’t know what success looks like. Once you have success, document the status of your exit criteria with times and dates and move on.

In regards to timing, when new content comes out, you should be immediately testing against your lab machines.

The Smaller, More Controlled Initial Test Pilot Phase

Once you’ve validated the content against your lab machines and have met your successful exit criteria, move on to a focused pilot group. This is stating the obvious but members of this pilot group need to be using machines that match the type of devices deployed to your entire corporation. That may mean you’re going to need to find dozens of different types of users to match finance, engineering, marketing, etc. Understanding your different user types is critical. Furthermore, these individuals need to be flexible and must be willing to communicate when things don’t work. Build feedback channels into your process that are easy to use. If something breaks, the pilot test group user should know exactly how to contact you with the information you desire.

The next obvious step is to notify the pilot group that patches are coming so they know to be alert and aware.

Just like with the lab test phase, define your exit criteria and document it. Not everyone in your pilot group is going to be available 100 percent of the time. You may have to settle with a 100 percent approval rating from your test group, or up to seven business days. Define what success is, establish the process, and once you can quantify whether you’ve hit your metrics, move on. If you can’t meet your success metrics due to issues, single out the offender and proceed with the rest while you move the offender into the exception handling process.

Ideally, you should be able to wrap up this phase within three to 10 business days from the initial content publication date.

The Larger, Greater Reach Secondary Pilot Phase

The larger pilot phase is nearly identical to the first, only this time increasing the amount of participants. Maybe you move from 2 percent of your machines to five to 10 percent of your machines. Due to the greater number of participants, you may need to be more lax in your exit criteria. You may decide that if 75 percent of your participants approve, or if seven business days have transpired, you have sufficient information to move on. When things break, people complain. When it comes to patch, no news is often good news, that’s why we can put time down as a successful metric. It’s not perfect, there are always those applications that are run on a periodic basis, but it’s a start. Just document were you stand and move on.

Again, just make sure the proper notification is going out to your pilot group so they know to be aware and provide them with dates they need to respond by. Ensure the proper channels are in place so the pilot users can communicate the good and the bad.

This phase should put you two to three weeks out from the content’s publication date. Remember, time is not on your side. The key is to be disciplined, yet focused on getting the content released.

The Mass Rollout Phase

Here comes the fun part, let’s roll out the patch updates to the rest of the organization. While no process will prevent every potential bad outcome, enter into the mass rollout phase with confidence. It won’t hurt to hope and pray at this point either, but at least you’ve done your part.

The key to this portion is to capture the feedback and learn. If a certain individual or group is adversely affected, understand how he/she/them differ from your pilot test groups. Consider adding in this individual or a member from the group into your pilot phase for future testing. The outspoken, high-energy individuals are usually willing to participate in such programs, so bring them in.

Make sure you’re using your patch tool to track the progress. You should be able to hit a pretty high success criteria within days of release.

Where I see most companies struggle here is with off-network devices. Ideally your patch tool should be able to handle the delivery of a patch regardless of where it sits, whether on or off the network. If it doesn’t, you may want to re-evaluate your tool. Machines are always going to be offline or off network, sometimes for days, weeks, months or even indefinitely. Build your process around the exceptions and you’ll be successful.

The Exception Handling Phase

Use this phase to deal with all of the one-off scenarios. It may be a single system that causes you grief. It may be a patch that will not apply. It may be hardware that fails months after patches have been applied, making you vulnerable to items you thought you had patched months ago.

Every scenario is going to be unique and require a different set of steps to troubleshoot. Figure out what you can and then re-insert it back into your patch workflow when it’s ready to be re-evaluated. For me, what’s important, is to document, learn and use what you can to enhance your established patch process so that with each iteration it gets better and better.

Conclusion

Using your patch tool, you should be able track the trends and understand how quickly you can turn around a patch. What may take 35 days can be gradually improved to 30 days, then into the 20s, the 10s and maybe even into single digits. The size of an organization will have such an impact on turn around that comparing against others is not as important as comparing against yourself.

Now, after implemented your newly minted and customized patch process, continue to hope and pray what you’ve done is enough. Zero-day threats will always be with us, having 100 percent prevention is nigh impossible. But having done all that you can do and covering what is in your control, you can now use that prayer and hope  to overcome what is outside your control.

How to Remove Adobe Flash with LANDESK Management Suite

Do you think it’s time to remove Adobe’s Flash Player from your OS X boxes?  One glance at the Common Vulnerabilities and Exposure’s latest update should get you motivated!

If that doesn’t work, check out Ars Technica’s latest blog on the details of the two 0-day exploits.

It is passed time to remove Flash.   Get it out of your environment and get it out now.

Luckily, removing Flash is not very difficult.  With the simple script below, Flash can be quickly relegated to the depths of trash bin, never to be recovered.

#!/bin/bash

UninstallFlash=(

"/Applications/Utilities/Adobe Flash Player Install Manager.app"

"/Library/Internet Plug-Ins/Flash Player.plugin"

"/Library/Internet Plug-Ins/flashplayer.xpt"

"/Library/PreferencePanes/Flash Player.prefPane"

"/Library/Receipts/Adobe Flash Player.pkg"

)

for FlashObject in "${UninstallFlash[@]}"; do

rm -rf "${FlashObject}"

done

If you’re a LANDESK customer, you can create a package file out of the script if you would like, following the steps outlined in the previous blog post on the “The 3 Step Process of Bundling Scripts with Pkgbuild for a Payload-less Package Deployment

For our Adobe Flash instance, an abbreviated version of 3 steps will be as follows:

Step 1 – Create a “Remove Adobe Flash” folder structure and build script for the PkgBuild utility.  Insert the code below for your “package_the_script.sh” script, changing your identifier as needed.

#!/bin/bash

PKG_ID=Remove_Adobe_Flash

sudo pkgbuild --identifier com.appleintheenterprise.$PKG_ID --nopayload --scripts ./scripts ./$PKG_ID.pkg

 

Once saved, run the command “sudo chmod a+x package_the_script.sh” to ensure execute permissions are set.  You may also want to sign the package if using Gatekeeper.

Step 2 – Copy the Uninstall flash script in the beginning of the article into the Scripts/postinstall file

#!/bin/bash

UninstallFlash=(

"/Applications/Utilities/Adobe Flash Player Install Manager.app"

"/Library/Internet Plug-Ins/Flash Player.plugin"

"/Library/Internet Plug-Ins/flashplayer.xpt"

"/Library/PreferencePanes/Flash Player.prefPane"

"/Library/Receipts/Adobe Flash Player.pkg"

)

for FlashObject in "${UninstallFlash[@]}"; do

rm -rf "${FlashObject}"

done

Step 3 – Create the Remove_Adobe_Flash Package by opening Terminal and browsing to wherever you saved your package_the_script file.  Once there, type “sudo ./package_the_script.sh”.  So doing should create your uninstaller package.  Now, zip the pkg file and copy it to your software distribution share and build your LANDESK deployment package.

  • From the LANDESK Console, open Tools > Distribution > Distribution Packages
  • Inside the menu tree, highlight My Packages or Public Packages and then select the New Package button on the menubar and select New Macintosh Package
  • Give the package a name, description and point the primary file to the zip file created previously
  • Fill out the Metadata details if desired
  • Save the package

To create a task to deploy your LANDESK package, walk through the steps below:

  • Right click on the package created and select Create Scheduled Task
  • Target the desired machine(s), user(s) or query(ies)
  • Right click on the task and select properties
  • Set the desired Task type under Task Settings
  • If you desire the end user to be able to initiate the task, set the radio button in the Portal Settings to either Recommended or Optional, otherwise set it to Required and it will automatically begin the upgrade during the next maintenance window
  • Change the Reboot Settings on the task to one that forces a reboot
  • Schedule the task

To validate your success, browse to http://www.adobe.com/software/flash/about/ on the machine and you should see that the plug-in is missing.

Where’s the Moat Around my OS X Castle?

We all want to feel secure and protected, right? Kings, queens and other powerful individuals from ages past, built moats to protect their investments and the people they cared for. Today, while we may not all be kings or queens, we still have the desire to protect ourselves and our personal property.

If you’re a Mac user with the belief that your OS X moat is impenetrable, protecting you from all foreign potential conquerors, it’s time to perk up and use a bit of caution.

According to Pedro Vilaca, a well-known security expert for OS X, the moat around your personal world housed on your Mac has a major flaw. In Pedro’s blog titled, The Empire Strikes Back Apple – how your Mac firmware security is completely broken, he discusses that by simply putting your machine to sleep, an attacker can compromise the device; gaining root access to the firmware.

So where did your moat around our OS X castle go?

In the age of global connectivity, creating a moat sufficient to protect our personal worlds is proving to be increasingly difficult. Just ask the US government who publicly announced they’ve potentially suffered the biggest data breach in government history – a breach that could affect up to 4 million government employees.

Unfortunately, our desire for uber-convenience enabled by near ubiquitous access to the Internet, with all of the benefits constant connectivity grants us, is also causing us to find there are more people trying to tear down our moats than people capable of building and maintaining them. After all, we’re the figurative kings and queens, we don’t build the moats; we just take advantage of them.

So what does this mean for you and me?

Well, the bad news is that unless you’re on a mid-2014 or later Mac model, the exploit is accessible without direct, physical access to the machine itself. Essentially, anybody with the proper wherewithal can compromise the firmware on your machine and take it over simply by targeting you before you put the machine to sleep. Then, when it wakes up, your moat will have suddenly vanished, giving the attacker full-root access to the firmware and subsequently all of the data on your device.

With firmware level access, more calculated attacks are feasible and they could happen without you even knowing about it. Jose Pagilery, writing for CNN money, says that because the vulnerability gives attackers time, they can plot an attack on a much larger scale, similar to the Sony Pictures hack from late last year.

This is really bad. If your machine were infected, even if you buy a new hard drive, or format your existing drive, and reinstall OS X, it wouldn’t take care of the issue. The compromise happens at such a low level on the machine that you won’t be able to get rid of it until Apple releases a new firmware that builds the moat anew.

The good news, well, there isn’t a lot to write about just yet. As a personal user, exercise caution and prudence in the sites you visit on the web. Apple has not publicly commented on the vulnerability yet and there are no known fixes. If you’re not famous or a public figure, the odds are ever in your favor that you will not be randomly targeted. As a corporation, however, there is cause for concern as your employees may be a deliberate target.

If you are in charge of corporate machines, for now, make note of the Boot ROM version on all of your Macs. In LANDESK, this can be done by creating a query and storing it for later use. When Apple releases a firmware update, you’ll already have a list of all vulnerable machines and will quickly be able to take action to repair.

To create this query, right click on the My Queries or Public Queries from the Network View and select New Query.

  1. Give your query a name, something like Mac Boot Rom Version would suffice.
  2. From the Machines Components pick list in the upper left-hand quadrant, scroll down and find the attribute Type, it’s the second to last object under Computer. Select Type, then select the = sign from the operators column and select MAC from the right hand column for scanned values.
  3. Hit insert to create the query syntax
  4. In the Machine Components colulmn in the bottom left-hand quadrant, expand BIOS and add Boot ROM Version to the column set of data to display.
  5. Save the query.

Once you have the query, get on the phone and call Apple asking them when a security fix will be coming. They’ve fixed the exploit on their newer hardware, either knowingly or unknowingly, regardless of which it is, they have the receipt to fix the moat…at least this time and your castle needs it.