D
Diane Ross
Maybe this will shed some light on the problems some users are experiencing.
Posted on the Unsanity Blog:
<http://unsanity.org/>
March 16, 2007
SHOCK AND AWE: HOW INSTALLING APPLE'S UPDATES CAN RENDER YOUR MAC UNBOOTABLE
AND HOW YOU CAN PREVENT IT
Yes, I am fully aware that I just told every one to update to Mac OS X
10.4.9 and I know this may be construed as me telling people not to.
When you see the "Optimizing System Performance" phase of a software update,
Mac OS X is really updating prebinding. Updating prebinding has a very, very
nasty bug in it (look at _dyld_update_prebinding). If multiple processes are
updating prebinding at the same time, then it is possible for a system file
to be completely zero'd out. Basically, all data in the file is deleted and
it is replaced with nothing. This bug is usually triggered when updating Mac
OS X and every update to Mac OS X has the potential to render your system
unbootable depending on if the "right" file is deleted or not. It's
triggered during the "Optimizing System Performance" phase of installing an
update. This phase is actually just running update_prebinding. If you launch
an application that links to libraries that are not yet prebound, there is a
chance one of those files will be zero'd out as dyld automatically redoes
the prebinding on that file.
I've been tracking this particular bug for about 18 months now. Most of the
real "random" failures reported on various Mac OS X "troubleshooting" sites
after a user has installed an Apple software update are actually
manifestations of this bug. By real I mean not imagined problems or ones
that have been there for a very long time but the user is just now noticing
it and artificially connecting the cause to the recent update (it's called
Pareidolia). Yes, this nasty prebinding bug has been reported to Apple and
yes, it is 100% reproducible if you want to reproduce it.
Every single time you install an update to Mac OS X whether it be an iTunes
update, a QuickTime update, an update for daylight saving time, a security
update, an Airport update, or an actual Mac OS X update, you can be hit by
this bug. In order to prevent yourself from being smacked in the face by
this bug, follow this simple rule: When "Optimize System Performance"
appears during the update process do not touch your computer and definitely
do not launch any applications. Just back away from your computer box as if
it were a swarm of bees. Yes, it does mean that if you install the Mac OS X
10.4.9 update, you may get hit by the bug.
I think it's important to note that APE's use of update_prebinding at login
is not affected due to the time at which APE runs it on the ICBMs. In order
for the nasty prebinding bug to manifest, multiple prebinding operations
must be going on (either explicit or implicit). Assuming some implicit
action needed to be done, it's already been done by the login window
application (the one that shows that bar graph when you boot) on any shared
libraries it links to. And once you log in, even forcing a prebinding
operation won't actually cause any files to be reprebound since that action
was just done.
Symptoms and Signs
The worst sign you've been hit by this bug is an inability to boot after
installing a Mac OS X update. Sometimes the little wheel will just keep on
spinning. Other times you'll get to the point where you should see your
desktop but all you see is a blue screen (because [the] loginwindow is
repeatedly crashing due to a missing library). The "easiest" sign is an
application will crash either at launch or when you do a specific action and
the console.log /Applications/Utilities/Console (or a crash log) will spew
out a message about dyld that says: "Reason: no suitable image found." and
then sometimes "file to short" [sic]. The file is too short because it is
zero-length. There is an example of someone being hit by this on the
internets.
Sadly, most people suffering from this bug never have a chance to see an
error message to find out what file was zero'd out, especially when it
prevents boot. One of the things that can help a lot in troubleshooting the
problem is booting in verbose mode. Setting this option is really quite
easy. Open the terminal (/Applications/Utilties/Terminal) and type:
sudo nvram boot-args="-v"
Then just enter your password. From then on, every time you reboot, you'll
be in verbose mode. You can also hold Command-V at boot time to boot into
verbose mode for that boot only.
Verbose mode basically just doesn't show the Apple logo (OEM logo on the
ICBMs) at boot time. You get to see all the gritty ugly details in booting
Mac OS X. Note: You will not understand everything that's going on during
booting in verbose mode. You don't need to. All you need to do is recognize
what is normal and what isn't. Also, it really helps to boot into verbose
mode after an upgrade as the first reboot can take an exceedingly long time
(up to 10 minutes) and booting into verbose mode will help you know your Mac
hasn't frozen. (Seeing "diskarb not ready" is normal, especially after a
system update).
Preventative Measures
Well, I already listed it: When "Optimize System Performance" appears during
the update process do not touch your computer and definitely do not launch
any applications. Just back away from your computer box as if it were a
swarm of bees.
I should note that this bug seems a lot more likely to happen on an ICBM as
the prebinding operation has to work on two sides of a fat file (the x86
side and the PowerPC side) so it takes longer, which means there is more
time for you to trigger the bug.
Solving the Problem if You're Hit
Recovering from this bug is not the easiest thing in the world. If it's a
simple crash, you can just copy a "good" version of the file off a Mac
running the exact same build as the busted one or you can just run the most
recent combo updater (assuming the zero'd out file is in the combo updater).
If it is really bad, you'll have to boot the busted Mac into FireWire target
disk mode and try to run the combo updater off a working machine. TDM is not
an option if the other Mac has a different processor architecture of it is
is running a newer version of Mac OS X.
What Will Apple Do?
Who knows. This bug has been filed with Apple, along with steps to reproduce
it 100% of the time (at least in my testing). It was marked as a duplicate,
which means the bug was already in Apple's system before I filed it. And
since it is duplicate, I don't know what is going on with it. Yes, before
anyone mentions it, I know prebinding is deprecated. However, Mac OS X still
does it when installing Apple updates. It doesn't matter that it is
deprecated if it still happens, after all.
Even if prebinding goes completely away in Mac OS X 10.5 that doesn't solve
the problem for Mac OS X 10.4.x users. Security updates will continue to be
released for Mac OS X 10.4.x until Mac OS X 10.6 is released (which could be
many, many years from now) and every one of those updates carries the risk
of "destroying" the Mac of the person that installs it until this bug is
fixed.
--
Diane Ross, Microsoft Mac MVP
Entourage Help Page
<http://www.entourage.mvps.org/>
One of the top five MS Entourage resources listed on the Entourage Blog.
<http://blogs.msdn.com/entourage/>
Posted on the Unsanity Blog:
<http://unsanity.org/>
March 16, 2007
SHOCK AND AWE: HOW INSTALLING APPLE'S UPDATES CAN RENDER YOUR MAC UNBOOTABLE
AND HOW YOU CAN PREVENT IT
Yes, I am fully aware that I just told every one to update to Mac OS X
10.4.9 and I know this may be construed as me telling people not to.
When you see the "Optimizing System Performance" phase of a software update,
Mac OS X is really updating prebinding. Updating prebinding has a very, very
nasty bug in it (look at _dyld_update_prebinding). If multiple processes are
updating prebinding at the same time, then it is possible for a system file
to be completely zero'd out. Basically, all data in the file is deleted and
it is replaced with nothing. This bug is usually triggered when updating Mac
OS X and every update to Mac OS X has the potential to render your system
unbootable depending on if the "right" file is deleted or not. It's
triggered during the "Optimizing System Performance" phase of installing an
update. This phase is actually just running update_prebinding. If you launch
an application that links to libraries that are not yet prebound, there is a
chance one of those files will be zero'd out as dyld automatically redoes
the prebinding on that file.
I've been tracking this particular bug for about 18 months now. Most of the
real "random" failures reported on various Mac OS X "troubleshooting" sites
after a user has installed an Apple software update are actually
manifestations of this bug. By real I mean not imagined problems or ones
that have been there for a very long time but the user is just now noticing
it and artificially connecting the cause to the recent update (it's called
Pareidolia). Yes, this nasty prebinding bug has been reported to Apple and
yes, it is 100% reproducible if you want to reproduce it.
Every single time you install an update to Mac OS X whether it be an iTunes
update, a QuickTime update, an update for daylight saving time, a security
update, an Airport update, or an actual Mac OS X update, you can be hit by
this bug. In order to prevent yourself from being smacked in the face by
this bug, follow this simple rule: When "Optimize System Performance"
appears during the update process do not touch your computer and definitely
do not launch any applications. Just back away from your computer box as if
it were a swarm of bees. Yes, it does mean that if you install the Mac OS X
10.4.9 update, you may get hit by the bug.
I think it's important to note that APE's use of update_prebinding at login
is not affected due to the time at which APE runs it on the ICBMs. In order
for the nasty prebinding bug to manifest, multiple prebinding operations
must be going on (either explicit or implicit). Assuming some implicit
action needed to be done, it's already been done by the login window
application (the one that shows that bar graph when you boot) on any shared
libraries it links to. And once you log in, even forcing a prebinding
operation won't actually cause any files to be reprebound since that action
was just done.
Symptoms and Signs
The worst sign you've been hit by this bug is an inability to boot after
installing a Mac OS X update. Sometimes the little wheel will just keep on
spinning. Other times you'll get to the point where you should see your
desktop but all you see is a blue screen (because [the] loginwindow is
repeatedly crashing due to a missing library). The "easiest" sign is an
application will crash either at launch or when you do a specific action and
the console.log /Applications/Utilities/Console (or a crash log) will spew
out a message about dyld that says: "Reason: no suitable image found." and
then sometimes "file to short" [sic]. The file is too short because it is
zero-length. There is an example of someone being hit by this on the
internets.
Sadly, most people suffering from this bug never have a chance to see an
error message to find out what file was zero'd out, especially when it
prevents boot. One of the things that can help a lot in troubleshooting the
problem is booting in verbose mode. Setting this option is really quite
easy. Open the terminal (/Applications/Utilties/Terminal) and type:
sudo nvram boot-args="-v"
Then just enter your password. From then on, every time you reboot, you'll
be in verbose mode. You can also hold Command-V at boot time to boot into
verbose mode for that boot only.
Verbose mode basically just doesn't show the Apple logo (OEM logo on the
ICBMs) at boot time. You get to see all the gritty ugly details in booting
Mac OS X. Note: You will not understand everything that's going on during
booting in verbose mode. You don't need to. All you need to do is recognize
what is normal and what isn't. Also, it really helps to boot into verbose
mode after an upgrade as the first reboot can take an exceedingly long time
(up to 10 minutes) and booting into verbose mode will help you know your Mac
hasn't frozen. (Seeing "diskarb not ready" is normal, especially after a
system update).
Preventative Measures
Well, I already listed it: When "Optimize System Performance" appears during
the update process do not touch your computer and definitely do not launch
any applications. Just back away from your computer box as if it were a
swarm of bees.
I should note that this bug seems a lot more likely to happen on an ICBM as
the prebinding operation has to work on two sides of a fat file (the x86
side and the PowerPC side) so it takes longer, which means there is more
time for you to trigger the bug.
Solving the Problem if You're Hit
Recovering from this bug is not the easiest thing in the world. If it's a
simple crash, you can just copy a "good" version of the file off a Mac
running the exact same build as the busted one or you can just run the most
recent combo updater (assuming the zero'd out file is in the combo updater).
If it is really bad, you'll have to boot the busted Mac into FireWire target
disk mode and try to run the combo updater off a working machine. TDM is not
an option if the other Mac has a different processor architecture of it is
is running a newer version of Mac OS X.
What Will Apple Do?
Who knows. This bug has been filed with Apple, along with steps to reproduce
it 100% of the time (at least in my testing). It was marked as a duplicate,
which means the bug was already in Apple's system before I filed it. And
since it is duplicate, I don't know what is going on with it. Yes, before
anyone mentions it, I know prebinding is deprecated. However, Mac OS X still
does it when installing Apple updates. It doesn't matter that it is
deprecated if it still happens, after all.
Even if prebinding goes completely away in Mac OS X 10.5 that doesn't solve
the problem for Mac OS X 10.4.x users. Security updates will continue to be
released for Mac OS X 10.4.x until Mac OS X 10.6 is released (which could be
many, many years from now) and every one of those updates carries the risk
of "destroying" the Mac of the person that installs it until this bug is
fixed.
--
Diane Ross, Microsoft Mac MVP
Entourage Help Page
<http://www.entourage.mvps.org/>
One of the top five MS Entourage resources listed on the Entourage Blog.
<http://blogs.msdn.com/entourage/>