Some recent projects I’ve been involved in have required some serious processing power to render gigapixel images and the original 2xDual Core XEONs weren’t up to it. I found a matching pair of cheap E5345 QUAD core processors on eBay and fitted them (that’s another post that will be on here soon along with photos of the CPU swap out process). I threw some SSD’s in a RAID array in there for good measure too and the combination of those two things made a phenomenal difference to the rendering times. My Snow Leopard 10.6.7 install was nice and stable, not a single crash/hang/kernel panic or reboot.

Then Apple released a Snow Leopard update 10.6.8 to “ready you for Lion” so I dutifully upgraded the 10.6.7 I had installed on my SSD’s and it all went terribly wrong. The system now erratically rebooted itself and a look at the logs showed Kernel Panics all over the place. I have a second OS X 10.6.7 for emegency situations, booted into that and ran that for a few days and it was all fine. I rolled back my 10.6.8 to 10.6.7 using a TimeMachine backup and everything was stable again.

I thought it might just be a *glitch* with 10.6.8 and patiently waited for Lion. When Lion was released I decided to not take a chance upgrading my stable 10.6.7 and so I wiped 2 of the drives I had in a RAID array and installed a fresh copy of Lion which I downloaded from the App Store (if you want to know how to create a bootable Lion install from Apples Lion download leave a comment and I’ll post that soon). Even the fresh install of Lion was showing a random kernel panic every 5minutes to an hour and rebooting the system.

I thought it might be that 10.6.8 and Lion didn’t like the SSD drives that were still mounted even though they weren’t the boot drives. The MacPro 1,1 came with 4x drive sled bays for SATA hard drives but there were also another 2x hidden SATA ports on the motherboard that I connected the SSD drives to and mounted them in the spare optical bay. So I unplugged the SSD drives from the 2 hidden SATA ports and set about using 10.6.8 and Lion (each OS on a different RAID partition) but still both were rebooting because of a kernel panic.

I’ve opened up a report at http://bugreport.apple.com/ and they’ve requested my kernel crash logs which are in /Library/Logs/Diagnostic….

These from 10.6.8

another

another

another

The following is a crash log from a completely clean and fresh install (not upgrade) of Lion on a new RAID array

and the latest Lion one after upgrading my graphics card to a 5770.

This post will continue to get updated as and when there is a cause or solution found to my crashing.

3rd August 2011 – Have heard from others who have upgraded a MacPro 1,1 like me and running Lion without a problem. They all seem to have upgraded their graphics card to Apple’s ATI 5700 card so I ordered one of these from Apple today for a whopping £203 which should hopefully arrive tomorrow. Once installed I shall boot back into my fresh Lion installation for a few hours and see if there are any more crashes.

4th August 2011 –  I received  the new 5770 Apple upgrade kit and will set about installing this later on this evening and report back……..
…the 5770 was installed without a hitch and the MacPro booted up first time with no errors. I logged into my stable 10.6.7 and tinkered around playing 1080p videos full screen on my 30″ ACD and lots of GPU intensive tasks and it seemed much smoother at HD video playback etc. After a few hours I decided the card was working as it should so crossed my fingers and booted into my Lion installation. Again all seemed ok, nice smooth 1080p playback, I even tried some 4k video playback and it was smooth full screen. After about half an hour I thought maybe it was all stable so I thought I’d do a bit of housekeeping and repair any disk permission errors using Disk Utility. However, Disk Utility still refuses to verify or repair permissions of RAID arrays in Lion. There’s a lot of info on the Apple discussion forums about this so I opted to try verifying permissions from Terminal and it found some errors. So I then tried to repair those errors via terminal and left that running and went to bed. About 2 mins later I heard the tell-tale “boing” of the Apple start-up chime…it had crashed and rebooted whilst trying to repair permissions.

5th August 2011 - I went back to the machine this morning and sure enough I was presented with the log in screen as it had rebooted after a kernel panic. I logged in but there was no prompt to submit a crash report to Apple as there normally is and there is nothing logged in /Library/Logs/Diagnostic/ where the kernel panic reports normally live. So I can rule out my graphics card as the KP (Kernel Panic) happens with both original 7300GT and brand new Apple ATI 5770 cards. Time to start eliminating other items. Next on my hit list is RAM. I only have 7GB installed and it all seems ok when I’m booted in 10.6.7 and the Apple Hardware Test that comes on the OS X install CD returned no errors but for completeness I thought I’d run some extensive memory tests. I downloaded two programs Rember and Memtest, the former is a GUI for the latter. I tried running Rember from within OS X and it returned no errors but as some of the memory is not accessible whilst booted into the OS I installed Memtest and rebooted my computer into single user mode (hold down CMD+S when you reboot). Once in single user mode I typed

to run memtest tests on all memory sectors twice. I’ve left that running and come to work but I’ll report back later if it found any errors. If there are no RAM errors I might take half the modules out anyway, boot Lion and see if it KPs and if it does swap with the other half of the RAM and see if that KPs. If it does it KPs with every single module then I doubt it’s a RAM issue. I’ll then look to install a non-RAID drive and install a fresh copy of Lion on that to see if the KP still happen on a non-RAID drive. I still think this is an Apple induced issue with 10.6.8 and Lion, I’m going to start looking into whether 64bit EFI might be to blame (the MacPro only has a 32bit EFI but 64bit capable processors).

Memtest completed two complete thorough RAM module checks and all passed without a fault so RAM ruled out.

19th August 2011 - Right I’m still getting KPs on my Lion test install and am sticking with 10.6.7 for now so I can keep working but I’ve now pretty much ruled out any hardware faults.

• I bought a new ATI 5770 to replace my 7300GT and still KP.
• I removed my SSD RAID array on the 5th and 6th SATA Ports and still KP.
• I tried booting with only Lion HDDs in and removed the 2nd RAID array and still KP.
• I bought another 8GB RAM and ran with just that in on its own and still get KP so RAM ruled out.
• I ran fans at near full speed and COU core temps stayed below 40°C and still KP.
• I ran fans at default speed and cranked my CPU usage to 100% for about 4 hours and no KP in LION so I doubt the CPUs are bad. Once I stopped the CPU stress test I opened Safari and browsed the web a little and it KP’d.
• I’ve run with WiFi and Bluetooth disabled to see if those were a cause but still got KP. I might try removing them from the motherboard completely though just to check but they are genuine Apple modules.

I also tried tinkering around copying IOPCI kexts and IOPLatformFamily kexts and copying copies of these from a working 10.6.7 installation over the Lion versions but that didn’t work either.

Where does this leave me? Basically nowhere. I’m still buggered if I use 10.6.7 or 10.7.0 or 10.7.1 and get regular KPs. I’ve not managed to go more than 1 hour without a KP, except funnily enough when I was stress testing the CPU and it could happily sit there all day at 800% CPU usage. Having now bought another 8GB of RAM and run with just that in and the original 7GB removed I can rule out having bad RAM, plus the fact it works in 10.6.7 for days without a KP means it definitely software related.

I still haven’t got around to breaking apart one of my RAID arrays and trying to install Lion on just a single non-RAIDed drive. I do still think RAID could be part of the cause, since in Lion I still cannot verify or repair disk permissions on a RAID array using disk utility but I can using diskutil command in terminal but most of the time I get a KP about 90% through….suspicious. But then my hopes are dashed by people like @poo in the comments who has the same issue but doesn’t have any RAID arrays. But then again his problems may be something completely different.

For now I’m sticking with 10.6.7 and every so often trying Lion and some tweaks but having now spent £203 on an ATI 5770 and £180 on another 4GB RAM I’m reluctant to waste much more time or money on this and just hope Apple find a solution/cure soon?

1st September - I’m still running 10.6.7 and had no time to do any more experimenting to resolve my KPs but I have just had a reply from Apple Tech via bugreport.apple.com (some 6 weeks later) and they now need a complete Kernel Core Dump which I’ve never heard of before and they’ve included some complex instructions on how to setup a 2nd Mac running OS X as a Kernel Dump Server which has to be connected via ethernet to my MacPro and then modify the nvram of my MacPro so that all kernel dumps are sent to the IP address of the 2nd computer. That way they grab every single kernel instruction and upload that to them. The file could be up to 1GB in size so they’ve given me ftp details to upload the file. So it seems Apple are investigating this which is promising. I’ll update once I hear any more.

13th September - I eventually found time to copy clone the contents of 2 of my drives that were in a RAID array to my Mac Mini Server and pulled them out. I installed an old 80GB Seagate Barracuda drive in HDD Bay 3, booted into my good 10.6.7 and used Disk Utility to format it as HFS with a GUID partition. I then downloaded the latest Lion installation package (10.7.1) from the App store and installed that on the newly formatted drive. Once booted I ran the terminal commands as instructed by Apple Tech Support so that any crash logs will now be saved across the network to my Mac Mini Server and am currently waiting for a kernel panic. Sod’s law it won’t bloody KP now! I’ve unmounted the other remaining drives in the MacPro so that to all intents and purposes this is now just a MacPro with a single 80GB (non-raid) drive installed. Guess I’ll just have to keep playing in Lion to see if it’s become any more stable. NOTE – Disk Utility now lets me verify and repair permissions on the Lion disk which it didn’t before when I had Lion installed on a RAID array. Watch this space.

14th September - Dammit! After using it for several hours last night it seemed fine, went to bed, woke up and it had kernel panic in the night and rebooted, the crash log for submission to Apple mentioned “spinlock acquisition timed out” or similar I’ve seen that many now I don’t take much notice to them. I checked my MacMini Server which is meant to collect the core kernel dumps and there was nothing. Even though I had changed the nvram settings as per Apple’s request so my MacMini Server would collect all the core kernel dumps, I’d forgotten to restart the computer after making nvram changes. So I restarted again, and then forced a test kernel panic (if you want to read up more on how to setup a 2nd machine to collect your core kernel dumps read this) to check if the core dumps were now going through to my MacMini Server but they still weren’t. I’ve left a message with Apple tech for help on this matter but in the interim I’m going to try and setup a MacBook Pro as a dump server and see if that works. Anyway, we can rule out RAID as being the cause of any KPs. I know one or two commenters below said they get KP and don’t use RAID, not that I didn’t believe them, I just had to run through all my elimination processes. RAM replaced, new HDD (non-RAID), new 5770 Graphics card….erm I’m running out of replaceable hardware items that could be to blame. I really hope it is an Apple bug with 10.6.8 and Lion (10.7 and 10.7.1) and not faulty CPU. I’ve now accepted that I’m going to have to try eliminating the CPU in all this and need to roll back to stock CPUs. I’d like to get the CoreDump working properly first and submit them to Apple so that can hopefully see somthing that I can’t. Sorry to say this again, but watch this space…

NOTE – It is becoming more and more apparent that only people with upgraded CPU are suffering from kernel panics and crashes in Lion. From the comments below you will see people with X5355, X5365, E5345 etc. Did the MacPro 1,1 ever ship with 8-cores and if so what was the factory OEM 8-core CPU that it shipped with?

15th September - Right so I’ve not had a Kernel panic for nearly 2 days. What did I do to accomplish this, I wish I could say for sure. First things first, I tried setting up my MacMini Server to be a core kernel dump so when i did have a KP 1GB+ of crucial logs would be logged on there for me to upload to Apple. I had problems with this, if I tried to force a kernel panic by typing:

sudo dtrace -w -n “BEGIN{ panic();}”

at the command prompt it would begin connecting to my MacMini Server’s IP, correctly find it and retrieve it’s MAC address and then timeout trying to send the logs. After hours and hours trying to figure it out, including setting up a Macbook Air and Pro as a core dump server I finally figured out that by default the kernel dumps go out over ethernet EN0 which is ethernet port 1 and I had my cable plugged into EN1 (ethernet port 2). Once I switched ports and triggered a kp manually the core dumps were sent correctly to my remote machine. OK so all I needed now was to wait for it to kernel panic. Didn’t take long, mouse froze and then poof it rebooted without any kernel core dump going to my remote server. WTF! That didn’t work then. Normally I get a kernel panic screen or the terminal prompt scrolling across the top of my desktop image. Right I’ll just keep using it until I get a proper kernel panic I thought.

So carried on using it for a day with basic core apps, safari, chrome and that’s about it. No crashes. I left it playing full screen streaming HD from the NASA site all night to see if overheating GPU or similar could be the cause, nothing. Now here’s some things I did do

but I’m sure I’d done before.

1. I had to update my flash when I went to NASA website I think so I downloaded latest Flash.
2. I removed my ethernet port 2 from Network pane in System preferences and I mean removed not just turned off.
3. I removed my Airport from Network pane in System preferences and I mean removed not just turned off. There have been others with iMacs reporting Airport cards causing KPs and to roll back kexts, since I’m hardwired to my TimeCapsule I just thought fuck it and removed it from the list.
4. I normally have an Apple iSight camera connected to the Firewire port on my Apple 30″ cinema display, I have unplugged that.
5. I normally have a USB multi-card reader connected to the USB ports on my 30″ ACD, I have disconnected that.

I’ve now been up for 1 day 22 hours. In that time I’ve setup two email accounts in Mail app and downloaded 30,000+ emails and I’ve installed iLife 11 and run software update to bring all the iLife apps up to date.

Initially he thought it was just one of his Safari plugins but even after disabling all Safari plugins the [SP] still appears as shown in the screen shot below.

I suggested it could be to show that it’s a Secure Page hence the SP but it shows even on non-https pages and can clearly be seen on the basic vanilla Google homepage in the screenshot above.

If anybody is able to help shed any light on this please leave a comment below as he seems to be one of only a handful experiencing this strange problem.

UPDATE: Shortly after publishing this post, @benmarsh managed to solve it all on his own…he’d enabled the Debug tools in Safari and unchecked the option to “Use multi-process windows” so the [SP] stood for Single Process.

benmarsh Ben Marsh

Aha! [SP] refers to Single Process – I remembered I unchecked Debug -> User Multi-process Windows yesterday! D’oh!

I suddenly started getting http 403 errors on my self hosted [with native apache installation] site that worked perfectly until Feb 5th 2010. The errors showing up in the access log looked like:

[Fri Feb 19 02:05:25 2010] [crit] [client 127.0.0.1] (13)Permission denied: /Users/username/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable, referer: http://127.0.0.1/

I’m not sure what changed, perhaps an OS X update from Apple, but after restoring all /private/etc/apache2/* files and the my /Users/username/Sites/ folder via TimeMachine to a date when things were working it still wouldn’t play ball.

Eventually I came across the soultion, it seems the chmod levels of the default folders had changed so doing the following in Terminal:

chmod 701 ~
chmod 705 ~/Sites

Solved everything…phew

I opened up Activity Monitor to see if it was a RAM shortage when I noticed my CPU history was almost maxed out, but NOT by the photoshop process. My cpu cycles were getting chewed up by a system process called ATSServer .

My hard disk activity was high considering I wasn’t actively doing anything at the time.

A little research on the web indicated that ATSServer was OS X Spotlight’s indexing process. Others having this problem had recently copied a large number of PDF’s to their machine and that’s exactly what I had done too.

Spotlight doesn’t just index the filenames, where possible it indexes the entire contents of PDFs, emails, etc and the process seemed to be stuck trying to index the PDF’s I had recently added.

Activity Monitor showed that the process had a full 24hrs+ of cpu time so I don’t think it was simply a case of waiting longer for it to complete. Instead I went into Preferences->Spotlight and added the folder that had all the newly added PDF’s to the privacy folder settings. This, however, did not stop the runaway process nor the thrashing of the hard drive, so i took the decision to Force Quit the ATSServer process and all is well again.

I’m obviously going to have to reboot to start the service running and hopefully things will be back to normal.

I left them all downloading and updating but one, the Safari update, refused to work. It said I had insufficient access rights but considering I’m the only user and I’ve always been setup as an administrator this was strange.

I tried repairing disk permissions but that didn’t work. Then I thought perhaps it just needs to be restarted having just installed all the other updates so I dutifully rebooted and the same thing happened when I rebooted.

The Update Safari can’t be saved. You do not have appropriate access privileges. The installer package has been moved to the Trash. To try again, open the package from the Finder.

I checked the Trash and there was nothing in there so I had a root around and found an easy solution…using Finder to browse my hard drive I navigated to /Library/Updates/

and then deleted the Safari folder that was in there:

and then tried to install the Safari update again from the Software Update window and it downloaded and installed the update without a problem.

He’d never used a Mac before so I showed him the basics, email, surf, download, music, video, DVD player, iPhoto, etc and left him to it.

My Macbook Air has now got a fault and needs sending to Apple for 2 weeks for repair so I’ve relieved my father of the Macbook Pro for the 2 weeks I’m Air-less.

Considering my father’s been using the Macbook Pro regularly I was a little shocked to see the show-off widget showing 134 days uptime!

That’s more than I get out of my windows servers at work.

I know many of you will leave comments saying you’ve seen up-times on windows servers/linux boxes, etc going into the years BUT for a laptop, that’s in use by my computer inept father to remain up for over 4 months is worth a screenshot at least:

There were, however, 500MB of updates waiting from Apple so it’s time to wave good-bye to the show-off uptime and follow in the footsteps of Windows and reboot!

I have 2x4GB SDHC cards which each hold around 40mins on the highest quality setting, when they get full I normally just open iMovie 08 and import all the footage before wiping the card. The problem is, I sometimes do this on my MacBook Air and was shocked to find a warning saying I was low on disk space on the Air. I had 10GB free before I started importing 4GB of video but I didn’t realise that iMovie 08 un-packs all the video from the video camera’s compressed format so the 4GB ends up being ~16GB+. Importing the video on my MacBook Air is also quite a slow process, on my Mac Pro it’s not too slow but I needed a solution to copying the card contents whilst on the road in order to free up the space on the card.

After a little experimentation I’ve opted to use Apple’s built in application, Disk Utility, to make a straight clone of the SDHC card as a disk image file (dmg). The benefit of doing it this way into a disk image rather than just dragging and dropping the SDHC card contents into a folder on your desktop is when you remount the disk image file iMovie 08 recognises it as a connected camera and starts the video import wizard, creating the disk image is also stupidly faster than importing the footage into iMovie which requires heavy processor time.

I’m also using it as a useful backup procedure taking a snapshot of each SDHC card before I format them. Even though I have all videos ultimately imported into iMovie 08, should iMovie decide to corrupt its database or wipe all the imported files I can also start from scratch if I have clones of all the raw footage.

There primary reason, however, I want to backup the raw footage is to preserve the 5.1 surround sound data that the Panasonic SD-100 records. At present iMovie 08 doesn’t support anything other than stereo soundtracks, so regardless of how fancy your camera is and how many surround mics it has, if you are using iMovie 08 the 5.1 sound is going to get down-sampled to stereo. I know Final Cut Pro handles 5.1 surround channels but that’s a little too advanced and out of my price range at present. I’m hoping that iMovie 09 will support 5.1 channels and also BlueRay burning options but we’ll have to wait and see. Rather than kick myself next year when I suddenly have the ability to edit movies with 5.1 surround and realise the additional channels have all been down-mixed to stereo and lost forever, if I take a backup of the SDHC cards now into DMG format then I can re-import them as and when needed.

Create Disk Image (.dmg) from a SDHC card

Open you Applications folder and find the Utilities folder:

then select Disk Utility:

Ensure you have connected your SDHC card to your computer using a card reader and that it is appearing on your desktop or in your finder window. You should see the SDHC card mounted in the left pane of Disk Utility, more than likely it will have a sub-disk branching off it. Click on the SDHC card in the left pane, if there is a sub-disk make sure you click on the highest level (i.e. the top branche or the upper one of the two).

Once the disk is selected click on New Image at the top of the Disk Utility program window:

This will popup a new window asking you where you wish to save the image, I’d suggest you create a new folder called “SDHC Backups” and save them all in the one folder, be sure to give each image file a unique name so you can find it later:

Note that the default option is for a compressed image, whilst this might work and save you some space it’s not the quickest method to create a backup image and I haven’t tested how well it works, so change it to Read-Only with no encryption. I will do a few experiments with compressed images and see how processor intensive they are, any increase in time taken to create an image and how much space is actually saved compared to a non compressed read-only image, I’ll update this article once I’ve done that. UPDATE – On a Mac Pro the compressed image takes no longer to create however there is very little space saving, uncompressed = 3.68GB whilst compressed = 3.66GB so I’m going to stick with uncompressed read-only for now.

That’s it, hit Save and it will show you the progress:

You should now have an image file the same size as the SDHC card in the folder you specified on your hard drive. You can now eject the SDHC card from your computer, and just to double check, double click the image file you just created to mount it on your computer. You can click skip on the validating stage if you are feeling confident, once you see it mounted in Finder open up iMovie and it should detect that a camera has been connected even though the SDHC card is no longer plugged in.