Some recent projects I’ve been involved in have required some serious processing power to render gigapixel images and the original 2xDual Core XEONs weren’t up to it. I found a matching pair of cheap E5345 QUAD core processors on eBay and fitted them (that’s another post that will be on here soon along with photos of the CPU swap out process). I threw some SSD’s in a RAID array in there for good measure too and the combination of those two things made a phenomenal difference to the rendering times. My Snow Leopard 10.6.7 install was nice and stable, not a single crash/hang/kernel panic or reboot.

Then Apple released a Snow Leopard update 10.6.8 to “ready you for Lion” so I dutifully upgraded the 10.6.7 I had installed on my SSD’s and it all went terribly wrong. The system now erratically rebooted itself and a look at the logs showed Kernel Panics all over the place. I have a second OS X 10.6.7 for emegency situations, booted into that and ran that for a few days and it was all fine. I rolled back my 10.6.8 to 10.6.7 using a TimeMachine backup and everything was stable again.

I thought it might just be a *glitch* with 10.6.8 and patiently waited for Lion. When Lion was released I decided to not take a chance upgrading my stable 10.6.7 and so I wiped 2 of the drives I had in a RAID array and installed a fresh copy of Lion which I downloaded from the App Store (if you want to know how to create a bootable Lion install from Apples Lion download leave a comment and I’ll post that soon). Even the fresh install of Lion was showing a random kernel panic every 5minutes to an hour and rebooting the system.

I thought it might be that 10.6.8 and Lion didn’t like the SSD drives that were still mounted even though they weren’t the boot drives. The MacPro 1,1 came with 4x drive sled bays for SATA hard drives but there were also another 2x hidden SATA ports on the motherboard that I connected the SSD drives to and mounted them in the spare optical bay. So I unplugged the SSD drives from the 2 hidden SATA ports and set about using 10.6.8 and Lion (each OS on a different RAID partition) but still both were rebooting because of a kernel panic.

I’ve opened up a report at http://bugreport.apple.com/ and they’ve requested my kernel crash logs which are in /Library/Logs/Diagnostic….

These from 10.6.8

another

another

another

The following is a crash log from a completely clean and fresh install (not upgrade) of Lion on a new RAID array

and the latest Lion one after upgrading my graphics card to a 5770.

This post will continue to get updated as and when there is a cause or solution found to my crashing.

3rd August 2011 – Have heard from others who have upgraded a MacPro 1,1 like me and running Lion without a problem. They all seem to have upgraded their graphics card to Apple’s ATI 5700 card so I ordered one of these from Apple today for a whopping £203 which should hopefully arrive tomorrow. Once installed I shall boot back into my fresh Lion installation for a few hours and see if there are any more crashes.

4th August 2011 –  I received  the new 5770 Apple upgrade kit and will set about installing this later on this evening and report back……..
…the 5770 was installed without a hitch and the MacPro booted up first time with no errors. I logged into my stable 10.6.7 and tinkered around playing 1080p videos full screen on my 30″ ACD and lots of GPU intensive tasks and it seemed much smoother at HD video playback etc. After a few hours I decided the card was working as it should so crossed my fingers and booted into my Lion installation. Again all seemed ok, nice smooth 1080p playback, I even tried some 4k video playback and it was smooth full screen. After about half an hour I thought maybe it was all stable so I thought I’d do a bit of housekeeping and repair any disk permission errors using Disk Utility. However, Disk Utility still refuses to verify or repair permissions of RAID arrays in Lion. There’s a lot of info on the Apple discussion forums about this so I opted to try verifying permissions from Terminal and it found some errors. So I then tried to repair those errors via terminal and left that running and went to bed. About 2 mins later I heard the tell-tale “boing” of the Apple start-up chime…it had crashed and rebooted whilst trying to repair permissions.

5th August 2011 - I went back to the machine this morning and sure enough I was presented with the log in screen as it had rebooted after a kernel panic. I logged in but there was no prompt to submit a crash report to Apple as there normally is and there is nothing logged in /Library/Logs/Diagnostic/ where the kernel panic reports normally live. So I can rule out my graphics card as the KP (Kernel Panic) happens with both original 7300GT and brand new Apple ATI 5770 cards. Time to start eliminating other items. Next on my hit list is RAM. I only have 7GB installed and it all seems ok when I’m booted in 10.6.7 and the Apple Hardware Test that comes on the OS X install CD returned no errors but for completeness I thought I’d run some extensive memory tests. I downloaded two programs Rember and Memtest, the former is a GUI for the latter. I tried running Rember from within OS X and it returned no errors but as some of the memory is not accessible whilst booted into the OS I installed Memtest and rebooted my computer into single user mode (hold down CMD+S when you reboot). Once in single user mode I typed

to run memtest tests on all memory sectors twice. I’ve left that running and come to work but I’ll report back later if it found any errors. If there are no RAM errors I might take half the modules out anyway, boot Lion and see if it KPs and if it does swap with the other half of the RAM and see if that KPs. If it does it KPs with every single module then I doubt it’s a RAM issue. I’ll then look to install a non-RAID drive and install a fresh copy of Lion on that to see if the KP still happen on a non-RAID drive. I still think this is an Apple induced issue with 10.6.8 and Lion, I’m going to start looking into whether 64bit EFI might be to blame (the MacPro only has a 32bit EFI but 64bit capable processors).

Memtest completed two complete thorough RAM module checks and all passed without a fault so RAM ruled out.

19th August 2011 - Right I’m still getting KPs on my Lion test install and am sticking with 10.6.7 for now so I can keep working but I’ve now pretty much ruled out any hardware faults.

• I bought a new ATI 5770 to replace my 7300GT and still KP.
• I removed my SSD RAID array on the 5th and 6th SATA Ports and still KP.
• I tried booting with only Lion HDDs in and removed the 2nd RAID array and still KP.
• I bought another 8GB RAM and ran with just that in on its own and still get KP so RAM ruled out.
• I ran fans at near full speed and COU core temps stayed below 40°C and still KP.
• I ran fans at default speed and cranked my CPU usage to 100% for about 4 hours and no KP in LION so I doubt the CPUs are bad. Once I stopped the CPU stress test I opened Safari and browsed the web a little and it KP’d.
• I’ve run with WiFi and Bluetooth disabled to see if those were a cause but still got KP. I might try removing them from the motherboard completely though just to check but they are genuine Apple modules.

I also tried tinkering around copying IOPCI kexts and IOPLatformFamily kexts and copying copies of these from a working 10.6.7 installation over the Lion versions but that didn’t work either.

Where does this leave me? Basically nowhere. I’m still buggered if I use 10.6.7 or 10.7.0 or 10.7.1 and get regular KPs. I’ve not managed to go more than 1 hour without a KP, except funnily enough when I was stress testing the CPU and it could happily sit there all day at 800% CPU usage. Having now bought another 8GB of RAM and run with just that in and the original 7GB removed I can rule out having bad RAM, plus the fact it works in 10.6.7 for days without a KP means it definitely software related.

I still haven’t got around to breaking apart one of my RAID arrays and trying to install Lion on just a single non-RAIDed drive. I do still think RAID could be part of the cause, since in Lion I still cannot verify or repair disk permissions on a RAID array using disk utility but I can using diskutil command in terminal but most of the time I get a KP about 90% through….suspicious. But then my hopes are dashed by people like @poo in the comments who has the same issue but doesn’t have any RAID arrays. But then again his problems may be something completely different.

For now I’m sticking with 10.6.7 and every so often trying Lion and some tweaks but having now spent £203 on an ATI 5770 and £180 on another 4GB RAM I’m reluctant to waste much more time or money on this and just hope Apple find a solution/cure soon?

1st September - I’m still running 10.6.7 and had no time to do any more experimenting to resolve my KPs but I have just had a reply from Apple Tech via bugreport.apple.com (some 6 weeks later) and they now need a complete Kernel Core Dump which I’ve never heard of before and they’ve included some complex instructions on how to setup a 2nd Mac running OS X as a Kernel Dump Server which has to be connected via ethernet to my MacPro and then modify the nvram of my MacPro so that all kernel dumps are sent to the IP address of the 2nd computer. That way they grab every single kernel instruction and upload that to them. The file could be up to 1GB in size so they’ve given me ftp details to upload the file. So it seems Apple are investigating this which is promising. I’ll update once I hear any more.

13th September - I eventually found time to copy clone the contents of 2 of my drives that were in a RAID array to my Mac Mini Server and pulled them out. I installed an old 80GB Seagate Barracuda drive in HDD Bay 3, booted into my good 10.6.7 and used Disk Utility to format it as HFS with a GUID partition. I then downloaded the latest Lion installation package (10.7.1) from the App store and installed that on the newly formatted drive. Once booted I ran the terminal commands as instructed by Apple Tech Support so that any crash logs will now be saved across the network to my Mac Mini Server and am currently waiting for a kernel panic. Sod’s law it won’t bloody KP now! I’ve unmounted the other remaining drives in the MacPro so that to all intents and purposes this is now just a MacPro with a single 80GB (non-raid) drive installed. Guess I’ll just have to keep playing in Lion to see if it’s become any more stable. NOTE – Disk Utility now lets me verify and repair permissions on the Lion disk which it didn’t before when I had Lion installed on a RAID array. Watch this space.

14th September - Dammit! After using it for several hours last night it seemed fine, went to bed, woke up and it had kernel panic in the night and rebooted, the crash log for submission to Apple mentioned “spinlock acquisition timed out” or similar I’ve seen that many now I don’t take much notice to them. I checked my MacMini Server which is meant to collect the core kernel dumps and there was nothing. Even though I had changed the nvram settings as per Apple’s request so my MacMini Server would collect all the core kernel dumps, I’d forgotten to restart the computer after making nvram changes. So I restarted again, and then forced a test kernel panic (if you want to read up more on how to setup a 2nd machine to collect your core kernel dumps read this) to check if the core dumps were now going through to my MacMini Server but they still weren’t. I’ve left a message with Apple tech for help on this matter but in the interim I’m going to try and setup a MacBook Pro as a dump server and see if that works. Anyway, we can rule out RAID as being the cause of any KPs. I know one or two commenters below said they get KP and don’t use RAID, not that I didn’t believe them, I just had to run through all my elimination processes. RAM replaced, new HDD (non-RAID), new 5770 Graphics card….erm I’m running out of replaceable hardware items that could be to blame. I really hope it is an Apple bug with 10.6.8 and Lion (10.7 and 10.7.1) and not faulty CPU. I’ve now accepted that I’m going to have to try eliminating the CPU in all this and need to roll back to stock CPUs. I’d like to get the CoreDump working properly first and submit them to Apple so that can hopefully see somthing that I can’t. Sorry to say this again, but watch this space…

NOTE – It is becoming more and more apparent that only people with upgraded CPU are suffering from kernel panics and crashes in Lion. From the comments below you will see people with X5355, X5365, E5345 etc. Did the MacPro 1,1 ever ship with 8-cores and if so what was the factory OEM 8-core CPU that it shipped with?

15th September - Right so I’ve not had a Kernel panic for nearly 2 days. What did I do to accomplish this, I wish I could say for sure. First things first, I tried setting up my MacMini Server to be a core kernel dump so when i did have a KP 1GB+ of crucial logs would be logged on there for me to upload to Apple. I had problems with this, if I tried to force a kernel panic by typing:

sudo dtrace -w -n “BEGIN{ panic();}”

at the command prompt it would begin connecting to my MacMini Server’s IP, correctly find it and retrieve it’s MAC address and then timeout trying to send the logs. After hours and hours trying to figure it out, including setting up a Macbook Air and Pro as a core dump server I finally figured out that by default the kernel dumps go out over ethernet EN0 which is ethernet port 1 and I had my cable plugged into EN1 (ethernet port 2). Once I switched ports and triggered a kp manually the core dumps were sent correctly to my remote machine. OK so all I needed now was to wait for it to kernel panic. Didn’t take long, mouse froze and then poof it rebooted without any kernel core dump going to my remote server. WTF! That didn’t work then. Normally I get a kernel panic screen or the terminal prompt scrolling across the top of my desktop image. Right I’ll just keep using it until I get a proper kernel panic I thought.

So carried on using it for a day with basic core apps, safari, chrome and that’s about it. No crashes. I left it playing full screen streaming HD from the NASA site all night to see if overheating GPU or similar could be the cause, nothing. Now here’s some things I did do

but I’m sure I’d done before.

1. I had to update my flash when I went to NASA website I think so I downloaded latest Flash.
2. I removed my ethernet port 2 from Network pane in System preferences and I mean removed not just turned off.
3. I removed my Airport from Network pane in System preferences and I mean removed not just turned off. There have been others with iMacs reporting Airport cards causing KPs and to roll back kexts, since I’m hardwired to my TimeCapsule I just thought fuck it and removed it from the list.
4. I normally have an Apple iSight camera connected to the Firewire port on my Apple 30″ cinema display, I have unplugged that.
5. I normally have a USB multi-card reader connected to the USB ports on my 30″ ACD, I have disconnected that.

I’ve now been up for 1 day 22 hours. In that time I’ve setup two email accounts in Mail app and downloaded 30,000+ emails and I’ve installed iLife 11 and run software update to bring all the iLife apps up to date.

There are only 3 official sources for the iPhone 3G in the UK:

1 – Apple
2 – Carphone Warehouse (CPW)
3 – O2

If you are an existing iPhone owner and wish to upgrade you cannot go to an Apple store! All upgrades for current iPhone users have to be done through Carphone Warehouse or O2.

The problem is, Apple have so far not given any stock of the White iPhone 3G to either O2 or CPW. This effectively means that existing iPhone users cannot upgrade to the White iPhone 3G and must, for now at least, settle for the black version.

I’ve made near daily visits and several phone calls to both O2 and Carphone Warehouse trying to confirm a date that they’ll be receiving stock of the White iPhone 3G so that I can upgrade and some of the responses I’ve been getting are shocking:

“oh no sir, there’s no white iPhone only black, you must be confusing it with the iPods?”

“the White iPhone is going to be exclusive to the Carphone Warehouse, O2 won’t have any”

“the White iPhone is going to be for Pay As You Go customers only”

“The White iPhone will launch between October and December along with the PAYG version”

For Pay As You Go customers only!! Don’t talk rubbish, why would a PAYG customer (who’s going to be paying well over the odds for the phone in the first place) be restricted to getting the White one, which is 16GB only? Christmas time? Really? The Apple store has plenty of them in right now.

There have been a few people who decided they didn’t want to wait and purchased a White iPhone 3G in the Apple stores (who have plenty) and signed up to a new O2 contract in store. When they got home they just called O2 and played *dumb* pretending they thought that was the way they were supposed to upgrade and had their original contract moved over and the new contract just signed in store cancelled. This is risky though, if O2 don’t take pity on you or think you’re trying to pull a fast one they could leave you stuck with 2 simultaneous iPhone contracts!

I’m the kind of person who would have normally tried doing the above *playing dumb* trick by now but unfortunately it’s not that straight forward for me. I need to simultaneously upgrade my existing iPhone and migrate from a personal to the newly available business tarriffs. I also need to do the same for one of my employees and migrate us both to the same shared business plan and then migrate two current O2 Nokia users to the same shared iPhone tariff but using the two superseeded original iPhones…and as luck would have it, all this is possible but only at an O2 store who don’t have White iPhones.

If you happen to know of any O2 stores that have White iPhones in stock please do leave a comment below and put me out of my misery.

UPDATE (23rd August 2008): Good news, existing iPhone owners can now officially upgrade to the white iPhone 3G at any Carphone Warhouse or O2 store (stock levels permitting). I walked by both CPW and O2 stores today and saw signs advertising white iPhones now in stock. Half an hour later and I’m walking out the door with a white iPhone 3G. I asked when they had stock of the white ones and they said yesterday (22nd August) so if you’ve been waiting, go get one, the white is DEFINITELY better than the black in my opinion as it doesn’t show up any finger marks or scratches on the back!

I picked my iPhone on launch day in the UK (November 9th) and after making a few calls several friends complained of a “creaking” sound coming down the phone.

I eventually narrowed the sound down to the plastic housing for the speaker and microphone, applying gently pressure to the sides of the housing when holding the phone caused the plastic to flex and creak. As the microphone is housed in this area the small sound was greatly amplified down the line to my friends.

I took a trip back to the Carphone Warehouse the next morning, explained and demonstrated the problem and they instantly agreed it was a fault and would issue a replacement. The problem was none of the sales staff were able to talk me through the exchange procedure for the iPhone as I was the first to return a faulty iPhone. They seemed to be under the impression when I connected the replacement iPhone to iTunes I would have to start a new 18 months contract and ask for a refund and termination of the previous contract. This seemed ludicrous to me so after an hour wait in store whilst they called somebody for advice they finally gave me a new iPhone but removed the SIM from my old faulty phone and installed it in the new one.

That was it as far as they were concerned, it should just work! WTF of course it won’t, I powered it on in front of them to prove my point, and sure enough the phone was in Emergency Call only mode until I connected to iTunes.

Somebody then reassured me that iTunes would recognise the phone as a replacement and I’d simply have to enter my old number to migrate my account to the newer handset. This brought up a new problem, since I was migrating from Vodafone to O2 and was porting my number, I had been issued with a temporary O2 phone number for 7 days. Did I enter this temporary O2 number or my number being ported across if iTunes asked?….blank faces from all the staff again!

I left the store and decided to just try it and see what happened, and for the benefit of anybody else wandering how the replacement iPhone process differs from the new sign up procedure I took screen shots at each stage:

Step 1: First Screen shown when replacement iPhone connected.

Step 2: iTunes knows an iPhone is already registered to my account and asks for my existing mobile number. At this stage I decide to enter the temporary mobile number that I’ve been assigned until my number ports from my old network.

Step 3: It then asks me to confirm my Apple ID and password.

Step 4: A brief wait while it checks my Apple ID against the mobile number given.

Step 5: I am then asked to review the Bill name and change to mobile number. I notice that it’s defaulted to the 9991231234 phone number for some reason but I go with it and see what happens.

Step 6: I’m presented with an error saying the iTunes store may be busy!

Step 7: I’m then present with the same review your O2 account information page but this time it’s picked up my O2 username and external email address I signed up to O2 with along with the temporary phone number I previously entered.

Step 8: Another short wait while it processes my activation. You’ll notice that it didn’t ask me to agree to any T&C’s again, nor did it ask me to verify my tariff.

Step 9: Activation complete it now asks if I wish to restore a backup taken from my previously faulty iPhone.

Step 10: I chose the most recent backup and hit continue and it takes a few seconds to restore the backup. (note I didn’t have any music or photos in the backup simply contacts, calendar and email settings)

Step 11: All done. The replacement phone is now connected to my iTunes and O2 accounts, the old phone contents have been restored on the new iPhone and I’m all set to go.