Quantcast
Subscribe to magazine
You are not logged in, click here to log in.
Up to Discussions in Story Discussion
140 Replies Last post: Feb 1, 2006 2:39 PM by b6s   Go to original post 1 ... 6 7 8 9 10 Previous Next
Click to view tjustin7's profile New Member
tjustin7
100 posts since
Jun 26, 2005
120. Jan 24, 2006 4:50 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
Well, I don't think these tests necessarily contradict Apple's results.
The SPEC tests Steve Jobs ran essentially test only the PROCESSOR aspect of the machines, and thus, the maximum overall performance that could be achieved on an application that makes extensive use of the CPU (as opposed to memory, disk, etc).

It should then be pointed out that the 'Real World Tests' effectively represent Beta Applications compiled with a Beta Version of a compiler running on a Beta of an Operating System, and are being compared to highly tuned/refined versions.

For instance, I sincerely doubt that the applications included here take any sort of advantage of SSE/MMX/etc - but they're being compared to Apps that have most likely been thoroughly optimized for Altivec.

In addition, I suspect that Rosetta's performance will continue to get better as well as the emulation software itself is optimized better.

In addition, OSX for Intel will also continue to improve in performance - thus improving performance of all the other applications as well (Rosetta or not).

In other words, while these tests may not have been specifically crafted to show the Intel Mac in poor light, these tests results really are the worst results you will see.

That said, I look forward to a review in about a month or so, when an update or two has come out for these applications / the compiler / and the operating system, and the massive cumulative effect that those updates will have on performance.

I think then we will have a much better grasp of what sort of performance can be had from applications when you move to a PROCESSOR that is twice as fast.

- My $.02
Reply
Click to view tjustin7's profile New Member
tjustin7
100 posts since
Jun 26, 2005
121. Jan 24, 2006 5:20 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
As I've pointed out before, it's pretty rare (if not unheard of) that a regular user would do something like start some heavy-duty app running a filter or render and then switch over and do something else.

<hr />


I'm not sure that I could agree with that assertion; Where I work, I can't afford to watch a progress bar for minutes (let alone hours) at a time, so it's rarely the case that I'm NOT multi-tasking with a heavy duty app running in the background (running full tilt) while I continue to work on other things.

For instance, I work with clients that use TIFF files - instead of PDF files, for their documents. Thus, it's not unheard of for me to end up with 1000's of TIFF images that need to be converted into a PDF file, and then make the resulting PDF file a text-searchable file. The latter half of this operation can take a significant amount of time (it's taken hours on a P4 3.2Ghz before).

While that crunches away, I'm doing other tasks as well - not necessarily major processor intensive tasks, but enough that I have to wait for things.

Maybe I'm not what you call a regular user, but I still find it quite debatable that there are users out there that simply watch a progress bar go across the screen while they sit back and don't use their computer for anything else.

- My $.02
Reply
Click to view Jason Snell's profile Macworld Editorial
Jason Snell
2,215 posts since
Dec 11, 2000
122. Jan 24, 2006 5:24 PM in response to: tjustin7
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
For instance, I work with clients that use TIFF files - instead of PDF files, for their documents. Thus, it's not unheard of for me to end up with 1000's of TIFF images that need to be converted into a PDF file, and then make the resulting PDF file a text-searchable file. The latter half of this operation can take a significant amount of time (it's taken hours on a P4 3.2Ghz before).

<hr />


And are you using a consumer computer such as an iMac for this work?

Because this is one of the issues here -- the iMac is a consumer system. It's not a pro system. And so it's most appropriate to judge it based on the most likely behaviors of the people who are going to use it. I'm pretty sure a Cinema 4D render will scream on the Intel system relative to the G5; but do people really render stuff in Cinema 4D on an iMac? :-)

Jason Snell, Editorial Director, Macworld
Reply
Click to view Jason Snell's profile Macworld Editorial
Jason Snell
2,215 posts since
Dec 11, 2000
123. Jan 24, 2006 5:30 PM in response to: tjustin7
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
effectively represent Beta Applications compiled with a Beta Version of a compiler running on a Beta of an Operating System, and are being compared to highly tuned/refined versions.

<hr />


Well... "effectively" lets you get away with a lot. Nobody's calling iLife '06 a beta, but you're right, the iLife developers were most likely not developing it on the final version of 10.4.4 or on a finished iMac...

I don't think anyone disputes that some degree of performance will be wrung out of these systems in various software updates. In fact, if you read page two of the story that began this thread, you will find me saying that very thing.

However, the question you have to ask yourself is, what is the point of testing the new Intel iMac? In my mind, it's to gauge what performance regular iMac buyers will see when they buy one and try to run the most common apps on it. Is that a combination of chip speed, bus speed, RAM, cache, video card, MP-friendliness of apps, etc.? Absolutely. But hey, that's what this computer is. A combination of all those things. And when someone buys it and takes it home, they're taking it all home.

I suspect that many of the applications we've tested do indeed take advantage of SSE/MMX/etc, because applications that write to Apple's accelerate framework will automatically pick up either SSE/MMX or AltiVec depending on which architecture they're being compiled for.

One prognostication I will disagree with you about: I don't think Rosetta's going to ever get faster. I don't think Apple will be motivated at all to improve it (hello, Classic!), and so the only way you'll see Rosetta apps running faster is that the systems it runs on will get faster, and the rising tide will life Rosetta's boat. Of course, that's just a prediction -- only time will tell who's right and who's wrong.

Jason Snell, Editorial Director, Macworld
Reply
Click to view tjustin7's profile New Member
tjustin7
100 posts since
Jun 26, 2005
124. Jan 24, 2006 9:18 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
And are you using a consumer computer such as an iMac for this work?

Because this is one of the issues here -- the iMac is a consumer system. It's not a pro system. And so it's most appropriate to judge it based on the most likely behaviors of the people who are going to use it. I'm pretty sure a Cinema 4D render will scream on the Intel system relative to the G5; but do people really render stuff in Cinema 4D on an iMac? :-)

<hr />


I guess it largely depends on how you define "consumer level".
The PC's at my workplace are just plain, off the shelf PC's - my work PC has the same specs as the PC the receptionist has, so it's nothing extraordinary (i.e. 3.2Ghz P4, 512MB Ram, etc).
But it's not exactly in the same league as a PC with multiple processors / gigs of ram, etc, at least in my book. Certainly not a quad-core G5 system.


And you point out that you should test it based on common behavior of users, but the iDvd test you ran took 15+ minutes... and I would expect a 15 minute wait to induce some multi-tasking among most users (i.e. surfing the web, possibly listening to some music, etc).

In my mind, the iMac is like a light duty truck - sure, it's not going to be used to haul steel I-Beams or tons of rock... but that doesn't mean it's not going to see use that might actually require a truck (hauling a sofa, etc).

And of course, if I wanted to be obstinate , I might try to make the argument that nobody would dare run Photoshop on an iMac (and certainly not in emulation).
Reply
Click to view enrigonz's profile New Member
enrigonz
53 posts since
May 18, 2004
125. Jan 25, 2006 12:10 PM in response to: MW Forums
Re: First Lab Tests: iMac with Intel Core Duo processor
I think this move by Apple is very Intel-like. Benchmarks, speed comparisons are lame in my opinion. They should not try to sell their products base on this speed theory, it should all be about optimization and stability, instead they jump on the benchmark wagon sacrificing stability and in some cases usability.

I've been an Apple Mac user for some time and this latest move by Jobs is really disappointing. Why introduce an iMac when not so long ago he'd introduce a nice and new G5 iMac W/ eyesight? I understand the PowerBook line needing an upgrade but the iMac was just not necessary at this time, maybe 6 months from now.

I lost a bit of respect for Apple on this move, I use Apple because I like the stability of the OS and not how many seconds faster I can do one thing or the other, for that I could have built a really nice Windows system with a really nice CPU 10,000RPM hard drive, etc. etc. that could probably do a few things faster here-n-there.



Reply
Click to view madgunde's profile New Member
madgunde
64 posts since
Nov 18, 2005
126. Jan 25, 2006 1:11 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
Jason,

I don't think anyone is really questioning your benchmarking process on it's own. The reason it is being scrutinized so closely is because of the fact that you are using your benchmarks to call out Apple on their 2x speed claims. First off, we know Apple's speed claims are based on CPU benchmarks, not real-world benchmarks, and they never tried to present them as real-world. Had you not chosen to criticize Apple's benchmarks, or at least clarified that Apple's 2x claims are for the CPU only, people wouldn't be so defensive.

A 10-30% speed improvement on it's own is still a very big accomplishment, especially in just 3 months. If those numbers are the worst case scenario for Universal apps, then that bodes very well for the Core Duo's future as applications and the OS get better optimized. It just seems that people are so fixated on that 2x number Apple has on their web site, and in disproving it, that ANYTHING less than 2x is going to be used by naysayers and conspiracy theorists as proof that the Intel move was the wrong one, or that Apple is lying about why they are switching (that it doens't have anything to do with performance/watt).

I'd just like to make a point that your choice of benchmarks, while valid, are not necessarily useful at arriving at an average performance score for the new iMac because it was so limited. The number of apps you tested were limited, the number of tests were limited and the test conditions were very selective. What if you used a 1 hour file instead of a 1 minute file for the iMovie tests? That would have skewed your conclusion considerably.

I've read elsewhere for instance, where a Handbrake h.264 conversion from DVD was 3x faster. All it takes is the inclusion of one of those "best case" benchmarks, and your conclusion would have been very different. You would have been praising the new iMac for living up to it's hype. By my calculations, adding just the h.264 test from the previous link would have resulted in the iMac Core Duo being over 2x faster than the iMac G5.

To sum up, the overall "times faster" results would vary drastically depending on what apps are being tested, and what operations are being used in the calculation. That being the case, it would have made more sense for you to simply conclude that the iMac Core Duo is anywhere from 0.9x to 1.8x as fast depending on the app and operation. That is a more valid conclusion, and also makes Apple's claim of 2x faster sound much less questionable.

Andy C.
Brampton, ON Canada
Reply
Click to view Jason Snell's profile Macworld Editorial
Jason Snell
2,215 posts since
Dec 11, 2000
127. Jan 25, 2006 1:24 PM in response to: madgunde
Re: First Lab Tests: iMac with Intel Core Duo processor
You do realize that the post you linked to included five tests, and yet you chose to reference only the Intel-based iMac's best result?

You are doing very much what Apple has done -- highlight the best-case scenarios. Put simply, that's not our job. Our job is to try and portray what real users will find when they use these systems, not to be Apple apologists.

As to your complaint that we could've tested more stuff, well, that's one reason why this story is headlined "First Lab Tests"... we've actually done quite a lot more testing, as you'll see when we post more of our tests. In this thread alone I've revealed the results of our Doom 3 test, and I can report that last night I managed to run a rendering test that was 3x faster on the Intel-based iMac than on the iMac G5. That'll all go in forthcoming stories -- but it won't change the fact that for general uses, this iMac is not going to prove anything approaching 2x, which anyone who visits apple.com/imac will see is a performance standard that Apple is using broadly to market their product.

Jason Snell, Editorial Director, Macworld
Reply
Click to view madgunde's profile New Member
madgunde
64 posts since
Nov 18, 2005
128. Jan 25, 2006 1:39 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
You do realize that the post you linked to included five tests, and yet you chose to reference only the Intel-based iMac's best result?

You are doing very much what Apple has done -- highlight the best-case scenarios. Put simply, that's not our job. Our job is to try and portray what real users will find when they use these systems, not to be Apple apologists.

<hr />


I purposely chose the highest score to demonstrate how much of an impact selective and limited testing can have on the conclusion. If you included all of the tests from my link, your conclusions would still have been much higher than they were. Even just including your Doom 3 test would have radically skewed your conclusion. My whole point was that based on limited testing, conclusions would vary greatly depending on the choices made. That makes arriving at a "1.1-1.3x faster" conclusion just as accurate as Apple's 2x claim. If ~1x is the low end, and 3x is the high end of the testing spectrum, isn't Apple's 2x claim a fair middle-road position for them to take?

As an aside, I'm curious, was your Doom 3 a Univeral Binary or running under Rosetta? I ask because I hadn't heard that Doom 3 had been updated to a UB yet.
Reply
Click to view Jason Snell's profile Macworld Editorial
Jason Snell
2,215 posts since
Dec 11, 2000
129. Jan 25, 2006 1:50 PM in response to: madgunde
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
Even just including your Doom 3 test would have radically skewed your conclusion.

<hr />


I don't think so. However, the additional testing certainly adds a whole extra dimension to the story here (one that really is more fitting for a longer, more in-depth piece than a "first lab tests" story -- hint, hint): namely that certain tasks (rendering being a good one) really benefit from the move to a dual-core processor, while others (most consumer applications) generally don't.

In reply to:<hr />
If ~1x is the low end, and 3x is the high end of the testing spectrum, isn't Apple's 2x claim a fair middle-road position for them to take?

<hr />


But if you look at the tests, for most general-use stuff 1.2x is a far more common result than 2x or 3x. We can talk about the theoretical performance of these systems until we're blue in the face, but the average consumer buying an iMac today is just not going to see 2x from their apps unless they're doing a few very specific things, like (some) video encodes, (some) games, and (some) rendering.

In reply to:<hr />
As an aside, I'm curious, was your Doom 3 a Univeral Binary or running under Rosetta?

<hr />


Universal version (not publicly available yet). Rosetta won't be running anything at 2x speeds anytime soon...

Jason Snell, Editorial Director, Macworld
Reply
Click to view madgunde's profile New Member
madgunde
64 posts since
Nov 18, 2005
130. Jan 25, 2006 2:08 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
Thanks for the info about Doom 3. I think any consumer looking at playing games will definitely care more about how Doom 3 runs than whether iMovie is only slightly faster than the previous iMac.

Oh, and I just couldn't help responding to this:
In reply to:<hr />
You are doing very much what Apple has done -- highlight the best-case scenarios

<hr />

If Apple only highlighted the best-case scenario, then wouldn't they have said the new iMac was over 3x faster?

I agree with you 100% that your benchmarks are useful for people planning on using the new iMac for the kinds of tasks you tested. People need to take the conclusion with a grain of salt and realize that their mileage will vary. A gamer might be able to expect better than 2x performance for instance.

In any case, I think as time goes on, and more apps get recompiled, and with better optimization, Apple's 2x claim is going to seem pretty fair, even if it doesn't seem that way right now. Lets face it, the iLife apps is not where the biggest speed gains are needed anyway. It's playing games or performing long operations (like converting video for your iPod) where people will benefit most from the biggest speed gains on a consumer machine like the iMac. People are running iLife quite well on iBooks, Mac minis and PowerBooks after all, but forget playing Doom 3 on those machines...
Reply
Click to view Jason Snell's profile Macworld Editorial
Jason Snell
2,215 posts since
Dec 11, 2000
131. Jan 25, 2006 2:17 PM in response to: madgunde
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
If Apple only highlighted the best-case scenario, then wouldn't they have said the new iMac was over 3x faster?

<hr />


You seem to be suggesting that Apple's only run five tests on these systems. They haven't -- they've probably run dozens, if not hundreds. And yet they've chosen (understandably, from a marketing perspective) five of the very best, if not the five very best.

When I contacted Apple about our initial set of tests, we asked them what they had seen in terms of good performance, and where. Without telling tales out of school, let me just say that a lot of the examples were in the 35-40 percent improvement range, with a few more in the 75-80 percent range.

So it's not as if Apple isn't acutely aware of the true speed ranges of tasks on these systems. It's that they've chosen to promote the five tests that are 1.7x and above. Fair enough -- that's their right as marketers. But our tests just don't back up that range.

Jason Snell, Editorial Director, Macworld
Reply
Click to view jdb8167's profile Enthusiast
jdb8167
1,503 posts since
Aug 30, 2004
132. Jan 25, 2006 2:22 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
but it won't change the fact that for general uses, this iMac is not going to prove anything approaching 2x, which anyone who visits apple.com/imac will see is a performance standard that Apple is using broadly to market their product.

<hr />
That applies to any general purpose computing on any relatively modern CPU. Do you really see a large improvement in Word when using a 1.42 GHz G4 v. a 2.1 GHz G5? Does that mean the G5 isn't twice as fast? You are making too much of the 2x thing when Apple is very careful, as has been pointed out many times, to only say that the CPU is twice as fast. They are making a technical argument with facts to back them up. You are benchmarking something different and your benchmarks are valuable. But to say that to not bash Apple for marketing the CPU at 2x the speed is to be an apologist is just wrong.

Apple explains the benchmarks on their website and Steve Jobs was very clear in the MacWorld keynote to point out that CPU performance is 2x but overall system performance won't be. You and Apple are benchmarking different things. That doesn't make you right and Apple wrong. It just means you are benchmarking differently.

Again, I appreciate your benchmarks. But I have to agree that your coloring of the results as some repudiation of Apple is coming across as fodder for the Apple haters and others who want to "prove" that Apple made a mistake on the Intel transition. I doubt that this is your intention but the tone is there in my opinion.
Reply
Click to view Jason Snell's profile Macworld Editorial
Jason Snell
2,215 posts since
Dec 11, 2000
133. Jan 25, 2006 2:32 PM in response to: jdb8167
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
Do you really see a large improvement in Word when using a 1.42 GHz G4 v. a 2.1 GHz G5? Does that mean the G5 isn't twice as fast?

<hr />


Some strawmen here -- it might be better to use something more processor-intensive, like iMovie or Photoshop, than Word. Also, I'm pretty sure that for many tasks a 2.1GHz G5 isn't twice as fast as a 1.42GHz G4, given the clock speeds. :;

In reply to:<hr />
You are making too much of the 2x thing when Apple is very careful, as has been pointed out many times, to only say that the CPU is twice as fast.

<hr />


As I have pointed out many times, I don't think Apple's been remotely as careful as people say they have been. Jobs' words at the Keynote were, indeed, quite careful. Apple's iMac page and the benchmarks it uses, however, are not nearly as careful. If you don't think "iMac. 2x faster." isn't suggesting a whole lot more than ("2x faster at certain tasks"), we will just have to disagree. If Apple had really wanted to be careful about it, they wouldn't have said "Rev up your digital life at speeds twice as fast as the previous iMac." They would've said "Rev up your digital life at speeds as much as twice as fast as the previous iMac." Interestingly, the press release is more careful -- "delivering performance that is up to twice that of its predecessor." But does anyone seriously think that regular people haven't been given the impression that this computer is two times as fast as its predecessor?

In reply to:<hr />
Again, I appreciate your benchmarks. But I have to agree that your coloring of the results as some repudiation of Apple is coming across as fodder for the Apple haters and others who want to "prove" that Apple made a mistake on the Intel transition.

<hr />


I suspect that one of the reasons this thread is so long has to do with politics. Yes, there are "Apple haters" out there who want to use our results for their own purposes. There are also, I might point out, numerous Apple apologists who will accuse anyone who doesn't praise Apple of being "Apple haters." All we can do is try to be as clear, fair, and responsible as possible -- our job is not to be a bunch of Apple boosters, nor is it to attack Apple at every turn. If our complaints about Apple's 2x claims are arming the opposition, I respectfully suggest that this could have all been avoided by Apple being a bit more restrained in the way they've marketed the speed of these systems.

Jason Snell, Editorial Director, Macworld
Reply
Click to view jdb8167's profile Enthusiast
jdb8167
1,503 posts since
Aug 30, 2004
134. Jan 25, 2006 3:02 PM in response to: Jason Snell
Re: First Lab Tests: iMac with Intel Core Duo processor
In reply to:<hr />
Some strawmen here -- it might be better to use something more processor-intensive, like iMovie or Photoshop, than Word. Also, I'm pretty sure that for many tasks a 2.1GHz G5 isn't twice as fast as a 1.42GHz G4, given the clock speeds.

<hr />
Your benchmarks are highly IO intensive. They aren't going to showcase only CPU performance. I'm sure you know that though. I deliberately chose a ridiculous comparison. It wasn't meant as a straw man but to make a point about expectations. You said in your post:
In reply to:<hr />
but it won't change the fact that for general uses, this iMac is not going to prove anything approaching 2x

<hr />
Word is general usage. If I didn't understand benchmarks, it would be easy to misconstrue your meaning of general uses. The point is that benchmarks can never be all inclusive. Apple obviously chose a benchmark that highlights practically the only difference in the new iMacs, the Intel CPU. They aren't being nefarious. They chose to highlight the one thing that differentiates the Intel Macs from the PPC Macs. This seems to me to be an obvious thing for marketing to do.

In reply to:<hr />
I don't think Apple's been remotely as careful as people say they have been.

<hr />
I do. They've put the details of their benchmark on the same webpage you quoted.

In reply to:<hr />
If our complaints about Apple's 2x claims are arming the opposition, I respectfully suggest that this could have all been avoided by Apple being a bit more restrained in the way they've marketed the speed of these systems.

<hr />
I'm not sure what you expect Apple to do here. A marketing campaign that says, New with Intel inside, up to 1.1 times faster? It is obvious that they are going to make the new iMacs look as good as possible while making sure they are honest about the benchmarks. In my mind, they've done that. Do you really think that using "up to 2x faster" is better somehow than "2x faster" with an explanation of the benchmark methodology? And, if they did that, then it should be "up to 3x faster" or even more since that would be provable in selected benchmarks as well. Apple has been restrained because they are using an industry standard benchmark. They didn't invent the benchmark, but they did select it because it met their marketing needs. That is perfectly normal and is in no way dishonest.
Reply
Go to original post 1 ... 6 7 8 9 10 Previous Next

Actions

More Like This