What do J.D. Power's quality ratings really measure?

Wed, Jun 24 2015

Check these recently released J.D. Power Initial Quality Study (IQS) results. Do they raise any questions in your mind?

Premium sports-car maker Porsche sits in first place for the third straight year, so are Porsches really the best-built cars in the U.S. market? Korean brands Kia and Hyundai are second and fourth, so are Korean vehicles suddenly better than their US, European, and Japanese competitors? Are workaday Chevrolets (seventh place) better than premium Buicks (11th), and Buicks better than luxury Cadillacs (21st), even though all are assembled in General Motors plants with the same processes and many shared parts? Are Japanese Acuras (26th) worse than German Volkswagens (24th)?

And is "quality" really what it used to be (and what most perceive it to be), a measure of build excellence? Or has it evolved into much more a measure of likeability and ease of use? To properly analyze these widely watched results, we must first understand what IQS actually studies, and what the numerical scores really mean.

First, as its name indicates, it's all about "initial" quality, measured by problems reported by new-vehicle owners in their first 90 days of ownership. If something breaks or falls off four months in, it doesn't count here. Second, the scores are problems per 100 vehicles, or PP100. So Power's 2015 IQS industry average of 112 PP100 translates to just 1.12 reported problems per vehicle. Third, no attempt is made to differentiate BIG problems from minor ones. Thus a transmission or engine failure counts the same as a squeaky glove box door, tricky phone pairing, inconsistent voice recognition, or anything else that annoys the owner.

Traditionally, a high-quality vehicle is one that is well-bolted together. It doesn't leak, squeak, rattle, shed parts, show gaps between panels, or break down and leave you stranded. By this standard, there are very few poor-quality new vehicles in today's U.S. market.

But what "quality" should not mean, is subjective likeability: ease of operation of the radio, climate controls, or seat adjusters, phone pairing, music downloading, sizes of touch pads on an infotainment screen, quickness of system response, or accuracy of voice-recognition. These are ergonomic "human factors" issues, not "quality" problems.

Yet these kinds of pleasability issues are now dominating today's JDP "quality" ratings. "Cars and trucks have never been built better, but frustration with audio, infotainment, and navigation features on new vehicles has never been worse," Automotive News senior writer Jesse Snyder pointed out back in 2012. "For the first time, complaints about such features surpassed those about engines and transmissions as the top category... [And] half the problems reported by vehicle owners after 90 days were design related – things that are confusing or hard to use rather than faulty or broken."

As Power's 2015 IQS press release points out, "Entertainment and connectivity systems remain the most problem-prone area for a third consecutive year, with voice recognition and Bluetooth pairing continuing to top the problem list." But is that really quality? J.D. Power says that when it asks consumers what quality means to them, their definition is much broader than just absence of defects.

"They also include 'perceived' quality, quality of materials, and very much design quality Â– not only is the component built as designed, but was it designed right in the first place," a J.D. Power executive said. "The purpose of the IQS, they contend, is to measure quality problems as defined and reported by consumers. "So we take the broad approach that if a consumer considers something a problem, it is counted as a problem, and we count them all the same for a number of reasons. When we look at the degree to which these problems impact overall satisfaction of the customer, there is virtually no difference between design problems and defects."

Also important to understand is that almost all automakers have gotten so good that the rankings among brands and gaps between them are far less meaningful than they once were. For example, in the 2015 IQS, the gap between Chevrolet in seventh place and Toyota in 10th is just three PP100, or 0.03 problems per vehicle, which seems statistically insignificant.

Of the 15 brands above industry average, Porsche, Kia, and Jaguar placed first through third at 80, 86, and 93 PP100, followed by Hyundai (95), Infiniti (97), BMW (99), Chevrolet (101), Lincoln (103), then Lexus and Toyota (tied at 104). So are affordable Korean Hyundais meaningfully better than Japanese luxury Lexus vehicles because their owners reported 0.09 fewer problems per vehicle? Are low-priced domestic Chevrolets significantly better than luxury German Mercedes-Benzes, which placed 15th at 111 PP100?

Power also loves to titillate the media by grouping results by national origin. "Korean brands lead the industry in initial quality... averaging 90 PP100," said the 2015 press release. "For the first time in the study, European brands (113 PP100) surpass Japanese brands (114 PP100), while domestic makes (114 PP100) equal the Japanese for only a second time."

Sincere kudos to Korean corporate cousins Hyundai and Kia for improving dramatically in recent years. But how meaningful are such comparisons when low-volume Porsche sits at the very top of this year's survey (how many proud new Porsche owners are willing to report any problems with their expensive rolling status symbols?) while high-volume German VW places 24th and Italian Fiat dead last? When small-volume Japanese luxury marque Infiniti sits fourth and hot-selling Japanese Subaru places an embarrassing 30th out of 33 ranked brands?

JDP's Vehicle Dependability Surveys (VDS), which track customer-reported problems after three years of ownership, are much more meaningful, in this view. The 2015 VDS results to be released later this year will rank owner-reported problems over a three-year span. Our advice: check the surveys, then comparison-drive your top choices before deciding.

Related Video:

By Gary Witzenburg