How to read tech analysts' shipment reports and forecasts

We're entering what I like to think of as the silly season, the time when the major tech analysis companies issue their quarterly reports on mobile device sales. IDC already released its report on Q4, while Canalys is due any day. These reports always generate press coverage; some of it insightful, most of it just repeating whatever the analysis companies said.

Once all the reports are out, I'm going to look through them and try to give some comments on what I think they mean. But in the meantime, I thought it would be good to post a note on the numbers themselves – how they're gathered, what they mean, and what to watch for.

First, let's differentiate between shipment reports and forecasts. The shipment reports typically discuss what happened in the previous calendar quarter, and are issued about a month after the quarter ended. Forecasts are generally issued about once a year, and predict what sales will be in the next five years or so. We'll do forecasts first.


"You don't actually use these things to make business decisions, do you?"
--A horrified industry analyst, when she realized why we had requested the latest forecast

I think tech industry forecasts are worthless. Completely worthless. That may sound harsh, but think about it for a minute – can you reliably predict the future? Can anyone you know do it? If anyone could predict the future, don't you think they'd get rich off the stock market rather than working at an analysis company? I once asked a very senior manager at one of the biggest tech analysis firms how they created their forecasts. He laughed. "The process involved pizza, beer, and a dart board."

Here's an example of how bad tech forecasts can be: In the handheld market, the big analysis firms once predicted that handheld sales would be 60 million units a year by now. Instead they're 20 million. That's a margin of error of about 200%. If NASA forecasted that badly, Neil Armstrong would have landed in Arkansas.

Since the forecasts are useless, why do the analysis companies create them? Two words: publicity, and money. If you create a forecast and issue a press release about it, a lot of reporters will write articles on it, all of them crediting your company and publicizing its name. The tech websites will repost your press release, and even more people will read about your work.

The other reason to do a forecast is because companies will buy it. If you work in a company and you're tasked with creating a business plan, your management will insist on having a forecast as a part of it. No one believes an internally-created forecast, because the employees are assumed to be biased. So you buy an external forecast. If management is trying to be especially conscientious, you'll be asked to buy several forecasts for due diligence.

I've seen companies assign teams of senior people to spend months massaging and cleaning and polishing the industry forecasts, so they can be prepared for use in the business plan.

It's all a waste if time. When you start with a cow pie, no matter how much you polish it, all you'll get is a polished cow pie. Companies would be much better served by getting together their brightest people, asking them to make a guess, and then writing that down. Chances are the folks in your company are more in touch with the market than the analysts (you talk to a lot more of your customers than they do).


Now let's talk about the quarterly shipment numbers

To be fair, I should acknowledge that the quarterly numbers are a lot more accurate than the forecasts. But there are still major problems. No one, absolutely no one, knows what's really happening in mobile device sales. The market's too complex, and some critical numbers simply aren't available. For example, Dell won't tell anyone what its precise unit sales are by product line. It's also notoriously difficult to get phone shipment numbers out of the operators. Sometimes that seems to be because they view the information as confidential, and sometimes it seems to be because the operators themselves don't have very good inventory tracking programs for their retail stores (they care a lot more about how many service plans they sell than which phones).

The most accurate sales numbers generally come from two companies, NPD and GfK. Both of them directly track sales of electronic devices through retailers (NPD in the US, GfK in Europe and several other countries). So, for example, with the NPD numbers you can find out exactly how many handhelds were sold last week at retail in the US. You'll get a database that includes brand, model, average selling price, and a lot of other information. But those numbers won't include the things NPD can't track -- Dell, direct sales by any manufacturer to a business, and smartphones and other devices sold through the operators.

You'll also never see the NPD and GfK numbers in public. They're pretty close to monopolies in sales tracking, and they charge enormous sums of money for their data. Nothing is given away free.

That means most of us are left looking at the quarterly shipment numbers compiled by companies like Gartner, IDC, and Canalys. At all three companies, the methodology is basically the same – they call every hardware vendor, ask them how many units they shipped in the quarter, and total up the numbers. The people who make the calls and compile the numbers are generally honest and hard-working, and they're doing the best they can with the (limited) resources available to them. But there are several things you need to be aware of when reading their numbers:

--The vendors can lie. It's hard for a relatively small company like Palm to lie about its unit shipments; you can just take Palm's quarterly revenue and divide it by whatever you think the average selling price is for its devices. But for a large company like Dell or HP, mobile devices are such a small percentage of their overall sales that there's no independent way to check their numbers. I am not saying that Dell or HP fabricate the numbers they report to the industry analysts; they're pretty conservative about their reputations, and US stock market regulators frown on a company lying about anything that might move its stock price. But the rules are looser in some other parts of the world, and I've heard persistent rumors of other companies cooking their shipment numbers to make themselves look good in the quarterly reports.

If you think about it, people working in tech companies have a very strong financial incentive to mislead the share tracking companies. It's fairly common for sales and marketing managers to have performance bonuses tied to market share. Guess how share is measured. It's like asking an eight-year-old to fill out his own report card.

--The numbers measure shipments into stores, not sales to customers. When a big company like Nokia ships a new smart phone, it rushes hundreds of thousands of units into stores and distribution warehouses. Those first shipments typically happen in a single quarter, and can cause a company's share to rise dramatically in the quarter of first shipment, and then plummet the next. The analysis companies and press rarely explain this, which is why you'll get dramatic press reports that a particular handheld or smartphone company jumped from nowhere to become a top five vendor in a single quarter. Ask yourself if that's really possible – do people really change their buying preferences that quickly? And does it happen every quarter or two?

--The third problem with the quarterly numbers is that they don't report shipments by model (because companies refuse to give out their model-specific shipments). So all of Nokia's smartphone sales are reported as a single lump, or at best are cut into a couple of not-very-helpful categories such as flip phone vs. candy bar phone. This makes it incredibly difficult to figure out what's happening in different market segments.

--The fourth problem is that different analysis companies cut the mobile market differently. To Gartner, a RIM Blackberry is a PDA but a Palm Treo is a smartphone. To IDC and Canalys, both are smartphones. This makes for huge disagreements about market size. Gartner says the PDA market is growing at a healthy clip; IDC says it's dropping steadily. (By the way, if you want to hear some catty comments, ask Gartner or IDC or Canalys to tell you what they think of the way the other company classifies RIM shipments.)

Given all of these challeges, why do the analysis companies bother to compile the quarterly numbers? Once again, publicity plays a role. The quarterly score-keeping press releases get lots of coverage. (I just did a Google search for "+Gartner +PDA +share". It produced 243,000 hits.)

But also, I have to admit it – as bad as the quarterly shipment numbers are, they're better than nothing. If you know their flaws, you can try to correct for them, and sometimes you'll be able to dig out a few insights. That's what I'm hoping to do after the new round of shipment press releases comes out.