Join our Partner Network Home Value Estimator

Pro Teck Insights

  • Subscribe Via Email

    • This field is for validation purposes and should be left unchanged.
National Housing Market Trends: Reality Distortion in the Most Widely Publicized Indices. A New Method for Better Results

Everyone wants to know what their home or investment is worth and where the market is heading. In some cases, the estimate of value and price trends influences the decision to invest in or walk away from a home.

For research on home price trends, the typical household has primarily the media reports from four widely publicized indices:

  • The S&P/Case-Shiller Home Price Index
  • The FHFA House Price Index
  • The FHFA House Price Index Including Refinancing Value Estimates
  • Reports from NAR, the National Association of REALTORs

In this month’s Lessons from the Data, we explore these indices and why we believe homeowners and investors can benefit from improvements in data and methodology.

The widely followed S&P/Case-Shiller (C-S) Repeat Sales Index makes headlines on a consistent basis, even though it is published with a 4-6 month lag due to its reliance on public records where sales can take months to appear.

In order to control for changes in quality or quantity of a changing basket of home sale transactions, C-S relies on repeat sales of the same property and tracks changes in the Index from a composite of all repeat sales. A possible weakness in this method is that the Index could exaggerate price declines for the non-distressed home seller because it includes both the normal non-distressed sales and bank REO foreclosure sales, which often represent huge discounts to actual market prices.

The REO sales skew the results downward when prices are falling and when prices increase will skew the results upward as the index will be starting from a lower than otherwise starting point. The C-S weighting system attempts to hold year-2000 initial sale weights constant so more expensive homes have more weight. Several filters attempt to screen out non-arm’s-length transactions and foreclosures, but foreclosures that are later sold as REO by banks are included as repeat sales. Other considerations include:

  • In markets with many foreclosures that become REO sales, there is an unusual and negatively biased impact on the price index if looking to estimate the impact on a typical owner versus a portfolio holder of the entire market
  • In markets with a number of new homes, the Index will be biased toward older homes that have sold twice, and so the Index is less representative of the typical home in that market

Another popular index is the quality-adjusted FHFA (Federal Housing Finance Administration) House Price Index, formerly known as the Office of Federal Housing Enterprise Oversight (OFHEO) House Price Index. This index is built using repeat sales transactions on single-family properties whose mortgages have been purchased or securitized by Fannie Mae or Freddie Mac since January 1975. One version of this index (FHFA Sales Price Index) relies upon the purchase prices for the properties. Limitations in the Index include it not covering the upper-priced markets, especially in California where many mortgages are above conforming loan limits and a significant time lag, making it less applicable for understanding current market trends.

With the FHFA Index, only conforming loans from the two GSEs are included, limiting the sample to those with mortgages of less than $417,000 as of 2012, (although we did see a temporary set of higher conforming loan limits in 2009 through part of 2010). This price limit makes the sample sufficient in areas with low density, inexpensive land and more affordable housing; but severely limits the applicability to higher-priced markets including most of California, metro markets such as Boston and New York, and coastal markets.

A variation on this Index is shown that includes both purchase transactions and refinancing transactions (FHFA Sale and Refi). Here the FHFA included appraisal based estimates of the homes which have not necessarily traded but where the owner wishes to refinance. This more stable part of the market is clearly representative of many less distressed markets, although again does not include the higher priced markets.

A fourth popular index is the median home prices as published by the National Association of REALTORs (NAR). This Index covers all markets and is not quality or size adjusted, so the mix in size and quality may affect the median price result. For example, if average sizes of sold homes are larger than in earlier periods then the Index will over-estimate the increase in home prices when prices are trending up.

Index Comparisons

Below in Figure 1 we compare four popular indices from mid-2006 to Q1 2012 for the entire US. The Case-Shiller Index suggests the greatest loss in value, with the NAR median next and the FHFA quality adjusted index as having the least amount of loss at around half the net loss suggested by C-S.   The FHFA Sales and Refinance series shown on top represents a large portion of the market which has not necessarily traded, but the appraisals used for refinancing are included in the price change estimate and here we observe even less of a price decline than even the quality adjusted estimates from the FHFA. The extent of differences in these estimates from an identical time period is significant.

Figure 1


As we drill down from a misleading national picture to the state level, we start to see a great deal of dispersion in terms of the average price change.  For example, in Figure 2 we use the changes in the FHFA House Price Index at the state level. What is remarkable is that several states have seen positive price movements since mid-2006 while others, like Nevada, are pulling the national index downward, like the tail wagging the dog.

Figure 2


We get a similar picture as we drill down even further using the same FHFA Index data, now at the CBSA level, shown in Figure 3.

Figure 3


The NAR results at the CBSA level look similar to that of the FHFA but shifted slightly to the left, as seen in Figure 4.

Figure 4


The NAR results include a wider range of price levels and also reflect the change in the mix of sales that has seen the upper priced markets as less active (with less subprime and lower loan to value mortgages on average) than the lower priced markets over the last five years.

Figure 5 below shows the same CBSA results for C-S. Note how there are no positive changes in prices.

Figure 5


The bottom tail of the distribution is similar to the FHFA but the upper end of the distribution is nowhere as positive. The overall picture is much more negative, making one wonder about how much the distressed REO sales are affecting the index?

This is not to say that REO sales do not represent value; they do in many markets where distress is dominant and regular sales are affected by the well documented contagion effect. But in many localized markets the distress sales do not dominate the market.

Using More Current Micro Market Trends Brings Reality Into Focus

One reason why we see variations in the NAR indices is because of the lack of control for size. When you control for size by simply reporting the price per square foot instead of the total price, the result is less volatility.

The Home Value Forecast Index uses median sold prices per square foot of living area to normalize for the effect of different sized homes which is, in almost all cases, the most important property characteristic in determining a home’s value.

Home Value Forecast, built using Collateral Analytics data, captures all sale data at time of transaction.  This insures that the indexes will reflect what is currently happening in the market of interest as opposed to reflecting what has happened in a subset of the market.

Another important benefit of using all the sales is that it enables the creation of more granular indexes such as for smaller ZIP codes and neighborhoods. In addition, it enables the creation of indexes for other specific property types (e.g. condominium, multi-family, etc.) or sale type (Regular, REO, Short Sale, etc.). Another methodology involves applying an automated valuation model to all homes each month and tracking the change in values so that a highly accurate micro index can be created for any submarket that has too few sale transactions to support a sale driven index.

Home Value Forecast data reveals that prices have moved less than reported for the typical home. It is just that the typical homes that were selling, or being forced to sell were slightly smaller in 2009 to present compared to the previous 5 years. This is shown in Figure 6 where we see that prices per square foot move much less than medians, where no controls for size are imposed.

Figure 6


As we move towards more granularity, the ZIP code level impact of distress will be felt in those markets with a significant proportion of distressed sales. In healthier markets where such sales are less common, we will get entirely different results.

The analysis from Home Value Forecast highlighted in Figure 7 shows that since mid-2006, looking at ZIP code/local level trends we have seen a distribution in median home price change from -70% to +25%, emphatically supporting our contention that all real estate is local and to understand trends you have to look at micro markets, not national averages.

Figure 7: Zip Code Level Aggregation


We dive into even more granularity below where we use neighborhood home price indexes tracked by Collateral Analytics. There are about 10 to 20 neighborhoods in the typical ZIP code, and each one is defined by a commonality of typical features. We see this is Figure 8 below.

Figure 8: Diving into Smaller Geographies


The distribution of price changes is similar, but we do see a fatter tail of markets with significantly increased prices since the trough and a somewhat thicker distribution showing more variation in price trends.

In Figure 9 we drill down all the way to the individual home level by using the Collateral Analytics CA Value AVM.  In particular, we ran an AVM for every single family home in the U.S. as of mid-2006 and calculated the percentage change to the current AVM value.

Figure 9: Drilling down to the Individual Home


Here again we see more variation in price movements, with even fatter tails or extremes in terms of homes that have moved up or down in price since 2006. What we see is that many home owners have done very well and many have been wiped out. This is the true reality behind the price trends. Anything above this level that averages individual homes together will miss the mark.

When we apply this individual home appraisal approach to all the homes in the U.S. in 2006 and again in 2012 we see that prices are actually a little higher in mid-2012, so many homeowners have weathered the storm without significant loss in value, but the housing headlines tend to focus more on the distressed impacts that continue to plague the more distressed markets.


All real estate trends are local, and what’s happening on a national average is of little consequence when trying to sell your $200,000 home in Phoenix or your $2,000,000 condo in New York City.

While interesting to look at, views the major indices as imperfect when used for comparison.  Specific limitations include:

  • The Case-Shiller Indices tend to overstate the price trends in both directions for the typical non-distressed homeowner, and may understate the decline for the distressed homeowner
  • The FHFA Indices miss the upper end of the market
  • The NAR Indices are very much influenced by a mix of size and quality that is not constant over time
  • All have significant lag time in reporting, unable to tell us where the market is today
  • None have the data or methodology to provide micro market or sub market indices, hence painting large geographic regions with a “one size fits all” median change, missing the significant differences in home appreciation that exists within these broader markets

Housing prices within ZIP codes don’t move in perfect correlation with the metro market; neither do housing prices in different neighborhoods within a single ZIP code necessarily move in lockstep. Local Realtors know this and divide MLS markets into neighborhoods that are subsets of ZIP codes or crossover ZIP codes.

As we drill down to more localized markets at the neighborhood level and individual home level we start to see the reality for homeowners and investors. If your market is still dominated by distressed sales you are negatively affected, but for many homeowners prices have started to move up. In fact, at the ZIP code level, about a quarter of all home price indices are positive since mid-2006. Maybe the glass is a quarter full for them, but you would never know that relying on national, state or even CBSA level price indices.

Today, with the benefits of technology, more robust and timely data and better geographic based analysis we can drill down to the homeowner’s micro market and need not rely on aggregation.

James R. Follain, Ph.D.

James R. Follain LLC and Advisor to FI Consulting

Norm Miller, PhD

Professor, Burnham-Moores Center for Real Estate

University of San Diego

Michael Sklarz, Ph.D.


Collateral Analytics

About was created from a strategic partnership between Pro Teck Valuation Services and Collateral Analytics. HVF provides insight into the current and future state of the U.S. housing market, and delivers 14 market snapshot graphs from the top 30 CBSAs. Each month, delivers a monthly briefing along with “Lessons from the Data,” an in-depth article based on trends unearthed in the data.

HVF is built using numerous data sources including public records, local market MLS and general economic data.  The top 750 CBSAs as well as data down to the ZIP code level for approximately 18,000 ZIPs are available with a corporate subscription to the service.