, , , , ,

According to Moneysense, and our Mayor, the best city in Canada to buy a home is Guelph.  While I agree that Guelph is a nice place to live, I think that it is silly to make that claim.

More importantly, should I believe anybody who says that they know? Moneysense’s ranking uses public information and massages it in some way.  Lots of people do this kind of exercise for many reasons (best country to live in (Canada is #2 in 2017, best country for business (Canada is #10 in 2016), …).  It would be better to know if information is relevant.  In other words, understanding the methodology is critical.  Professional statisticians and high quality data sources talk about methodology a lot.  If a methodology is not given, as was true of this exercise, attempting to reverse engineer it can raise important questions.

Based on the information provided by Moneysense, it appears that they ranked cities using four bits of information: average price, ratio of average income to home price, 5 year RoI (return on investment), and 5 year rent increase.  When multiple measures are considered, the methodology needs to explain how they are combined.

(Methodology would also show if the difference between #1 and #2 is meaningful, but that information was not provided.  It is not clear whether their RoI measure controls for quality [1] [2] [3] or, more likely and more troubling, uses the prices of whatever happens to be sold during a year.)

The following bit of reverse engineering uses regression analysis of the data reported for the top 35 cities to estimate the weights used in the ranking.

Rank= 36.57- 269.02* [5yr RoI]- 45.58* [5 yr Rent Incr.]

This equation is able to explain (i.e. for technically-minded people, “R2=”) 71.1 percent of the variation in rank.  Adding the two other variables raises the goodness of fit by less than 1.5 percentage points.  Using 5yr RoI only explains 66 percent of the variation in rank which, given how little information is provided, is not bad.

(The value of the intercept contains no predictive information.  The size of the coefficients is also deceptive large since, for example, the average RoI is less than 4 percent (i.e. 0.039).)

Using past prices to explain the best city to buy houses seems sensible if you want to tell a story of momentum.  But, if you want to tell a more accurate story then pay attention to the mystery of random shocks; they are the reason why past performance is more likely to overestimate when prediction is unusually high (a phenomenon known as “regression to the mean”).  Investors who have been warned about chasing trends in a stock market should know this already.  Especially for a ranking exercise, the extreme outcomes (i.e. being first or last) are mostly luck and are unlikely to be repeated.

To illustrate, consider some data from the Toronto Real Estate Board and the question: how much does knowing the price increases from the last five years help to predict price increase during the next year?  Fitting a regression line between the recent RoI and prices increases in the following year produces a goodness of fit of 67.9 percent, using data from 1989 to 2016: R2= 67.9%.  But, the data from early 1990s appears different and, excluding them reduces goodness of fit to 28.5%.  So, momentum does not make for a reliable story.

How would I analyse the question?  If you want to claim that a city is nicer place to live then you should ask: how many people moved into a city vs. how many people moved out?  While Statistics Canada releases this information eventually, a better source is U-Haul.  According to them, Guelph was #3 in terms of net in-migration last year.  Kamloops was #1 and is nowhere on Moneysense’s top 35.

Guelph is a nice city to live in with a good university full of exciting and new ideas, but this exercise measures almost nothing that people care about.  It is possible to measure lots of things but measurement does not make it meaningful.  My advice is to pay attention to the data and expect to see the methodology.  Only then will you know enough to trust what is said in a headline.