Parsimony: How many parameters per model?
Statistical Modeling, Causal Inference, and Social Science: Are we not Bayesians?
There is an interesting thread of discussion on Parsimony in modeling. My own model of reading eye movements has been accused to having too many parameters.
AG (12.10.04): A lot has been written in statistics about "parsimony"–that is, the desire to explain phenomena using fewer parameters–but I’ve never seen any good general justification for parsimony. (I don’t count "Occam’s Razor," or "Ockham’s Razor," or whatever, as a justification. You gotta do better than digging up a 700-year-old quote.)
Posted by: deb at April 15, 2005 01:19 PM
In response:
Deb - There’s a great quote by Peter Grunwald in his introductory chapter to "Advances in Minimum Description Length" (2005, p.16; MIT Press) that talks about parsimony.
It is often claimed that Occam’s razor is false — we often try to model real-world situations that are arbitrarily complex, so why should we favor simple models? In the words of Webb [1996], "What good are simple models of a complex world?" The short answer is: even if the true data-generating machinery is very complex, it may be a good strategy to prefer simple models for small sample sizes. Thus, MDL (and the corresponding form of Occam’s razor) is a strategy for inferring models from data ("choose simple models at small sample sizes"), not a statement about how the world works ("simple models are more likely to be true") — indeed, a strategy cannot be true or false; it is "clever" or "stupid." And the strategy of preferring simpler models is clever even if the data-generating process is highly complex
Posted by: Dan Navarro at April 17, 2005 01:28 AM
Finally, hereis a case where the simplest model fits., although the author acknowledges that this rarely happens.
[Update: 2 more links from the same blog, one old, one new. I like the following quote:
My favorite quote on this comes from Radford Neal’s book, Bayesian Learning for Neural Networks, pp. 103-104:
Sometimes a simple model will outperform a more complex model . . . Nevertheless, I believe that deliberately limiting the complexity of the model is not fruitful when the problem is evidently complex. Instead, if a simple model is found that outperforms some particular complex model, the appropriate response is to define a different complex model that captures whatever aspect of the problem led to the simple model performing well.
Exactly!
P.S. regarding the title of this entry: there’s an interesting paper by Albert Hirschman with this title.
Posted by Andrew at December 10, 2004 08:04 PM
]