One possible explanation is that the market has kept learning about the strength of demand and the weakness of supply over the years. It is consistently being surprised, in other words. That may be right, but it is a shaky argument: why is the market always being surprised in the same direction – that excess demand is greater than we thought?The story about the market consistently being surprised is more believable if you think about it not in terms of the market being surprised over and over again in one direction, but rather the market being surprised in one direction more often than it's being surprised in the other direction. Look at the price of crude oil over the past 6 months:
Another story that I think makes some sense is the one that Jeffrey Frankel and Jim Hamilton have promoted – that Fed monetary policy has played a role. [A] drop in real interest rates should cause commodity prices to rise. But here again, the decline would also have to be unanticipated to explain the continual increase in the price.
I think there is a lot of truth to the view that markets keep getting surprised in the direction that makes oil prices higher.
...
The problem for economists is that the market for oil is so complicated that we cannot very accurately calculate what the price of oil “should be” if there is no bubble. We have to read the entrails to figure out whether the price is really reflecting market fundamentals – demand, supply, real interest rates – or has a bubble component. As I look at the rising price, I wonder which story is most plausible: (1) the markets have been surprised over and over about demand by end users and production capabilities; (2) markets have been surprised over and over about how low real interest rates are; (3) there is a bubble. These stories may go together, in fact.
Clearly, the market has been "surprised" in both directions over the past 6 months -- it's just been surprised in the direction of higher prices more than it's been surprised in the direction of lower prices. I'd be willing to buy this story as the explanation for increasing oil prices (rather than a speculative bubble). But just as intraday movement in stock prices reflect liquidity concerns rather than information being priced in, I'm not sure that most of the movements in oil prices don't reflect major liquidity issues -- especially as the credit crunch has made liquidity such a dominant issue.
As far as I'm concerned, there's probably a bubble in agricultural commodities, but the jury is still out on whether there's an oil bubble. Oil is a unique beast, and cannot be lumped together with all of the other commodities. The debate over whether there's an oil bubble is fascinating, and is taking place at a considerably higher level than the debate over whether there was a housing bubble. (I'm just waiting until the hysteria and irrationality kicks into the oil bubble debate, because you know it will eventually.)