What drives population growth - food or energy?

A common assumption among population analysts is that food availability is the main driver of population growth. In fact, most will go so far as to define the carrying capacity of an environment primarily in terms of the food that it offers to the population under consideration. I have two major problems with this approach to population and carrying capacity, as outlined below.

My first objection is that this approach treats carrying capacity as a variable, and the expansion of agriculture as an increase in carrying capacity. This requires a definition of carrying capacity I do not subscribe to. The definition I am most comfortable with is, "The population level that an environment can support over the long term without damaging the ecology of the environment". An expansion of agriculture does not meet this definition because putting new land under the plow or increasing the production of existing farmland affects habitat, biodiversity, water levels and soil fertility among many environmental factors. In effect the expansion of agriculture requires that we draw on the natural capital of the environment. The repayment of this withdrawn capital does not enter the ecological equation as it should. The result is, by definition, not sustainable. In fact, the form of organized agriculture (which I have heard playfully called "totalitarian agriculture") practiced for the last ten thousand years is by definition unsustainable, especially when you consider that virtually all of the arable land on the planet is now under cultivation. Now, my definition of carrying capacity may be too strict and may be disputed by other ecologists, but it's the one that seems most comprehensive and reasonable to me.

My second problem is that energy is never mentioned in mainstream analyses that focus on food. The possibility that this omission may be wrong-headed is hinted at by the well-known studies that found 7 to 10 calories of fossil fuel embedded in every calorie of food we eat. In fact, I have developed a strong suspicion that rising per capita energy consumption has even more to do with population increase than rising food production. To investigate this possibility I created the graph below. It shows population, per capita grain consumption and per capita primary energy consumption from 1965 to 2005, all scaled to allow a visual estimate of correlation.