After a pretty good 30-year run, deregulation is on the political ropes. Although loosening the shackles on banking, trucking and airlines delivered lower prices, robust competition and political applause, it hasn’t worked for electricity. California and Virginia have already abandoned the project and other states are contemplating a similar retreat. For the first time in decades, Americans are inclined to think that regulation is the thin blue line between defenseless consumers and predatory capitalists.

So did free market reformers take deregulation too far? Yes and no.

Yes, because they promised rate reductions they had no business promising. No, because deregulation of some parts of the system was offset by more ambitious regulations elsewhere. The end result is even more economically artificial than the one we started with.

Many of the states that undertook utility restructuring in the late 1990s rolled back retail electricity prices and then froze them in place for years during the “transition” to retail competition. The headline-grabbing rate increases this year in Maryland (50% in Baltimore) and Illinois (24% in Chicago) occurred because the period of regulated prices ended — while during the freeze period the prices of the fuels used for generation (coal and natural gas) increased significantly.

The states sticking to the old regulatory regime had no rate freezes, instead passing on higher fuel costs to consumers through gradual price increases. The average increase in rates in the regulated states from 1990 through 2006 (one cent per kWh) is not statistically different from the increase in deregulated states (1.6 cents per kWh).

Still, electricity rates increased under deregulation, while rates decreased in deregulated industries like airlines, banking and trucking. Why?

Under the old regulatory regime, electricity generators received their costs plus an allowed return on capital. If generators’ costs differed, they received differing revenues. Prices were then established by a “weighted average” of all producer costs. Under deregulation, however, generators receive revenues based on the price charged by the most expensive generator whose output is necessary to meet demand in each hour.

While some may find such pricing to be odd, it is found in all commodity markets. Potatoes, for example, sell at the same price even though the cost of production varies across farmers. The supermarket does not price potatoes based on the “weighted average” of their acquisition costs, and producers do not sell at cost plus a modest mark-up. They sell at what the market will bear, and the market will bear the highest cost source of potatoes necessary to meet consumer demand.

Thus, in a regulatory regime, rising natural gas prices affect electricity prices only according to the percentage of electricity generated by natural gas (about 18.7% of supply nationwide in 2005). But in deregulated markets, all generators get revenues based on the price charged by the most expensive (often natural gas) plant in operation.

Does this mean that consumers are always worse off under market (marginal-cost) prices rather than regulated (weighted average) prices? Well, regulation certainly delivers lower prices than the market during shortages. But regulation delivers higher prices during times of relative abundance.

Few remember that the impetus for deregulation was the discrepancy between higher-priced “regulated” power (predominantly coal and nuclear) and lower-priced spot market power (predominantly from gas-fired power plants) when natural gas costs were low in the 1990s. The owners of coal and nuclear generation resisted market pricing because they believed (correctly at that time) that in a market-price regime they would not recover the capital costs of their much more capital-intensive generation.

Thus rate freezes in states “deregulating” electricity markets were not designed to protect consumers. They were meant to protect high-cost producers, and kept retail prices from falling, if the new markets were driven by marginal-cost pricing.

Still, markets appear to be worse than regulation because generators whose costs are lower than the most expensive players in the market will get “excess” revenues. But excess profits aren’t forever. Once returns are predictably higher than normal, entry will occur to dissipate them.

Consider Texas. Unlike other deregulated states, utilities in Texas were allowed to pass fuel-cost increases on to consumers on a yearly basis during the transition to full deregulation. There was no provision, however, for passing through fuel-cost decreases. Post-Katrina, natural-gas prices pushed the cost of electricity to between 15 cents and 19 cents per kWh. But electricity prices did not adjust down when natural gas prices fell.

Those high prices, which result in large profits for coal-fired plants, induced TXU, the largest generator in Texas, to announce plans to build 11 more coal plants. Ironically, the much-praised plan by Kohlberg Kravis Roberts & Co. to take over TXU and build only three coal-fired power plants will keep power costs higher than otherwise. Environmentalists and plant owners win while ratepayers lose.

A final worry is that deregulation means sky-high prices during peak demand periods, typically hot summer afternoons. True. But they would be more than offset by lower off-peak prices. That’s because in a regulatory regime ratepayers must still pay off, through higher rates, the capital costs of power plants sitting idle during off-peak demand periods. And there’s a lot of idle generating capacity. MIT economist Paul Joskow, for example, reports that in New England during 2001, 45% of the generating capacity produced only 7% of the total electricity.

In sum, allowing markets to dictate electricity prices is a good thing for consumers, even if they are sometimes higher than under regulation. Unfortunately — and here is the fly in the ointment — price deregulation has been accompanied by rules encouraging the legal separation of generation from transmission and the purchase of wholesale power through organized spot markets.

This approach is based on the belief that, while the generating sector is potentially quite competitive, the electricity transmission business is not. Thus, the argument goes, deregulation, in order to work properly, must sever the vertical integration of electricity generation, transmission, and distribution under a single corporate umbrella.

While this seems reasonable, there are good reasons why vertical integration makes sense in the electric power business. Unfortunately, none of those reasons have been given much of a hearing.

First, vertical integration is an efficient response to the so-called “holdup” problem. Investors in generating plants worry that, because the assets are costly, dedicated and immobile, they can be “held up” by transmission line owners. Investors in transmission lines fear being held up by generators. Vertical integration ends the fight.

Second, transmission and generation are substitutes for one another — and the right amount of investment in either is an economic, not an engineering, puzzle. Efficient investment in both may not be possible through decentralized arrangements (prices and contracts) between separately owned assets. In contrast, an organization that owns both generation and transmission assets is more likely to invest optimally in both.

Third and finally, vertical integration minimizes risk in the real-time operation of the system. The better coordinated are generation and transmission, the less chance there is of cascading blackouts and other problems. Coordination is far easier when there is one actor rather than hundreds.

These considerations largely explain why 10 of the 11 published studies on this issue conclude that vertical integration is the most efficient corporate organizational form for electricity providers. Unfortunately, the debate about utility restructuring has almost completely ignored those studies — assuming rather that vertical integration serves no useful purpose other than facilitating the market power of incumbent electricity providers.

Interestingly enough, the deregulators are trying to create a world that would probably never arise in a totally free electricity market. In a world of deregulated vertically integrated firms, both producers and consumers would almost certainly resist spot market relationships. During gluts, firms would not recover the cost of capital; and during shortages, electricity consumers would be vulnerable to economic extortion, as competitive entry and rivalry can’t happen overnight. Both firms and consumers would likely prefer long-term contracts, an arrangement that meets consumers’ interest in price protection and firms’ interest in cost recovery.

Accordingly, the equilibrium relationship between firms and consumers in a totally unregulated world might resemble that of the old regulatory regime, albeit an equilibrium achieved through contract. The only (unanswerable) question is how different the specifics of such hypothetical contracts would be from current regulatory practices.

True deregulation involves allowing market actors to run their businesses in whatever manner they like, price what the market will bear, and discover for themselves how best to deliver goods and services without government influencing those decisions with carrots and sticks. The faux deregulation we have in the electricity market unfortunately falls short on most of those counts. And that — rather than the rate increases — is the real problem.

Peter Van Doren is editor of Regulation magazine, published by the Cato Institute, where Jerry Taylor is a senior fellow and director of natural resource studies.