As I mentioned in an earlier Post, the Big Thinkers from the City of Boston decided it would be a great thing for building owners and operators to document and submit their utility consumption data, which would allow properties to be publicly  “scored” based on the Environmental Protection Agency’s Energy Star Portfolio Manager benchmarking tool.

Portfolio Manager is a simple tool that essentially calculates utility consumption per square foot and ranks buildings on that basis.  We have discussed previously why such benchmarks are fundamentally flawed and unreliable in principle.   But now a report has been released that demonstrates such benchmarks to be flawed in practice.

Which, logically thinking, only makes sense.  How can a broken idea lead to a correct result?  Only by dumb luck.

There is a description of this study in today’s Boston Globe:


If I find a link to the actual study I will add it as well.

I think this is it:


The money quote in the article is quite to the point:

“‘The benefits it [the reporting ordinance] achieves may be nil and the cost possibly significant,’ said [Robert] Stavins, who conducted the study in conjunction with Analysis Group, a Boston consulting firm.”

Hmm.  Well, that sounds helpful, no?

It is striking that the benefits are potentially “nil” according to this study,  given the huge investment in time and effort that has already been expended in collecting and entering data by various constituencies.  Surely someone, somewhere in the City of Boston carefully evaluated the Portfolio Manager benchmarking tool and employed thought experiments to examine a range of “what if” scenarios to prove the efficacy and reliability of benchmarks before rolling it out.


Now, no doubt, the study linked to above will be discredited by some because it was funded by a constituency that views the City’s reporting requirements to be burdensome and potentially deceptive when it comes to demonstrating their environmental stewardship.  “Ah, it’s motivated by self interest, ergo it can’t be valid.”

There are also stakeholders who have invested significant effort in establishing Portfolio Manager as the tool of choice for energy reporting and benchmarking for the City.  I have found it rare for people with that level of buy-in to reassess and change direction, even if bad results can be glimpsed on the horizon for those willing to look.

So it’ll be interesting to see if this report gets any traction.

If you’d like to see a description the ordinance itself, it is located here:


If you’d like to read my take down of simplistic benchmarks like Portfolio Manager, you can find it here:



Please bear in mind that most facility owners and operators embrace efficiency improvements and  want to be good environmental citizens.  However, it is frustrating in the extreme for them to be forced to use reporting tools that demonstrably will not deliver the kind of intelligence and the kind of results that the city claims will be achieved.  If only the policy makers has worked with the stakeholders beforehand, the current climate of discomfort and concern might have been largely avoided.  Water under the bridge, I suppose, but hopefully others will learn from our experience.

In general, though, we can take away a good lesson.  If you are going to do something, make the effort to do it carefully, accurately, and well.


Why I am Not a Fan of Most Energy Benchmarking

Benchmarking in the energy industry is generally capital-H Horrible, with all sorts of unexamined assumptions and presumptions that can readily  be shown to lead to incorrect conclusions.  I have railed against tools like Energy Star Portfolio Manager, but to no avail.  It is a bill of goods that has been sold to upper management and many consultants – who always likes things to be short, sweet and simple.  In this case, way too simple.

As a management tool, most benchmarks are awful.

My reasons for benchmarking skepticism are located here:


This is not to say that good benchmarking metrics can’t be created.  But we need to take the process away from the well intentioned policy types and have a discussion about what rigorous and accurate metrics should look like.

So what’s wrong with Portfolio Manager (PM)?

As I mentioned in the prior post, the City of Boston is proposing that Boston facilities begin to report their energy use and carbon emissions via a tool created by the Feds called the Energy Star Portfolio Manager (PM).

PM is a web-based tool into which one enters utility consumption and cost data.  PM then performs a couple of calculations:

  1. By looking at your location, the program determines the likely carbon emissions associated with your energy use.  For example, there is far less coal-generated power on the East Coast than there is in the Midwest, so the CO2 emissions per kilowatt-hour are corresponding lower thanks to our nuclear, gas-fired and hydro sources.
  2. By dividing your energy use by your square footage, the program determines an energy density.  By comparing the energy density of similar building types (e.g. office building) one can rank them.  And if one ranks high enough, one earns the “coveted” Energy Star rating.

First, some disclosure.  The full PM program may do waaaay more than mentioned above.  I am focusing on what I understand the City to be looking to do with it in this post.

Now, this is the starting point for all sorts of commentary, so bear with me as I jump about explaining the awfulness of this tool.

You will note, first of all, that PM is performing calculations that are so simple that one sort of questions why it’s even put up on the web.  Once I know the CO2 emissions per kilowatt-hour (kWh)  and per thousand pounds of steam (MLb) in my area, why can’t I just multiply my use by said factors and divide by my square footage?

Similarly, the energy content per kWh and per MLb is well known, so PM is not needed for this, either.

Now, that might sound petulant.  If PM does the job, even if it’s simple, then what’s the problem, right? But the fact is that PM is very cumbersome and time consuming to work with, whereas doing the same calculations in a local spreadsheet (or even better, a database!) is quicker and allows for much more flexible reporting.  So there’s that.  There’s also the fact that, while you can force me to put information into PM, there is no way you could force me to use it to monitor energy performance.  So it’s simply wasted time and effort in a world that increasingly values efficiency and performance.

My second beef with PM is its lack of detail.  PM does not care about the peak demand of a building.  Nor does it care whether you use energy during “on peak” times or “off peak” times.  Why does this matter?  Well, as usual, a thought experiment can illuminate things quite readily.

Imagine two identically sized manufacturing facilities that consume the same amount of energy each year.  Well, according to what I describe above, they will grade out with identical Portfolio Manager scores.  However, it is quite possible that one facility has implemented energy savings strategies that have dropped the peak building demand, and that they are operating for more hours (and manufacturing more product) without using more total energy.

So wait a minute.

One manufacturer is creating more product while using the same amount of energy, but it’s Energy Star rating is identical to its less efficient neighbor?  How can that be?

Simple.  The program is too unsophisticated to detect operational efficiency.

Let me say that again:

The program is too unsophisticated to detect operational efficiency.

Yes, ladies and gentlemen.  The Ordinance proposed by the City of Boston will use a measuring tool that literally cannot discern the difference between an efficiently run facility and an inefficiently run facility.

This is the stuff that drives you crazy…

I should really play with PM a bit to truly stress test it.  If/when I do so, I will report back.

I may come back and add to my awfulness list.  Let’s see how time permits.

What the hell, let’s start with a rant…

It all begins here:



So, the City of Boston has decided to announce a proposed “Building Energy Reporting and Disclosure Ordinance”.  Well.  This certainly sounds like a great idea.  Here’s a bit of the language:

“As a component of the City’s climate action plan to meet Mayor Menino’s greenhouse gas reduction goals, this ordinance would require all large and medium sized-buildings to report their annual energy and water use to the City of Boston. The proposed ordinance is intended to encourage building owners to participate in local utility energy efficiency programs and educate tenants on building performance.”

 “In order for Boston to continue to be a sustainability leader, our buildings must aggressively invest in energy efficiency,” Mayor Menino said.  “Bostonians demand buildings with high performance and this ordinance will encourage building owners to meet that demand.”

 “Major cities across the country have already adopted similar ordinances including New York City, Washington, D.C., San Francisco, Seattle and Minneapolis. Lessons learned from these cities have informed the Ordinance proposed by Mayor Menino which would require all large and medium sized buildings to report annual energy use, water use, and greenhouse gas emissions tracked through Energy Star Portfolio Manager to the City of Boston Environment Department. The City would then make energy and water use per square foot, Energy Star ratings, greenhouse gas emissions, and other identifying and contextual information for individual buildings available online.”

I mean, this sounds good, right?  What’s not to like?

Well, where do I begin?

First of all, as someone who has aggressively pursued energy conservation (and concomitant CO2 emission reductions) for well over a decade, I sort of resent the implicit message that the geniuses in City Hall (and probably the Kennedy School and other temples of wisdom not normally haunted by working-stiff engineers who actually make things work in the real world) suddenly conjured up this great idea that has seemingly eluded building managers and engineers for the past twenty years.

Secondly, the tool being used – the Energy Star Portfolio Manager (PM) – is an almost embarrassingly unsophisticated tool for evaluating energy performance in non-trivial facilities.  While we can discuss what this means later, the fact that this tool does not even bother tracking building energy demand, on-peak versus off-peak energy use, load factor or power factor speaks to the lack of thought that went into the selection of this platform.  Believe me, I have opinions about PM.

Thirdly, this ordinance is going to actively penalize those facilities, such as mine, that have saved millions and millions of kilowatt-hours and pounds of steam and ton-hours of cooling over the past decade.  I get no credit for all that work, while the schlub who’s done nothing can start doing easy stuff and look like an all star?  Really?  Do they have any idea how much more expensive it is to save energy after you’ve done dozens and dozens of projects already?  In a word, “No.”

Lastly (at least for now) this ordinance will require that we spend already scarce resources redundantly entering data into a tool of virtually no practical use in managing energy consumption, while our own sophisticated tools (in my case, a fully programmed, customized database) is shunted aside.

Let’s face it, at the core of this rant is resentment at the fact that a bunch of whiz-kids decided they came up with this great new idea, they were too important to talk with the poor stiffs who actually have to work with the building equipment and systems and know what it takes to actually implement and track energy savings or see what’s already been accomplished, and they clearly – based on the tracking tool they chose – completely don’t appreciate the complexity of the task.  Gee guys, really?  We should save energy?

All that said, the goals of making Boston more energy efficient and of reducing our carbon dioxide emissions are admirable.  But it really hurts to see such a worthwhile goal mapped out with a methodology that is so unsophisticated, and that could have been so much more promising had the city deigned to speak with the stakeholders before shoving this down 0ur throats

Of course, I’m also old enough to realize that that’s simply how the world works sometimes…