As I mentioned in an earlier Post, the Big Thinkers from the City of Boston decided it would be a great thing for building owners and operators to document and submit their utility consumption data, which would allow properties to be publicly  “scored” based on the Environmental Protection Agency’s Energy Star Portfolio Manager benchmarking tool.

Portfolio Manager is a simple tool that essentially calculates utility consumption per square foot and ranks buildings on that basis.  We have discussed previously why such benchmarks are fundamentally flawed and unreliable in principle.   But now a report has been released that demonstrates such benchmarks to be flawed in practice.

Which, logically thinking, only makes sense.  How can a broken idea lead to a correct result?  Only by dumb luck.

There is a description of this study in today’s Boston Globe:


If I find a link to the actual study I will add it as well.

I think this is it:


The money quote in the article is quite to the point:

“‘The benefits it [the reporting ordinance] achieves may be nil and the cost possibly significant,’ said [Robert] Stavins, who conducted the study in conjunction with Analysis Group, a Boston consulting firm.”

Hmm.  Well, that sounds helpful, no?

It is striking that the benefits are potentially “nil” according to this study,  given the huge investment in time and effort that has already been expended in collecting and entering data by various constituencies.  Surely someone, somewhere in the City of Boston carefully evaluated the Portfolio Manager benchmarking tool and employed thought experiments to examine a range of “what if” scenarios to prove the efficacy and reliability of benchmarks before rolling it out.


Now, no doubt, the study linked to above will be discredited by some because it was funded by a constituency that views the City’s reporting requirements to be burdensome and potentially deceptive when it comes to demonstrating their environmental stewardship.  “Ah, it’s motivated by self interest, ergo it can’t be valid.”

There are also stakeholders who have invested significant effort in establishing Portfolio Manager as the tool of choice for energy reporting and benchmarking for the City.  I have found it rare for people with that level of buy-in to reassess and change direction, even if bad results can be glimpsed on the horizon for those willing to look.

So it’ll be interesting to see if this report gets any traction.

If you’d like to see a description the ordinance itself, it is located here:


If you’d like to read my take down of simplistic benchmarks like Portfolio Manager, you can find it here:



Please bear in mind that most facility owners and operators embrace efficiency improvements and  want to be good environmental citizens.  However, it is frustrating in the extreme for them to be forced to use reporting tools that demonstrably will not deliver the kind of intelligence and the kind of results that the city claims will be achieved.  If only the policy makers has worked with the stakeholders beforehand, the current climate of discomfort and concern might have been largely avoided.  Water under the bridge, I suppose, but hopefully others will learn from our experience.

In general, though, we can take away a good lesson.  If you are going to do something, make the effort to do it carefully, accurately, and well.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s