Advanced TinyMCE Editor

TEST!!! Alex S. Brown, PMP IPMA-C

 

Download Article

Printer-friendly
(PDF)

Download Slides, light

“benchscore-light-ppt.pdf”>Printer-friendly (PDF) “benchscore-light.ppt”>MS PowerPoint

Download Slides, dark

Printer-friendly
(PDF)
MS PowerPoint

Download Sample Scorecard

MS Excel
(XLS)

Advanced TinyMCE Editor is a little better. At least I get to keep my DIV tags. Unfortunately it mangles any tag that is broken across a line! It also still has the extra BR problems.

Two versions of this paper are available. The “benchscore-short.html”>short version was prepared for a newsletter and
is approximately 2,000 words, or six pages. The long version (below) was
prepared for a 45-minute presentation, contains some additional details, and
runs approximately 3,800 words, or ten pages. Feel free to read either or
both versions.

One of the best pieces of advice for any project manager is, “Divide and
conquer.” Take a complex, large deliverable, and break it into parts. Deliver
it in small pieces, each of which can be managed separately, with as few
inter-dependencies as possible. This strategy is an absolute requirement at
times. Some efforts are so large that a single project manager cannot
reasonably track and manage the entire effort alone. Some efforts are so
complex that they need multiple managers, each with different specialized
knowledge. Thank goodness for “divide and conquer.”

There is a dark side to “divide and conquer”, however. At some point
senior management will ask, “But how are we doing overall?” The individual
status reports from each sub-project may not show the progress towards the
overall goal. Cross-team problems may be ignored. Multiple formats from
multiple authors can be confusing to read.

People want to see the forest, not just the trees. Creating an overall
status report is a possible solution.

One approach to an overall status report is simple:

  • collect text descriptions of status and goals from the manager of each
    sub-project,
  • submit them all to a single person
  • concatenate them into a single status report, editing the result for
    consistent format and language

This overall report can be extraordinarily helpful, to senior managers and
sub-project managers alike. Sometimes it is enough, but a project manager
should have additional tools in his or her toolbox.

A second approach is to create an overall schedule for the effort. These
schedules can be as simple as a milestone-only schedule, with key sub-project
deliverables tracked on a single plan. Dependencies and cross-team milestones
can be represented. More complex versions will have a single master schedule
that contains all tasks for all sub-projects, including actual and estimated
work hours. Scheduling software can help automate creating these master
schedules. Creating schedules is often a tool-specific challenge, and in my
experience there are easier ways to get clearer information to senior
management.

<br “Two sub-projects contribute paper reports and one project contributes a phone-in report. All feed into the overall project report.” />
Each Sub-Project Contributes to the Overall Scorecard

Any project can use benchmarks and scorecards, but they are especially
useful for projects composed of sub-projects. Each sub-project manager can
create a custom scorecard containing critical, relevant data for their
effort. Often a useful scorecard can be just a single printed page. Team
members can use the scorecards to track progress, gauging concrete results
against benchmark numbers. The overall project manager can then collect these
scorecards, combine the relevant data from each, and produce an overall
scorecard. Usually it does not capture all the metrics of all the
sub-projects, but it provides an accurate view of the forest of sub-projects.
Benchmarks can be combined to track overall project-level progress. The
approach is satisfying because each manager has local control over his or her
deliverable, and can set benchmark goals independently. Overall management
can still track progress towards the ultimate goal with confidence and
precision.

Based upon my experience managing software projects in financial services,
this work is never simple or easy. Perhaps in a very mature organization with
concrete, well-understood standards it would be. Most of us do not have the
privilege of working in such an organization. Using a step-by-step process,
though, anyone can establish some consistent standards for a multiple-project
effort. If these standards survive the effort, the overall manager can take
pride; he or she has increased the maturity of the organization while
completing the project.

A note on terminology: Some organizations talk about
“programs” composed of multiple “projects”. Other organizations use “program”
to mean a whole product line, while still others talk about “projects”
composed of multiple “programs”. This paper will avoid the term “program”
entirely. Instead “sub-projects” are parts of a “multiple-project effort” or
an “overall project”. “Project manager” means any project manager, while
“overall project manager” is someone running the overall project.
“Sub-project manager” is someone running a sub-project.

Step 1: Determine Senior Management Needs

The target audience for these benchmarks and scorecards is almost always
senior management. Most stakeholders are concerned with either day-to-day
progress at a low level, or overall completion dates and cost. Management
typically wants to see regular progress reports, to ensure that goals will be
met.

A meeting is almost always the first step, to establish frequency of
reporting, to gather a list of any key data they want to see, and to
understand what questions they expect to have answered at each reporting
period. At this stage, some reporting goals will be concrete, while some will
be vague. Sample goals might be:

  • “Weekly status reports by noon on Monday and a status meeting to
    discuss at 10 a.m. on Tuesday”
  • “An e-mail if anything seems off, otherwise a monthly meeting”
  • “Whatever numbers seem most important to you at the time, and of course
    any changes to delivery date”
  • “An up-to-date earned value graph, and control graphs for defects found
    and fixed”

Almost any combination of information is possible at this point. Of course
the overall project manager will have some ideas about metrics that he or she
wants to capture, but this meeting is about senior management needs. The
project manager can collect other data, so long as management’s needs are
met.

Step 2: Plan Sub-Project Metrics

Each sub-project needs metrics to contribute to the overall scorecard. The
sub-project should measure project results to show variance, problems, and
progress. The sub-project manager should come up with ideas about the metrics
that he or she wants to collect. He or she may collect traditional project
management metrics like earned value metrics, budget vs. actuals, and
schedule slippage. Domain-specific metrics are often also useful, like number
of defects opened or closed, feet of wire laid, number of rooms painted,
number of code-reviews completed, and so on. The possibilities are infinite.
The choice of metrics will depend upon the sub-project manager’s experience
with metrics as well as the domain. Some sub-project managers may even decide
to collect no metrics of their own. At this point in the process, that is
fine.

Step 3: Identify Common Metrics

Once the overall project manager has a list from each sub-project manager,
he or she can look for common metrics. Sometimes two managers will call the
same metric two different things, or calculate the same basic metric two
different ways. For instance, a straight budget calculation of actual costs
to-date will usually be greater than an earned-value calculation of actual
costs, due to differences in definition of earned and unearned costs. Note
areas of possible conflict as well as areas of agreement.

At this stage it is also critical to compare the sub-project manager’s
metrics against the senior management needs. The metrics list may meet some
management needs completely, while other management needs have no supporting
metrics. For the next step, it is critical to analyze the results and develop
a strategy to meet management needs.

Step 4: Negotiate with Sub-Project Managers

With a list of common metrics, conflicting metrics, and missing metrics,
it is time to talk to each sub-project manager. There will always be some
changes to negotiate at this phase. Changes at a sub-project level might
include:

  • changing data collection methods (i.e. use an earned-value budget or an
    accounting budget)
  • adding more data elements to the scorecard
  • renaming data elements
  • changing data collection/reporting tools to achieve consistent
    results
  • resolving differences in definitions (i.e. what does “50% complete”
    mean?)

The overall project manager may need to convince some sub-project managers
to begin creating a scorecard at all. It is not uncommon for people to manage
projects without a scorecard, and the main benefit of a common scorecard is
to the overall project manager and to the overall management team. Some
people may resist the standardization required.

Any sub-project manager who decided to collect no metrics in step #2 must
now agree to collect a set of basic data, in order to participate in the
overall project. These negotiations can be difficult. It is critical for the
overall project manager to have a simple set of basic data needs, and to know
exactly why each piece of data is necessary. Sometimes resistant managers
fight metrics at every step of the way. The overall project manager should be
prepared with advice or even practical help to begin data collection for each
sub-project.

Be aware that gaining their cooperation or acceptance at this phase is not
enough. Sometime resistant managers can misreport or report late during the
project execution, as a form of passive resistance. The overall project
manager needs to understand each sub-project manager; this negotiating period
is a perfect time to get to know the overall project team really well.
Wherever possible, try to generate goodwill and cooperation among the
managers. Avoid creating situations where a single sub-project manager
appears to win every negotiation, while everyone else looses.

Step 5: Set Standards

The overall project manager takes the results of all the negotiations and
creates a list of standards for the overall project scorecard. The standards
should describe who collects what data and when. Responsibilities for each
sub-project should be clear and precise. Measurement methods should be exact.
For instance, for “percent complete” for tasks, describe your method. Sample
methods include:

  • Either 0% or 100% – complete or not yet complete
  • 0% if not started, 50% when started, 100% when complete
  • (100* Act) / (Act + ETC)
  • Different percentages for each phase of completion: 0% not started, 10%
    in-progress, 75% almost done, 90% ready for review, 100% passed review

Clear standards are critical for consistent reporting. Even the
best-intentioned managers will vary in the way that they report information.
It is impossible to have 100% consistency for some types of metrics, but a
good standards document increases consistency better than any other single
tool.

While setting standards, beware of vague standards that are hard to
enforce or audit. Ideally, metrics are like the results of scientific
experiments: anyone else could use the same methods, observe the same
phenomena, and get the same results. The overall project manager should be
able to verify the sub-project scorecards, at least within some range of
certainty, like plus or minus 20%.

Networked tools for project scheduling, defect tracking, and work tracking
simplify reporting considerably. Sometimes the sub-project manager does not
even need to report their data to the overall project manager. If the data is
networked, the overall project manager can read and report on the data
independently. This kind of transparency is a huge benefit in multi-project
situations.

Step 6: Create an Overall Project Scorecard

<br “A scorecard showing counters for UIs, Classes, and Interfaces. Shows number of items coded, tested, required and remaining, with totals. Data appears in a table and a bar graph.” />
Scorecards can combine tables and graphs to show a snapshot of current
status

With a standard document in hand, designing the scorecard is
straightforward. A few key questions will drive formatting and layout:

  • What information is most important?
  • What order will we discuss this information at status meetings (if
    applicable)?
  • How are we delivering the material (on-line, web, e-mail, text-only,
    print, projector, etc.)?
  • What are the limits of the physical media (color, font size,
    etc.)?
  • Do we need to highlight any key information?
  • Do any readers need the data in a particular format (graphical,
    percentage, table of results, etc.)?
  • What formatting or aesthetic standards should we follow?
  • Is this report too expensive to deliver?

In general, a good scorecard flows clearly from one set of information to
the next, in the order that management wants to see and discuss it. Data is
clear, formatting does not get in the way, and it is easy to scan the page
for critical information. Totals are obvious and meaningful.

Time and time again, I have found that management appreciates a one-page
scorecard. Sometimes this single page just has summary data. Additional pages
can always show the detail. A single-page summary makes it easy to see
project status at a glance.

Step 7: Create Overall Benchmarks

<br “A spreadsheet shows goals vs. actuals on a monthly basis for classes, UIs, and interfaces. Goals extend into the future, beyond the actual values. Below each actual is a variance, highlighted in green if it is positive, red if negative.” />
Benchmarks show progress over time, measured against concrete goals.
Color helps show that this project is falling behind, and is ready for
new benchmarks.

Certain scorecard metrics will be expected to steadily increase or
decrease over time. These metrics are good candidates for benchmarking.
Imagine benchmarks as a snapshot of expected values over time, with one
expected value for each reporting period. The most common form of
benchmarking is earned value analysis. Managers take the schedule and
budget at the start of the project, and write the expected earned value at
each period throughout the project. Regular status reports show the actual
values compared to the benchmarks.

The overall project manager can use this technique for any metric on the
scorecard. Benchmarks can be extraordinarily useful for schedule-constrained
projects. Management can see with every report whether work is on schedule,
ahead or schedule, or behind schedule, without reviewing hundreds of
tasks.

Some advice for creating benchmarks:

  • Only benchmark values that change predictably over time: number of open
    defects might seem like a good value to benchmark, because it must be zero
    at the end, but if the value goes up and down every week, it may be a poor
    choice
  • Benchmark a few values that really matter, that predict project success
    and show progress: too many benchmarks are confusing
  • Estimate likely progress: take into account vacation time, project
    phases, and seasonal issues, instead of just assuming a linear progression,
    like “2 more each week”
  • Publish the benchmarks: benchmarks are a great motivational tool for
    many teams, encouraging team members to contribute to the overall
    goals
  • Use sub-project benchmarks: give sub-project managers control over
    their own benchmarks, and total their goals to set the overall project
    benchmarks

The overall project manager must understand the purpose of the benchmarks.
If there is a prize for the team at each met-or-exceeded benchmark, then set
the benchmark goals high. Perhaps the manager sets the benchmarks so that he
or she is only 30% sure that the team will make it each week. Perhaps the
benchmarks show project completion several weeks ahead of schedule.
Communicate these expectations to management and to the teams.

Sometimes benchmarks are a simple monitoring tool. The overall project
manager should then set the benchmark at a spot where he or she expects the
team to perform. Estimating uncertainty in the estimate is very valuable.
Benchmarks can serve as a form of control graph, showing whether results are
in control or out of control. The classic control graph has a single result
expected over time, with horizontal bars for average, high, and low values.
The benchmark graph would have an expected-result line that moves up or down
over time, with control bars above and below it. When using this approach,
uncertainty usually varies over time, so the uncertainty levels may change
with each reporting period. As each goal comes closer in time, the likelihood
of reaching the goal becomes more and more certain.

Sub-project managers may or may not choose to create their own scorecards
for each value. Over the life of a software project, one sub-project might
occasionally find and fix one or two defects, while another sub-project finds
and fixes a hundred defects per week. A defect scorecard and benchmark could
be useful for the overall project and the second sub-project, but not for the
first. Be flexible about applying metrics at the overall and sub-project
level.

Step 8: Senior Management Approval

When developing any reporting tool, it is critical to get approval of its
target audience. A good project manager will involve the audience throughout
the process. When and how to involve senior management always varies by
organization, but with a draft scorecard developed and benchmarks set, it is
essential to present a proposal to management.

This meeting is typically quite formal. The overall project manager
distributes the benchmarks and scorecard ahead of time, then reviews it step
by step with management. A face-to-face meeting is recommended, even if the
project communication calls for no face-to-face, regular status meetings.
People need time to review and discuss the results. Different senior managers
may need different information, and it helps them to hear each other defend
each part of the report. Removing or changing a scorecard section requires
everyone’s approval, and often requires discussion.

Usually there is at least one change needed. Often the organization or
format of the document requires changes. Sometimes the benchmark goals are
too aggressive, or not aggressive enough. Occasionally the metrics do not
meet management needs, and the scorecard goes back to the drawing board.
Usually the overall project manager can note the changes in a follow-up
e-mail and begin using the new report for the next project-reporting period.
Going through this process almost always produces a report that is better
than the tools that came before it. Even if it is not perfect, management
usually is ready to adopt it.

Step 9: Update Scorecards, Evaluate Actuals vs. Benchmarks

With each reporting period, metrics change. The overall project manager
should set a reporting schedule with all sub-project managers, to meet the
reporting schedule required by management. Having everyone report all data as
of the same date is critical to having a useful overall project scorecard.
Assembling and reporting data for a complex project often takes several days.
Even after the sub-project managers have finished their reports, it may take
a day for the overall project manager to file the overall report. A tightly
organized team can improve that time, by meeting or beating reporting
deadlines consistently. Data may be old by the time it is published; balance
the competing needs for timely data against the needs for accurate,
well-reviewed data.

Each sub-project manager may have different reporting responsibilities for
each reporting period. Typical responsibilities include:

  • Producing the sub-project scorecard and benchmarks
  • Reviewing and approving data in shared databases
  • Reporting additional data to the overall project manager, via e-mail or
    phone
  • Meeting with the overall project manager to review and confirm project
    results

It is useful for the whole project-management team to get together and
review the overall results, before publishing them to senior management. This
step ensures that no one is surprised by the results in the report. The
overall project manager can reconcile any conflicts between sub-project
managers and can make last-minute corrections to the scorecards.

Benchmark values change every week. The overall project manager fills in
the actuals for the week and compares them against the benchmarks. Any
variances should have an explanation.

The overall project manager publishes the scorecards and benchmarks to
management. The cycle repeats for every reporting period until the project is
complete.

Step 10: Refine and Update

Projects are all unique endeavors, and an overall project is a collection
of many unique endeavors. Even the best project manager will have trouble
predicting the course of an overall project. Benchmarks are usually the first
to change. Changes in staffing levels, unexpected project events (positive or
negative), changes to scope, or changes to estimates all should trigger a
review of benchmarks. Benchmarks loose their value as a management and
motivation tool as soon as they become too easy or too difficult.

Often the project’s metrics change as the project goes through different
phases. The overall project manager should monitor overall project
activities, to see when a particular metric is no longer relevant or when a
new one should be captured. The overall project manager may choose to design
a new scorecard for each phase or to keep a single scorecard, but remove and
add data elements as the project evolves. Either approach can be successful.
Adding new metrics mean that the overall project manager should consider
adding a new benchmark goal. Dropping old metrics sometimes means dropping a
benchmark.

This step goes hand-in-hand with updating scorecards. It may be triggered
anytime after reporting begins on a project. Critical events in the project
almost always trigger it, including changes to scope, budget, schedule, or
phase.

During project close-out, the use of the scorecards and their evolution
through the project often make their way into the lessons learned for the
project. The history of scorecards and benchmarks will help the managers of
future projects develop their own templates and standards.

Putting Scorecards and Benchmarks to Use

Following these ten steps, a project manager can have an easier transition
from managing stand-alone projects to managing projects that contain
sub-projects. These techniques are broadly applicable, useful for a wide
range of industries. There is some key recommendations for people who are new
to these tools:

  1. Keep the metrics simple and easy to understand
  2. Make sure that sub-project metrics “scale up” to the overall
    project
  3. Do not force metrics that work for one sub-project onto all
    sub-projects; only use what works
  4. Watch your math, because not all metrics sum up through simple
    addition; standard deviation adds using squares, for instance – see
    recommendation #1 to avoid this problem entirely
  5. Explicitly state the uncertainty of the numbers at the start; it is
    next to impossible to collect data completely accurately or to forecast
    benchmarks perfectly
  6. When benchmark values and actual values diverge, do not panic; figure
    out if the project is off-course or if the metrics are off-course and take
    corrective action
  7. Never, ever adjust the scorecard numbers in order to appear to meet
    goals; not only is it unethical behavior, it is increasing the projects’
    list of problems and threatening the bond of trust with project
    sponsors
  8. Do adjust metric calculations and benchmarks as needed; do so openly,
    with the consent of senior managers and sub-project managers
  9. Use the literature to select appropriate metrics; academics have
    studied and argued over appropriate metrics for decades, and it is
    expensive to ignore their advice
  10. Only use sophisticated metrics after gaining experience with simpler
    metrics, and only if the audience of the reports understand the complex
    metrics thoroughly.

To summarize: FOCUS ON COMMUNICATION. These reports are a tool to
communicate progress and status. Be creative, using color, graphs, charts,
tables, pictures, or anything that helps communicate. An effective report
will generate complete silence around a meeting-room table, followed by
grunts of surprise, grunts of “humph,” exclamations of “wow!” and finally
cries, “This is great,” or “NOW I understand.” The measure of effectiveness
for a scorecard or benchmark is its communication value. It is not enough to
have numbers that accurately reflect project progress, they must also
communicate that progress to the intended audience. Perhaps the ultimate
compliment is when senior management starts telling your peers to use
benchmarks and scorecards, and your peers come to you as the in-house expert
for advice.

Ultimately the scorecards and benchmarks are about seeing the forest,
despite all the trees. Managers who remain focused on communicating the state
of the project “forest” will not get lost in the “trees” of numbers in their
reports. Giving senior management the right level of vision, supported by
accurate, demonstrable facts, is the ultimate goal of the exercise.

Leave a Reply