Love the new WordPress Admin area. Word count is especially welcome when posting. “Revisions” feature is absolutely fantastic! Letting you go back and compare saved versions is really nice.

Configuration of toolbar in TinyMCE does not seem to work. Have had issues with that in the past, though. Perhaps it is a configuration problem for me. Could just require a clearing of cache.

Version 2.7 looks good enough to upgrade. Now need to check out the mailing list module.

Need to MAKE SURE not to edit NAVT menus until the 2.7-compatible version comes out. Editing menus with the incompatible version blows away the old menus. Will backup menus before upgrading and after upgrading, just in case.

• Available as a four-contact hour, four PDU on-line class through Real-Life Projects, Inc..

and more to come….

Two versions of this paper are available. The short version was prepared for a newsletter and is approximately 2,000 words, or six pages. The long version (below) was prepared for a 45-minute presentation, contains some additional details, and runs approximately 3,800 words, or ten pages. Feel free to read either or both versions.

One of the best pieces of advice for any project manager is, “Divide and conquer.” Take a complex, large deliverable, and break it into parts. Deliver it in small pieces, each of which can be managed separately, with as few inter-dependencies as possible. This strategy is an absolute requirement at times. Some efforts are so large that a single project manager cannot reasonably track and manage the entire effort alone. Some efforts are so complex that they need multiple managers, each with different specialized knowledge. Thank goodness for “divide and conquer.”

There is a dark side to “divide and conquer”, however. At some point senior management will ask, “But how are we doing overall?” The individual status reports from each sub-project may not show the progress towards the overall goal. Cross-team problems may be ignored. Multiple formats from multiple authors can be confusing to read.

People want to see the forest, not just the trees. Creating an overall status report is a possible solution.

One approach to an overall status report is simple:

• collect text descriptions of status and goals from the manager of each sub-project,
• submit them all to a single person
• concatenate them into a single status report, editing the result for consistent format and language

This overall report can be extraordinarily helpful, to senior managers and sub-project managers alike. Sometimes it is enough, but a project manager should have additional tools in his or her toolbox.

A second approach is to create an overall schedule for the effort. These schedules can be as simple as a milestone-only schedule, with key sub-project deliverables tracked on a single plan. Dependencies and cross-team milestones can be represented. More complex versions will have a single master schedule that contains all tasks for all sub-projects, including actual and estimated work hours. Scheduling software can help automate creating these master schedules. Creating schedules is often a tool-specific challenge, and in my experience there are easier ways to get clearer information to senior management.

• Each Sub-Project Contributes to the Overall Scorecard

Any project can use benchmarks and scorecards, but they are especially useful for projects composed of sub-projects. Each sub-project manager can create a custom scorecard containing critical, relevant data for their effort. Often a useful scorecard can be just a single printed page. Team members can use the scorecards to track progress, gauging concrete results against benchmark numbers. The overall project manager can then collect these scorecards, combine the relevant data from each, and produce an overall scorecard. Usually it does not capture all the metrics of all the sub-projects, but it provides an accurate view of the forest of sub-projects. Benchmarks can be combined to track overall project-level progress. The approach is satisfying because each manager has local control over his or her deliverable, and can set benchmark goals independently. Overall management can still track progress towards the ultimate goal with confidence and precision.

Based upon my experience managing software projects in financial services, this work is never simple or easy. Perhaps in a very mature organization with concrete, well-understood standards it would be. Most of us do not have the privilege of working in such an organization. Using a step-by-step process, though, anyone can establish some consistent standards for a multiple-project effort. If these standards survive the effort, the overall manager can take pride; he or she has increased the maturity of the organization while completing the project.

A note on terminology: Some organizations talk about “programs” composed of multiple “projects”. Other organizations use “program” to mean a whole product line, while still others talk about “projects” composed of multiple “programs”. This paper will avoid the term “program” entirely. Instead “sub-projects” are parts of a “multiple-project effort” or an “overall project”. “Project manager” means any project manager, while “overall project manager” is someone running the overall project. “Sub-project manager” is someone running a sub-project.

Step 1: Determine Senior Management Needs

The target audience for these benchmarks and scorecards is almost always senior management. Most stakeholders are concerned with either day-to-day progress at a low level, or overall completion dates and cost. Management typically wants to see regular progress reports, to ensure that goals will be met.

A meeting is almost always the first step, to establish frequency of reporting, to gather a list of any key data they want to see, and to understand what questions they expect to have answered at each reporting period. At this stage, some reporting goals will be concrete, while some will be vague. Sample goals might be:

• “Weekly status reports by noon on Monday and a status meeting to discuss at 10 a.m. on Tuesday”
• “An e-mail if anything seems off, otherwise a monthly meeting”
• “Whatever numbers seem most important to you at the time, and of course any changes to delivery date”
• “An up-to-date earned value graph, and control graphs for defects found and fixed”

Almost any combination of information is possible at this point. Of course the overall project manager will have some ideas about metrics that he or she wants to capture, but this meeting is about senior management needs. The project manager can collect other data, so long as management’s needs are met.

Step 2: Plan Sub-Project Metrics

Each sub-project needs metrics to contribute to the overall scorecard. The sub-project should measure project results to show variance, problems, and progress. The sub-project manager should come up with ideas about the metrics that he or she wants to collect. He or she may collect traditional project management metrics like earned value metrics, budget vs. actuals, and schedule slippage. Domain-specific metrics are often also useful, like number of defects opened or closed, feet of wire laid, number of rooms painted, number of code-reviews completed, and so on. The possibilities are infinite. The choice of metrics will depend upon the sub-project manager’s experience with metrics as well as the domain. Some sub-project managers may even decide to collect no metrics of their own. At this point in the process, that is fine.

Step 3: Identify Common Metrics

Once the overall project manager has a list from each sub-project manager, he or she can look for common metrics. Sometimes two managers will call the same metric two different things, or calculate the same basic metric two different ways. For instance, a straight budget calculation of actual costs to-date will usually be greater than an earned-value calculation of actual costs, due to differences in definition of earned and unearned costs. Note areas of possible conflict as well as areas of agreement.

At this stage it is also critical to compare the sub-project manager’s metrics against the senior management needs. The metrics list may meet some management needs completely, while other management needs have no supporting metrics. For the next step, it is critical to analyze the results and develop a strategy to meet management needs.

Step 4: Negotiate with Sub-Project Managers

With a list of common metrics, conflicting metrics, and missing metrics, it is time to talk to each sub-project manager. There will always be some changes to negotiate at this phase. Changes at a sub-project level might include:

• changing data collection methods (i.e. use an earned-value budget or an accounting budget)
• adding more data elements to the scorecard
• renaming data elements
• changing data collection/reporting tools to achieve consistent results
• resolving differences in definitions (i.e. what does “50% complete” mean?)

The overall project manager may need to convince some sub-project managers to begin creating a scorecard at all. It is not uncommon for people to manage projects without a scorecard, and the main benefit of a common scorecard is to the overall project manager and to the overall management team. Some people may resist the standardization required.

Any sub-project manager who decided to collect no metrics in step #2 must now agree to collect a set of basic data, in order to participate in the overall project. These negotiations can be difficult. It is critical for the overall project manager to have a simple set of basic data needs, and to know exactly why each piece of data is necessary. Sometimes resistant managers fight metrics at every step of the way. The overall project manager should be prepared with advice or even practical help to begin data collection for each sub-project.

Be aware that gaining their cooperation or acceptance at this phase is not enough. Sometime resistant managers can misreport or report late during the project execution, as a form of passive resistance. The overall project manager needs to understand each sub-project manager; this negotiating period is a perfect time to get to know the overall project team really well. Wherever possible, try to generate goodwill and cooperation among the managers. Avoid creating situations where a single sub-project manager appears to win every negotiation, while everyone else looses.

Step 5: Set Standards

The overall project manager takes the results of all the negotiations and creates a list of standards for the overall project scorecard. The standards should describe who collects what data and when. Responsibilities for each sub-project should be clear and precise. Measurement methods should be exact. For instance, for “percent complete” for tasks, describe your method. Sample methods include:

• Either 0% or 100% – complete or not yet complete
• 0% if not started, 50% when started, 100% when complete
• (100* Act) / (Act + ETC)
• Different percentages for each phase of completion: 0% not started, 10% in-progress, 75% almost done, 90% ready for review, 100% passed review

Clear standards are critical for consistent reporting. Even the best-intentioned managers will vary in the way that they report information. It is impossible to have 100% consistency for some types of metrics, but a good standards document increases consistency better than any other single tool.

While setting standards, beware of vague standards that are hard to enforce or audit. Ideally, metrics are like the results of scientific experiments: anyone else could use the same methods, observe the same phenomena, and get the same results. The overall project manager should be able to verify the sub-project scorecards, at least within some range of certainty, like plus or minus 20%.

Networked tools for project scheduling, defect tracking, and work tracking simplify reporting considerably. Sometimes the sub-project manager does not even need to report their data to the overall project manager. If the data is networked, the overall project manager can read and report on the data independently. This kind of transparency is a huge benefit in multi-project situations.

Step 6: Create an Overall Project Scorecard

• Scorecards can combine tables and graphs to show a snapshot of current status

With a standard document in hand, designing the scorecard is straightforward. A few key questions will drive formatting and layout:

• What information is most important?
• What order will we discuss this information at status meetings (if applicable)?
• How are we delivering the material (on-line, web, e-mail, text-only, print, projector, etc.)?
• What are the limits of the physical media (color, font size, etc.)?
• Do we need to highlight any key information?
• Do any readers need the data in a particular format (graphical, percentage, table of results, etc.)?
• What formatting or aesthetic standards should we follow?
• Is this report too expensive to deliver?

In general, a good scorecard flows clearly from one set of information to the next, in the order that management wants to see and discuss it. Data is clear, formatting does not get in the way, and it is easy to scan the page for critical information. Totals are obvious and meaningful.

Time and time again, I have found that management appreciates a one-page scorecard. Sometimes this single page just has summary data. Additional pages can always show the detail. A single-page summary makes it easy to see project status at a glance.

• Step 7: Create Overall Benchmarks

  • Benchmarks show progress over time, measured against concrete goals. Color helps show that this project is falling behind, and is ready for new benchmarks.

  Certain scorecard metrics will be expected to steadily increase or decrease over time. These metrics are good candidates for benchmarking. Imagine benchmarks as a snapshot of expected values over time, with one expected value for each reporting period. The most common form of benchmarking is earned value analysis. Managers take the schedule and budget at the start of the project, and write the expected earned value at each period throughout the project. Regular status reports show the actual values compared to the benchmarks.

The overall project manager can use this technique for any metric on the scorecard. Benchmarks can be extraordinarily useful for schedule-constrained projects. Management can see with every report whether work is on schedule, ahead or schedule, or behind schedule, without reviewing hundreds of tasks.

Some advice for creating benchmarks:

• Only benchmark values that change predictably over time: number of open defects might seem like a good value to benchmark, because it must be zero at the end, but if the value goes up and down every week, it may be a poor choice
• Benchmark a few values that really matter, that predict project success and show progress: too many benchmarks are confusing
• Estimate likely progress: take into account vacation time, project phases, and seasonal issues, instead of just assuming a linear progression, like “2 more each week”
• Publish the benchmarks: benchmarks are a great motivational tool for many teams, encouraging team members to contribute to the overall goals
• Use sub-project benchmarks: give sub-project managers control over their own benchmarks, and total their goals to set the overall project benchmarks

The overall project manager must understand the purpose of the benchmarks. If there is a prize for the team at each met-or-exceeded benchmark, then set the benchmark goals high. Perhaps the manager sets the benchmarks so that he or she is only 30% sure that the team will make it each week. Perhaps the benchmarks show project completion several weeks ahead of schedule. Communicate these expectations to management and to the teams.

Sometimes benchmarks are a simple monitoring tool. The overall project manager should then set the benchmark at a spot where he or she expects the team to perform. Estimating uncertainty in the estimate is very valuable. Benchmarks can serve as a form of control graph, showing whether results are in control or out of control. The classic control graph has a single result expected over time, with horizontal bars for average, high, and low values. The benchmark graph would have an expected-result line that moves up or down over time, with control bars above and below it. When using this approach, uncertainty usually varies over time, so the uncertainty levels may change with each reporting period. As each goal comes closer in time, the likelihood of reaching the goal becomes more and more certain.

Sub-project managers may or may not choose to create their own scorecards for each value. Over the life of a software project, one sub-project might occasionally find and fix one or two defects, while another sub-project finds and fixes a hundred defects per week. A defect scorecard and benchmark could be useful for the overall project and the second sub-project, but not for the first. Be flexible about applying metrics at the overall and sub-project level.

Step 8: Senior Management Approval

When developing any reporting tool, it is critical to get approval of its target audience. A good project manager will involve the audience throughout the process. When and how to involve senior management always varies by organization, but with a draft scorecard developed and benchmarks set, it is essential to present a proposal to management.

This meeting is typically quite formal. The overall project manager distributes the benchmarks and scorecard ahead of time, then reviews it step by step with management. A face-to-face meeting is recommended, even if the project communication calls for no face-to-face, regular status meetings. People need time to review and discuss the results. Different senior managers may need different information, and it helps them to hear each other defend each part of the report. Removing or changing a scorecard section requires everyone’s approval, and often requires discussion.

Usually there is at least one change needed. Often the organization or format of the document requires changes. Sometimes the benchmark goals are too aggressive, or not aggressive enough. Occasionally the metrics do not meet management needs, and the scorecard goes back to the drawing board. Usually the overall project manager can note the changes in a follow-up e-mail and begin using the new report for the next project-reporting period. Going through this process almost always produces a report that is better than the tools that came before it. Even if it is not perfect, management usually is ready to adopt it.

Step 9: Update Scorecards, Evaluate Actuals vs. Benchmarks

With each reporting period, metrics change. The overall project manager should set a reporting schedule with all sub-project managers, to meet the reporting schedule required by management. Having everyone report all data as of the same date is critical to having a useful overall project scorecard. Assembling and reporting data for a complex project often takes several days. Even after the sub-project managers have finished their reports, it may take a day for the overall project manager to file the overall report. A tightly organized team can improve that time, by meeting or beating reporting deadlines consistently. Data may be old by the time it is published; balance the competing needs for timely data against the needs for accurate, well-reviewed data.

Each sub-project manager may have different reporting responsibilities for each reporting period. Typical responsibilities include:

• Producing the sub-project scorecard and benchmarks
• Reviewing and approving data in shared databases
• Reporting additional data to the overall project manager, via e-mail or phone
• Meeting with the overall project manager to review and confirm project results

It is useful for the whole project-management team to get together and review the overall results, before publishing them to senior management. This step ensures that no one is surprised by the results in the report. The overall project manager can reconcile any conflicts between sub-project managers and can make last-minute corrections to the scorecards.

Benchmark values change every week. The overall project manager fills in the actuals for the week and compares them against the benchmarks. Any variances should have an explanation.

The overall project manager publishes the scorecards and benchmarks to management. The cycle repeats for every reporting period until the project is complete.

Step 10: Refine and Update

Projects are all unique endeavors, and an overall project is a collection of many unique endeavors. Even the best project manager will have trouble predicting the course of an overall project. Benchmarks are usually the first to change. Changes in staffing levels, unexpected project events (positive or negative), changes to scope, or changes to estimates all should trigger a review of benchmarks. Benchmarks loose their value as a management and motivation tool as soon as they become too easy or too difficult.

Often the project’s metrics change as the project goes through different phases. The overall project manager should monitor overall project activities, to see when a particular metric is no longer relevant or when a new one should be captured. The overall project manager may choose to design a new scorecard for each phase or to keep a single scorecard, but remove and add data elements as the project evolves. Either approach can be successful. Adding new metrics mean that the overall project manager should consider adding a new benchmark goal. Dropping old metrics sometimes means dropping a benchmark.

This step goes hand-in-hand with updating scorecards. It may be triggered anytime after reporting begins on a project. Critical events in the project almost always trigger it, including changes to scope, budget, schedule, or phase.

During project close-out, the use of the scorecards and their evolution through the project often make their way into the lessons learned for the project. The history of scorecards and benchmarks will help the managers of future projects develop their own templates and standards.

Putting Scorecards and Benchmarks to Use

Following these ten steps, a project manager can have an easier transition from managing stand-alone projects to managing projects that contain sub-projects. These techniques are broadly applicable, useful for a wide range of industries. There is some key recommendations for people who are new to these tools:

1. Keep the metrics simple and easy to understand
2. Make sure that sub-project metrics “scale up” to the overall project
3. Do not force metrics that work for one sub-project onto all sub-projects; only use what works
4. Watch your math, because not all metrics sum up through simple addition; standard deviation adds using squares, for instance – see recommendation #1 to avoid this problem entirely
5. Explicitly state the uncertainty of the numbers at the start; it is next to impossible to collect data completely accurately or to forecast benchmarks perfectly
6. When benchmark values and actual values diverge, do not panic; figure out if the project is off-course or if the metrics are off-course and take corrective action
7. Never, ever adjust the scorecard numbers in order to appear to meet goals; not only is it unethical behavior, it is increasing the projects’ list of problems and threatening the bond of trust with project sponsors
8. Do adjust metric calculations and benchmarks as needed; do so openly, with the consent of senior managers and sub-project managers
9. Use the literature to select appropriate metrics; academics have studied and argued over appropriate metrics for decades, and it is expensive to ignore their advice
10. Only use sophisticated metrics after gaining experience with simpler metrics, and only if the audience of the reports understand the complex metrics thoroughly.

To summarize: FOCUS ON COMMUNICATION. These reports are a tool to communicate progress and status. Be creative, using color, graphs, charts, tables, pictures, or anything that helps communicate. An effective report will generate complete silence around a meeting-room table, followed by grunts of surprise, grunts of “humph,” exclamations of “wow!” and finally cries, “This is great,” or “NOW I understand.” The measure of effectiveness for a scorecard or benchmark is its communication value. It is not enough to have numbers that accurately reflect project progress, they must also communicate that progress to the intended audience. Perhaps the ultimate compliment is when senior management starts telling your peers to use benchmarks and scorecards, and your peers come to you as the in-house expert for advice.

Ultimately the scorecards and benchmarks are about seeing the forest, despite all the trees. Managers who remain focused on communicating the state of the project “forest” will not get lost in the “trees” of numbers in their reports. Giving senior management the right level of vision, supported by accurate, demonstrable facts, is the ultimate goal of the exercise.

Two versions of this paper are available. The short version was prepared for a newsletter and is approximately 2,000 words, or six pages. The long version (below) was prepared for a 45-minute presentation, contains some additional details, and runs approximately 3,800 words, or ten pages. Feel free to read either or both versions.

One of the best pieces of advice for any project manager is, “Divide and conquer.” Take a complex, large deliverable, and break it into parts. Deliver it in small pieces, each of which can be managed separately, with as few inter-dependencies as possible. This strategy is an absolute requirement at times. Some efforts are so large that a single project manager cannot reasonably track and manage the entire effort alone. Some efforts are so complex that they need multiple managers, each with different specialized knowledge. Thank goodness for “divide and conquer.”

There is a dark side to “divide and conquer”, however. At some point senior management will ask, “But how are we doing overall?” The individual status reports from each sub-project may not show the progress towards the overall goal. Cross-team problems may be ignored. Multiple formats from multiple authors can be confusing to read.

People want to see the forest, not just the trees. Creating an overall status report is a possible solution.

One approach to an overall status report is simple:

• collect text descriptions of status and goals from the manager of each sub-project,
• submit them all to a single person
• concatenate them into a single status report, editing the result for consistent format and language

This overall report can be extraordinarily helpful, to senior managers and sub-project managers alike. Sometimes it is enough, but a project manager should have additional tools in his or her toolbox.

A second approach is to create an overall schedule for the effort. These schedules can be as simple as a milestone-only schedule, with key sub-project deliverables tracked on a single plan. Dependencies and cross-team milestones can be represented. More complex versions will have a single master schedule that contains all tasks for all sub-projects, including actual and estimated work hours. Scheduling software can help automate creating these master schedules. Creating schedules is often a tool-specific challenge, and in my experience there are easier ways to get clearer information to senior management.

• Each Sub-Project Contributes to the Overall Scorecard

Any project can use benchmarks and scorecards, but they are especially useful for projects composed of sub-projects. Each sub-project manager can create a custom scorecard containing critical, relevant data for their effort. Often a useful scorecard can be just a single printed page. Team members can use the scorecards to track progress, gauging concrete results against benchmark numbers. The overall project manager can then collect these scorecards, combine the relevant data from each, and produce an overall scorecard. Usually it does not capture all the metrics of all the sub-projects, but it provides an accurate view of the forest of sub-projects. Benchmarks can be combined to track overall project-level progress. The approach is satisfying because each manager has local control over his or her deliverable, and can set benchmark goals independently. Overall management can still track progress towards the ultimate goal with confidence and precision.

Based upon my experience managing software projects in financial services, this work is never simple or easy. Perhaps in a very mature organization with concrete, well-understood standards it would be. Most of us do not have the privilege of working in such an organization. Using a step-by-step process, though, anyone can establish some consistent standards for a multiple-project effort. If these standards survive the effort, the overall manager can take pride; he or she has increased the maturity of the organization while completing the project.

A note on terminology: Some organizations talk about “programs” composed of multiple “projects”. Other organizations use “program” to mean a whole product line, while still others talk about “projects” composed of multiple “programs”. This paper will avoid the term “program” entirely. Instead “sub-projects” are parts of a “multiple-project effort” or an “overall project”. “Project manager” means any project manager, while “overall project manager” is someone running the overall project. “Sub-project manager” is someone running a sub-project.

Step 1: Determine Senior Management Needs

The target audience for these benchmarks and scorecards is almost always senior management. Most stakeholders are concerned with either day-to-day progress at a low level, or overall completion dates and cost. Management typically wants to see regular progress reports, to ensure that goals will be met.

A meeting is almost always the first step, to establish frequency of reporting, to gather a list of any key data they want to see, and to understand what questions they expect to have answered at each reporting period. At this stage, some reporting goals will be concrete, while some will be vague. Sample goals might be:

• “Weekly status reports by noon on Monday and a status meeting to discuss at 10 a.m. on Tuesday”
• “An e-mail if anything seems off, otherwise a monthly meeting”
• “Whatever numbers seem most important to you at the time, and of course any changes to delivery date”
• “An up-to-date earned value graph, and control graphs for defects found and fixed”

Almost any combination of information is possible at this point. Of course the overall project manager will have some ideas about metrics that he or she wants to capture, but this meeting is about senior management needs. The project manager can collect other data, so long as management’s needs are met.

Step 2: Plan Sub-Project Metrics

Each sub-project needs metrics to contribute to the overall scorecard. The sub-project should measure project results to show variance, problems, and progress. The sub-project manager should come up with ideas about the metrics that he or she wants to collect. He or she may collect traditional project management metrics like earned value metrics, budget vs. actuals, and schedule slippage. Domain-specific metrics are often also useful, like number of defects opened or closed, feet of wire laid, number of rooms painted, number of code-reviews completed, and so on. The possibilities are infinite. The choice of metrics will depend upon the sub-project manager’s experience with metrics as well as the domain. Some sub-project managers may even decide to collect no metrics of their own. At this point in the process, that is fine.

Step 3: Identify Common Metrics

Once the overall project manager has a list from each sub-project manager, he or she can look for common metrics. Sometimes two managers will call the same metric two different things, or calculate the same basic metric two different ways. For instance, a straight budget calculation of actual costs to-date will usually be greater than an earned-value calculation of actual costs, due to differences in definition of earned and unearned costs. Note areas of possible conflict as well as areas of agreement.

At this stage it is also critical to compare the sub-project manager’s metrics against the senior management needs. The metrics list may meet some management needs completely, while other management needs have no supporting metrics. For the next step, it is critical to analyze the results and develop a strategy to meet management needs.

Step 4: Negotiate with Sub-Project Managers

With a list of common metrics, conflicting metrics, and missing metrics, it is time to talk to each sub-project manager. There will always be some changes to negotiate at this phase. Changes at a sub-project level might include:

• changing data collection methods (i.e. use an earned-value budget or an accounting budget)
• adding more data elements to the scorecard
• renaming data elements
• changing data collection/reporting tools to achieve consistent results
• resolving differences in definitions (i.e. what does “50% complete” mean?)

The overall project manager may need to convince some sub-project managers to begin creating a scorecard at all. It is not uncommon for people to manage projects without a scorecard, and the main benefit of a common scorecard is to the overall project manager and to the overall management team. Some people may resist the standardization required.

Any sub-project manager who decided to collect no metrics in step #2 must now agree to collect a set of basic data, in order to participate in the overall project. These negotiations can be difficult. It is critical for the overall project manager to have a simple set of basic data needs, and to know exactly why each piece of data is necessary. Sometimes resistant managers fight metrics at every step of the way. The overall project manager should be prepared with advice or even practical help to begin data collection for each sub-project.

Be aware that gaining their cooperation or acceptance at this phase is not enough. Sometime resistant managers can misreport or report late during the project execution, as a form of passive resistance. The overall project manager needs to understand each sub-project manager; this negotiating period is a perfect time to get to know the overall project team really well. Wherever possible, try to generate goodwill and cooperation among the managers. Avoid creating situations where a single sub-project manager appears to win every negotiation, while everyone else looses.

Step 5: Set Standards

The overall project manager takes the results of all the negotiations and creates a list of standards for the overall project scorecard. The standards should describe who collects what data and when. Responsibilities for each sub-project should be clear and precise. Measurement methods should be exact. For instance, for “percent complete” for tasks, describe your method. Sample methods include:

• Either 0% or 100% – complete or not yet complete
• 0% if not started, 50% when started, 100% when complete
• (100* Act) / (Act + ETC)
• Different percentages for each phase of completion: 0% not started, 10% in-progress, 75% almost done, 90% ready for review, 100% passed review

Clear standards are critical for consistent reporting. Even the best-intentioned managers will vary in the way that they report information. It is impossible to have 100% consistency for some types of metrics, but a good standards document increases consistency better than any other single tool.

While setting standards, beware of vague standards that are hard to enforce or audit. Ideally, metrics are like the results of scientific experiments: anyone else could use the same methods, observe the same phenomena, and get the same results. The overall project manager should be able to verify the sub-project scorecards, at least within some range of certainty, like plus or minus 20%.

Networked tools for project scheduling, defect tracking, and work tracking simplify reporting considerably. Sometimes the sub-project manager does not even need to report their data to the overall project manager. If the data is networked, the overall project manager can read and report on the data independently. This kind of transparency is a huge benefit in multi-project situations.

Step 6: Create an Overall Project Scorecard

• Scorecards can combine tables and graphs to show a snapshot of current status

With a standard document in hand, designing the scorecard is straightforward. A few key questions will drive formatting and layout:

• What information is most important?
• What order will we discuss this information at status meetings (if applicable)?
• How are we delivering the material (on-line, web, e-mail, text-only, print, projector, etc.)?
• What are the limits of the physical media (color, font size, etc.)?
• Do we need to highlight any key information?
• Do any readers need the data in a particular format (graphical, percentage, table of results, etc.)?
• What formatting or aesthetic standards should we follow?
• Is this report too expensive to deliver?

In general, a good scorecard flows clearly from one set of information to the next, in the order that management wants to see and discuss it. Data is clear, formatting does not get in the way, and it is easy to scan the page for critical information. Totals are obvious and meaningful.

Time and time again, I have found that management appreciates a one-page scorecard. Sometimes this single page just has summary data. Additional pages can always show the detail. A single-page summary makes it easy to see project status at a glance.

• Step 7: Create Overall Benchmarks

  • Benchmarks show progress over time, measured against concrete goals. Color helps show that this project is falling behind, and is ready for new benchmarks.

  Certain scorecard metrics will be expected to steadily increase or decrease over time. These metrics are good candidates for benchmarking. Imagine benchmarks as a snapshot of expected values over time, with one expected value for each reporting period. The most common form of benchmarking is earned value analysis. Managers take the schedule and budget at the start of the project, and write the expected earned value at each period throughout the project. Regular status reports show the actual values compared to the benchmarks.

The overall project manager can use this technique for any metric on the scorecard. Benchmarks can be extraordinarily useful for schedule-constrained projects. Management can see with every report whether work is on schedule, ahead or schedule, or behind schedule, without reviewing hundreds of tasks.

Some advice for creating benchmarks:

• Only benchmark values that change predictably over time: number of open defects might seem like a good value to benchmark, because it must be zero at the end, but if the value goes up and down every week, it may be a poor choice
• Benchmark a few values that really matter, that predict project success and show progress: too many benchmarks are confusing
• Estimate likely progress: take into account vacation time, project phases, and seasonal issues, instead of just assuming a linear progression, like “2 more each week”
• Publish the benchmarks: benchmarks are a great motivational tool for many teams, encouraging team members to contribute to the overall goals
• Use sub-project benchmarks: give sub-project managers control over their own benchmarks, and total their goals to set the overall project benchmarks

The overall project manager must understand the purpose of the benchmarks. If there is a prize for the team at each met-or-exceeded benchmark, then set the benchmark goals high. Perhaps the manager sets the benchmarks so that he or she is only 30% sure that the team will make it each week. Perhaps the benchmarks show project completion several weeks ahead of schedule. Communicate these expectations to management and to the teams.

Sometimes benchmarks are a simple monitoring tool. The overall project manager should then set the benchmark at a spot where he or she expects the team to perform. Estimating uncertainty in the estimate is very valuable. Benchmarks can serve as a form of control graph, showing whether results are in control or out of control. The classic control graph has a single result expected over time, with horizontal bars for average, high, and low values. The benchmark graph would have an expected-result line that moves up or down over time, with control bars above and below it. When using this approach, uncertainty usually varies over time, so the uncertainty levels may change with each reporting period. As each goal comes closer in time, the likelihood of reaching the goal becomes more and more certain.

Sub-project managers may or may not choose to create their own scorecards for each value. Over the life of a software project, one sub-project might occasionally find and fix one or two defects, while another sub-project finds and fixes a hundred defects per week. A defect scorecard and benchmark could be useful for the overall project and the second sub-project, but not for the first. Be flexible about applying metrics at the overall and sub-project level.

Step 8: Senior Management Approval

When developing any reporting tool, it is critical to get approval of its target audience. A good project manager will involve the audience throughout the process. When and how to involve senior management always varies by organization, but with a draft scorecard developed and benchmarks set, it is essential to present a proposal to management.

This meeting is typically quite formal. The overall project manager distributes the benchmarks and scorecard ahead of time, then reviews it step by step with management. A face-to-face meeting is recommended, even if the project communication calls for no face-to-face, regular status meetings. People need time to review and discuss the results. Different senior managers may need different information, and it helps them to hear each other defend each part of the report. Removing or changing a scorecard section requires everyone’s approval, and often requires discussion.

Usually there is at least one change needed. Often the organization or format of the document requires changes. Sometimes the benchmark goals are too aggressive, or not aggressive enough. Occasionally the metrics do not meet management needs, and the scorecard goes back to the drawing board. Usually the overall project manager can note the changes in a follow-up e-mail and begin using the new report for the next project-reporting period. Going through this process almost always produces a report that is better than the tools that came before it. Even if it is not perfect, management usually is ready to adopt it.

Step 9: Update Scorecards, Evaluate Actuals vs. Benchmarks

With each reporting period, metrics change. The overall project manager should set a reporting schedule with all sub-project managers, to meet the reporting schedule required by management. Having everyone report all data as of the same date is critical to having a useful overall project scorecard. Assembling and reporting data for a complex project often takes several days. Even after the sub-project managers have finished their reports, it may take a day for the overall project manager to file the overall report. A tightly organized team can improve that time, by meeting or beating reporting deadlines consistently. Data may be old by the time it is published; balance the competing needs for timely data against the needs for accurate, well-reviewed data.

Each sub-project manager may have different reporting responsibilities for each reporting period. Typical responsibilities include:

• Producing the sub-project scorecard and benchmarks
• Reviewing and approving data in shared databases
• Reporting additional data to the overall project manager, via e-mail or phone
• Meeting with the overall project manager to review and confirm project results

It is useful for the whole project-management team to get together and review the overall results, before publishing them to senior management. This step ensures that no one is surprised by the results in the report. The overall project manager can reconcile any conflicts between sub-project managers and can make last-minute corrections to the scorecards.

Benchmark values change every week. The overall project manager fills in the actuals for the week and compares them against the benchmarks. Any variances should have an explanation.

The overall project manager publishes the scorecards and benchmarks to management. The cycle repeats for every reporting period until the project is complete.

Step 10: Refine and Update

Projects are all unique endeavors, and an overall project is a collection of many unique endeavors. Even the best project manager will have trouble predicting the course of an overall project. Benchmarks are usually the first to change. Changes in staffing levels, unexpected project events (positive or negative), changes to scope, or changes to estimates all should trigger a review of benchmarks. Benchmarks loose their value as a management and motivation tool as soon as they become too easy or too difficult.

Often the project’s metrics change as the project goes through different phases. The overall project manager should monitor overall project activities, to see when a particular metric is no longer relevant or when a new one should be captured. The overall project manager may choose to design a new scorecard for each phase or to keep a single scorecard, but remove and add data elements as the project evolves. Either approach can be successful. Adding new metrics mean that the overall project manager should consider adding a new benchmark goal. Dropping old metrics sometimes means dropping a benchmark.

This step goes hand-in-hand with updating scorecards. It may be triggered anytime after reporting begins on a project. Critical events in the project almost always trigger it, including changes to scope, budget, schedule, or phase.

During project close-out, the use of the scorecards and their evolution through the project often make their way into the lessons learned for the project. The history of scorecards and benchmarks will help the managers of future projects develop their own templates and standards.

Putting Scorecards and Benchmarks to Use

Following these ten steps, a project manager can have an easier transition from managing stand-alone projects to managing projects that contain sub-projects. These techniques are broadly applicable, useful for a wide range of industries. There is some key recommendations for people who are new to these tools:

1. Keep the metrics simple and easy to understand
2. Make sure that sub-project metrics “scale up” to the overall project
3. Do not force metrics that work for one sub-project onto all sub-projects; only use what works
4. Watch your math, because not all metrics sum up through simple addition; standard deviation adds using squares, for instance – see recommendation #1 to avoid this problem entirely
5. Explicitly state the uncertainty of the numbers at the start; it is next to impossible to collect data completely accurately or to forecast benchmarks perfectly
6. When benchmark values and actual values diverge, do not panic; figure out if the project is off-course or if the metrics are off-course and take corrective action
7. Never, ever adjust the scorecard numbers in order to appear to meet goals; not only is it unethical behavior, it is increasing the projects’ list of problems and threatening the bond of trust with project sponsors
8. Do adjust metric calculations and benchmarks as needed; do so openly, with the consent of senior managers and sub-project managers
9. Use the literature to select appropriate metrics; academics have studied and argued over appropriate metrics for decades, and it is expensive to ignore their advice
10. Only use sophisticated metrics after gaining experience with simpler metrics, and only if the audience of the reports understand the complex metrics thoroughly.

To summarize: FOCUS ON COMMUNICATION. These reports are a tool to communicate progress and status. Be creative, using color, graphs, charts, tables, pictures, or anything that helps communicate. An effective report will generate complete silence around a meeting-room table, followed by grunts of surprise, grunts of “humph,” exclamations of “wow!” and finally cries, “This is great,” or “NOW I understand.” The measure of effectiveness for a scorecard or benchmark is its communication value. It is not enough to have numbers that accurately reflect project progress, they must also communicate that progress to the intended audience. Perhaps the ultimate compliment is when senior management starts telling your peers to use benchmarks and scorecards, and your peers come to you as the in-house expert for advice.

Ultimately the scorecards and benchmarks are about seeing the forest, despite all the trees. Managers who remain focused on communicating the state of the project “forest” will not get lost in the “trees” of numbers in their reports. Giving senior management the right level of vision, supported by accurate, demonstrable facts, is the ultimate goal of the exercise.

Download Article

Printer-friendly (PDF)

Download Slides, light

Printer-friendly (PDF) MS PowerPoint

Download Slides, dark

Printer-friendly (PDF) MS PowerPoint

Download Sample Scorecard

MS Excel (XLS)

Two versions of this paper are available. The short version was prepared for a newsletter and is approximately 2,000 words, or six pages. The long version (below) was prepared for a 45-minute presentation, contains some additional details, and runs approximately 3,800 words, or ten pages. Feel free to read either or both versions.

One of the best pieces of advice for any project manager is, “Divide and conquer.” Take a complex, large deliverable, and break it into parts. Deliver it in small pieces, each of which can be managed separately, with as few inter-dependencies as possible. This strategy is an absolute requirement at times. Some efforts are so large that a single project manager cannot reasonably track and manage the entire effort alone. Some efforts are so complex that they need multiple managers, each with different specialized knowledge. Thank goodness for “divide and conquer.”

There is a dark side to “divide and conquer”, however. At some point senior management will ask, “But how are we doing overall?” The individual status reports from each sub-project may not show the progress towards the overall goal. Cross-team problems may be ignored. Multiple formats from multiple authors can be confusing to read.

People want to see the forest, not just the trees. Creating an overall status report is a possible solution.

One approach to an overall status report is simple:

• collect text descriptions of status and goals from the manager of each sub-project,
• submit them all to a single person
• concatenate them into a single status report, editing the result for consistent format and language

This overall report can be extraordinarily helpful, to senior managers and sub-project managers alike. Sometimes it is enough, but a project manager should have additional tools in his or her toolbox.

A second approach is to create an overall schedule for the effort. These schedules can be as simple as a milestone-only schedule, with key sub-project deliverables tracked on a single plan. Dependencies and cross-team milestones can be represented. More complex versions will have a single master schedule that contains all tasks for all sub-projects, including actual and estimated work hours. Scheduling software can help automate creating these master schedules. Creating schedules is often a tool-specific challenge, and in my experience there are easier ways to get clearer information to senior management.

Each Sub-Project Contributes to the Overall Scorecard

Any project can use benchmarks and scorecards, but they are especially useful for projects composed of sub-projects. Each sub-project manager can create a custom scorecard containing critical, relevant data for their effort. Often a useful scorecard can be just a single printed page. Team members can use the scorecards to track progress, gauging concrete results against benchmark numbers. The overall project manager can then collect these scorecards, combine the relevant data from each, and produce an overall scorecard. Usually it does not capture all the metrics of all the sub-projects, but it provides an accurate view of the forest of sub-projects. Benchmarks can be combined to track overall project-level progress. The approach is satisfying because each manager has local control over his or her deliverable, and can set benchmark goals independently. Overall management can still track progress towards the ultimate goal with confidence and precision.

Based upon my experience managing software projects in financial services, this work is never simple or easy. Perhaps in a very mature organization with concrete, well-understood standards it would be. Most of us do not have the privilege of working in such an organization. Using a step-by-step process, though, anyone can establish some consistent standards for a multiple-project effort. If these standards survive the effort, the overall manager can take pride; he or she has increased the maturity of the organization while completing the project.

A note on terminology: Some organizations talk about “programs” composed of multiple “projects”. Other organizations use “program” to mean a whole product line, while still others talk about “projects” composed of multiple “programs”. This paper will avoid the term “program” entirely. Instead “sub-projects” are parts of a “multiple-project effort” or an “overall project”. “Project manager” means any project manager, while “overall project manager” is someone running the overall project. “Sub-project manager” is someone running a sub-project.

Step 1: Determine Senior Management Needs

The target audience for these benchmarks and scorecards is almost always senior management. Most stakeholders are concerned with either day-to-day progress at a low level, or overall completion dates and cost. Management typically wants to see regular progress reports, to ensure that goals will be met.

A meeting is almost always the first step, to establish frequency of reporting, to gather a list of any key data they want to see, and to understand what questions they expect to have answered at each reporting period. At this stage, some reporting goals will be concrete, while some will be vague. Sample goals might be:

• “Weekly status reports by noon on Monday and a status meeting to discuss at 10 a.m. on Tuesday”
• “An e-mail if anything seems off, otherwise a monthly meeting”
• “Whatever numbers seem most important to you at the time, and of course any changes to delivery date”
• “An up-to-date earned value graph, and control graphs for defects found and fixed”

Almost any combination of information is possible at this point. Of course the overall project manager will have some ideas about metrics that he or she wants to capture, but this meeting is about senior management needs. The project manager can collect other data, so long as management’s needs are met.

Step 2: Plan Sub-Project Metrics

Each sub-project needs metrics to contribute to the overall scorecard. The sub-project should measure project results to show variance, problems, and progress. The sub-project manager should come up with ideas about the metrics that he or she wants to collect. He or she may collect traditional project management metrics like earned value metrics, budget vs. actuals, and schedule slippage. Domain-specific metrics are often also useful, like number of defects opened or closed, feet of wire laid, number of rooms painted, number of code-reviews completed, and so on. The possibilities are infinite. The choice of metrics will depend upon the sub-project manager’s experience with metrics as well as the domain. Some sub-project managers may even decide to collect no metrics of their own. At this point in the process, that is fine.

Step 3: Identify Common Metrics

Once the overall project manager has a list from each sub-project manager, he or she can look for common metrics. Sometimes two managers will call the same metric two different things, or calculate the same basic metric two different ways. For instance, a straight budget calculation of actual costs to-date will usually be greater than an earned-value calculation of actual costs, due to differences in definition of earned and unearned costs. Note areas of possible conflict as well as areas of agreement.

At this stage it is also critical to compare the sub-project manager’s metrics against the senior management needs. The metrics list may meet some management needs completely, while other management needs have no supporting metrics. For the next step, it is critical to analyze the results and develop a strategy to meet management needs.

Step 4: Negotiate with Sub-Project Managers

With a list of common metrics, conflicting metrics, and missing metrics, it is time to talk to each sub-project manager. There will always be some changes to negotiate at this phase. Changes at a sub-project level might include:

• changing data collection methods (i.e. use an earned-value budget or an accounting budget)
• adding more data elements to the scorecard
• renaming data elements
• changing data collection/reporting tools to achieve consistent results
• resolving differences in definitions (i.e. what does “50% complete” mean?)

The overall project manager may need to convince some sub-project managers to begin creating a scorecard at all. It is not uncommon for people to manage projects without a scorecard, and the main benefit of a common scorecard is to the overall project manager and to the overall management team. Some people may resist the standardization required.

Any sub-project manager who decided to collect no metrics in step #2 must now agree to collect a set of basic data, in order to participate in the overall project. These negotiations can be difficult. It is critical for the overall project manager to have a simple set of basic data needs, and to know exactly why each piece of data is necessary. Sometimes resistant managers fight metrics at every step of the way. The overall project manager should be prepared with advice or even practical help to begin data collection for each sub-project.

Be aware that gaining their cooperation or acceptance at this phase is not enough. Sometime resistant managers can misreport or report late during the project execution, as a form of passive resistance. The overall project manager needs to understand each sub-project manager; this negotiating period is a perfect time to get to know the overall project team really well. Wherever possible, try to generate goodwill and cooperation among the managers. Avoid creating situations where a single sub-project manager appears to win every negotiation, while everyone else looses.

Step 5: Set Standards

The overall project manager takes the results of all the negotiations and creates a list of standards for the overall project scorecard. The standards should describe who collects what data and when. Responsibilities for each sub-project should be clear and precise. Measurement methods should be exact. For instance, for “percent complete” for tasks, describe your method. Sample methods include:

• Either 0% or 100% – complete or not yet complete
• 0% if not started, 50% when started, 100% when complete
• (100* Act) / (Act + ETC)
• Different percentages for each phase of completion: 0% not started, 10% in-progress, 75% almost done, 90% ready for review, 100% passed review

Clear standards are critical for consistent reporting. Even the best-intentioned managers will vary in the way that they report information. It is impossible to have 100% consistency for some types of metrics, but a good standards document increases consistency better than any other single tool.

While setting standards, beware of vague standards that are hard to enforce or audit. Ideally, metrics are like the results of scientific experiments: anyone else could use the same methods, observe the same phenomena, and get the same results. The overall project manager should be able to verify the sub-project scorecards, at least within some range of certainty, like plus or minus 20%.

Networked tools for project scheduling, defect tracking, and work tracking simplify reporting considerably. Sometimes the sub-project manager does not even need to report their data to the overall project manager. If the data is networked, the overall project manager can read and report on the data independently. This kind of transparency is a huge benefit in multi-project situations.

Step 6: Create an Overall Project Scorecard

Scorecards can combine tables and graphs to show a snapshot of current status

With a standard document in hand, designing the scorecard is straightforward. A few key questions will drive formatting and layout:

• What information is most important?
• What order will we discuss this information at status meetings (if applicable)?
• How are we delivering the material (on-line, web, e-mail, text-only, print, projector, etc.)?
• What are the limits of the physical media (color, font size, etc.)?
• Do we need to highlight any key information?
• Do any readers need the data in a particular format (graphical, percentage, table of results, etc.)?
• What formatting or aesthetic standards should we follow?
• Is this report too expensive to deliver?

In general, a good scorecard flows clearly from one set of information to the next, in the order that management wants to see and discuss it. Data is clear, formatting does not get in the way, and it is easy to scan the page for critical information. Totals are obvious and meaningful.

Time and time again, I have found that management appreciates a one-page scorecard. Sometimes this single page just has summary data. Additional pages can always show the detail. A single-page summary makes it easy to see project status at a glance.

Step 7: Create Overall Benchmarks

Benchmarks show progress over time, measured against concrete goals. Color helps show that this project is falling behind, and is ready for new benchmarks.

Certain scorecard metrics will be expected to steadily increase or decrease over time. These metrics are good candidates for benchmarking. Imagine benchmarks as a snapshot of expected values over time, with one expected value for each reporting period. The most common form of benchmarking is earned value analysis. Managers take the schedule and budget at the start of the project, and write the expected earned value at each period throughout the project. Regular status reports show the actual values compared to the benchmarks.

The overall project manager can use this technique for any metric on the scorecard. Benchmarks can be extraordinarily useful for schedule-constrained projects. Management can see with every report whether work is on schedule, ahead or schedule, or behind schedule, without reviewing hundreds of tasks.

Some advice for creating benchmarks:

• Only benchmark values that change predictably over time: number of open defects might seem like a good value to benchmark, because it must be zero at the end, but if the value goes up and down every week, it may be a poor choice
• Benchmark a few values that really matter, that predict project success and show progress: too many benchmarks are confusing
• Estimate likely progress: take into account vacation time, project phases, and seasonal issues, instead of just assuming a linear progression, like “2 more each week”
• Publish the benchmarks: benchmarks are a great motivational tool for many teams, encouraging team members to contribute to the overall goals
• Use sub-project benchmarks: give sub-project managers control over their own benchmarks, and total their goals to set the overall project benchmarks

The overall project manager must understand the purpose of the benchmarks. If there is a prize for the team at each met-or-exceeded benchmark, then set the benchmark goals high. Perhaps the manager sets the benchmarks so that he or she is only 30% sure that the team will make it each week. Perhaps the benchmarks show project completion several weeks ahead of schedule. Communicate these expectations to management and to the teams.

Sometimes benchmarks are a simple monitoring tool. The overall project manager should then set the benchmark at a spot where he or she expects the team to perform. Estimating uncertainty in the estimate is very valuable. Benchmarks can serve as a form of control graph, showing whether results are in control or out of control. The classic control graph has a single result expected over time, with horizontal bars for average, high, and low values. The benchmark graph would have an expected-result line that moves up or down over time, with control bars above and below it. When using this approach, uncertainty usually varies over time, so the uncertainty levels may change with each reporting period. As each goal comes closer in time, the likelihood of reaching the goal becomes more and more certain.

Sub-project managers may or may not choose to create their own scorecards for each value. Over the life of a software project, one sub-project might occasionally find and fix one or two defects, while another sub-project finds and fixes a hundred defects per week. A defect scorecard and benchmark could be useful for the overall project and the second sub-project, but not for the first. Be flexible about applying metrics at the overall and sub-project level.

Step 8: Senior Management Approval

When developing any reporting tool, it is critical to get approval of its target audience. A good project manager will involve the audience throughout the process. When and how to involve senior management always varies by organization, but with a draft scorecard developed and benchmarks set, it is essential to present a proposal to management.

This meeting is typically quite formal. The overall project manager distributes the benchmarks and scorecard ahead of time, then reviews it step by step with management. A face-to-face meeting is recommended, even if the project communication calls for no face-to-face, regular status meetings. People need time to review and discuss the results. Different senior managers may need different information, and it helps them to hear each other defend each part of the report. Removing or changing a scorecard section requires everyone’s approval, and often requires discussion.

Usually there is at least one change needed. Often the organization or format of the document requires changes. Sometimes the benchmark goals are too aggressive, or not aggressive enough. Occasionally the metrics do not meet management needs, and the scorecard goes back to the drawing board. Usually the overall project manager can note the changes in a follow-up e-mail and begin using the new report for the next project-reporting period. Going through this process almost always produces a report that is better than the tools that came before it. Even if it is not perfect, management usually is ready to adopt it.

Step 9: Update Scorecards, Evaluate Actuals vs. Benchmarks

With each reporting period, metrics change. The overall project manager should set a reporting schedule with all sub-project managers, to meet the reporting schedule required by management. Having everyone report all data as of the same date is critical to having a useful overall project scorecard. Assembling and reporting data for a complex project often takes several days. Even after the sub-project managers have finished their reports, it may take a day for the overall project manager to file the overall report. A tightly organized team can improve that time, by meeting or beating reporting deadlines consistently. Data may be old by the time it is published; balance the competing needs for timely data against the needs for accurate, well-reviewed data.

Each sub-project manager may have different reporting responsibilities for each reporting period. Typical responsibilities include:

• Producing the sub-project scorecard and benchmarks
• Reviewing and approving data in shared databases
• Reporting additional data to the overall project manager, via e-mail or phone
• Meeting with the overall project manager to review and confirm project results

It is useful for the whole project-management team to get together and review the overall results, before publishing them to senior management. This step ensures that no one is surprised by the results in the report. The overall project manager can reconcile any conflicts between sub-project managers and can make last-minute corrections to the scorecards.

Benchmark values change every week. The overall project manager fills in the actuals for the week and compares them against the benchmarks. Any variances should have an explanation.

The overall project manager publishes the scorecards and benchmarks to management. The cycle repeats for every reporting period until the project is complete.

Step 10: Refine and Update

Projects are all unique endeavors, and an overall project is a collection of many unique endeavors. Even the best project manager will have trouble predicting the course of an overall project. Benchmarks are usually the first to change. Changes in staffing levels, unexpected project events (positive or negative), changes to scope, or changes to estimates all should trigger a review of benchmarks. Benchmarks loose their value as a management and motivation tool as soon as they become too easy or too difficult.

Often the project’s metrics change as the project goes through different phases. The overall project manager should monitor overall project activities, to see when a particular metric is no longer relevant or when a new one should be captured. The overall project manager may choose to design a new scorecard for each phase or to keep a single scorecard, but remove and add data elements as the project evolves. Either approach can be successful. Adding new metrics mean that the overall project manager should consider adding a new benchmark goal. Dropping old metrics sometimes means dropping a benchmark.

This step goes hand-in-hand with updating scorecards. It may be triggered anytime after reporting begins on a project. Critical events in the project almost always trigger it, including changes to scope, budget, schedule, or phase.

During project close-out, the use of the scorecards and their evolution through the project often make their way into the lessons learned for the project. The history of scorecards and benchmarks will help the managers of future projects develop their own templates and standards.

Putting Scorecards and Benchmarks to Use

Following these ten steps, a project manager can have an easier transition from managing stand-alone projects to managing projects that contain sub-projects. These techniques are broadly applicable, useful for a wide range of industries. There is some key recommendations for people who are new to these tools:

1. Keep the metrics simple and easy to understand
2. Make sure that sub-project metrics “scale up” to the overall project
3. Do not force metrics that work for one sub-project onto all sub-projects; only use what works
4. Watch your math, because not all metrics sum up through simple addition; standard deviation adds using squares, for instance – see recommendation #1 to avoid this problem entirely
5. Explicitly state the uncertainty of the numbers at the start; it is next to impossible to collect data completely accurately or to forecast benchmarks perfectly
6. When benchmark values and actual values diverge, do not panic; figure out if the project is off-course or if the metrics are off-course and take corrective action
7. Never, ever adjust the scorecard numbers in order to appear to meet goals; not only is it unethical behavior, it is increasing the projects’ list of problems and threatening the bond of trust with project sponsors
8. Do adjust metric calculations and benchmarks as needed; do so openly, with the consent of senior managers and sub-project managers
9. Use the literature to select appropriate metrics; academics have studied and argued over appropriate metrics for decades, and it is expensive to ignore their advice
10. Only use sophisticated metrics after gaining experience with simpler metrics, and only if the audience of the reports understand the complex metrics thoroughly.

To summarize: FOCUS ON COMMUNICATION. These reports are a tool to communicate progress and status. Be creative, using color, graphs, charts, tables, pictures, or anything that helps communicate. An effective report will generate complete silence around a meeting-room table, followed by grunts of surprise, grunts of “humph,” exclamations of “wow!” and finally cries, “This is great,” or “NOW I understand.” The measure of effectiveness for a scorecard or benchmark is its communication value. It is not enough to have numbers that accurately reflect project progress, they must also communicate that progress to the intended audience. Perhaps the ultimate compliment is when senior management starts telling your peers to use benchmarks and scorecards, and your peers come to you as the in-house expert for advice.

Ultimately the scorecards and benchmarks are about seeing the forest, despite all the trees. Managers who remain focused on communicating the state of the project “forest” will not get lost in the “trees” of numbers in their reports. Giving senior management the right level of vision, supported by accurate, demonstrable facts, is the ultimate goal of the exercise.

Download Article

Printer-friendly
(PDF)

Download Slides, light

“benchscore-light-ppt.pdf”>Printer-friendly (PDF) “benchscore-light.ppt”>MS PowerPoint

Download Slides, dark

Printer-friendly
(PDF) MS PowerPoint

Download Sample Scorecard

MS Excel
(XLS)

Advanced TinyMCE Editor is a little better. At least I get to keep my DIV tags. Unfortunately it mangles any tag that is broken across a line! It also still has the extra BR problems.

Two versions of this paper are available. The “benchscore-short.html”>short version was prepared for a newsletter and
is approximately 2,000 words, or six pages. The long version (below) was
prepared for a 45-minute presentation, contains some additional details, and
runs approximately 3,800 words, or ten pages. Feel free to read either or
both versions.

One of the best pieces of advice for any project manager is, “Divide and
conquer.” Take a complex, large deliverable, and break it into parts. Deliver
it in small pieces, each of which can be managed separately, with as few
inter-dependencies as possible. This strategy is an absolute requirement at
times. Some efforts are so large that a single project manager cannot
reasonably track and manage the entire effort alone. Some efforts are so
complex that they need multiple managers, each with different specialized
knowledge. Thank goodness for “divide and conquer.”

There is a dark side to “divide and conquer”, however. At some point
senior management will ask, “But how are we doing overall?” The individual
status reports from each sub-project may not show the progress towards the
overall goal. Cross-team problems may be ignored. Multiple formats from
multiple authors can be confusing to read.

People want to see the forest, not just the trees. Creating an overall
status report is a possible solution.

One approach to an overall status report is simple:

• collect text descriptions of status and goals from the manager of each
  sub-project,
• submit them all to a single person
• concatenate them into a single status report, editing the result for
  consistent format and language

This overall report can be extraordinarily helpful, to senior managers and
sub-project managers alike. Sometimes it is enough, but a project manager
should have additional tools in his or her toolbox.

A second approach is to create an overall schedule for the effort. These
schedules can be as simple as a milestone-only schedule, with key sub-project
deliverables tracked on a single plan. Dependencies and cross-team milestones
can be represented. More complex versions will have a single master schedule
that contains all tasks for all sub-projects, including actual and estimated
work hours. Scheduling software can help automate creating these master
schedules. Creating schedules is often a tool-specific challenge, and in my
experience there are easier ways to get clearer information to senior
management.

“Two sub-projects contribute paper reports and one project contributes a phone-in report. All feed into the overall project report.” />
Each Sub-Project Contributes to the Overall Scorecard

Any project can use benchmarks and scorecards, but they are especially
useful for projects composed of sub-projects. Each sub-project manager can
create a custom scorecard containing critical, relevant data for their
effort. Often a useful scorecard can be just a single printed page. Team
members can use the scorecards to track progress, gauging concrete results
against benchmark numbers. The overall project manager can then collect these
scorecards, combine the relevant data from each, and produce an overall
scorecard. Usually it does not capture all the metrics of all the
sub-projects, but it provides an accurate view of the forest of sub-projects.
Benchmarks can be combined to track overall project-level progress. The
approach is satisfying because each manager has local control over his or her
deliverable, and can set benchmark goals independently. Overall management
can still track progress towards the ultimate goal with confidence and
precision.

Based upon my experience managing software projects in financial services,
this work is never simple or easy. Perhaps in a very mature organization with
concrete, well-understood standards it would be. Most of us do not have the
privilege of working in such an organization. Using a step-by-step process,
though, anyone can establish some consistent standards for a multiple-project
effort. If these standards survive the effort, the overall manager can take
pride; he or she has increased the maturity of the organization while
completing the project.

A note on terminology: Some organizations talk about
“programs” composed of multiple “projects”. Other organizations use “program”
to mean a whole product line, while still others talk about “projects”
composed of multiple “programs”. This paper will avoid the term “program”
entirely. Instead “sub-projects” are parts of a “multiple-project effort” or
an “overall project”. “Project manager” means any project manager, while
“overall project manager” is someone running the overall project.
“Sub-project manager” is someone running a sub-project.

Step 1: Determine Senior Management Needs

The target audience for these benchmarks and scorecards is almost always
senior management. Most stakeholders are concerned with either day-to-day
progress at a low level, or overall completion dates and cost. Management
typically wants to see regular progress reports, to ensure that goals will be
met.

A meeting is almost always the first step, to establish frequency of
reporting, to gather a list of any key data they want to see, and to
understand what questions they expect to have answered at each reporting
period. At this stage, some reporting goals will be concrete, while some will
be vague. Sample goals might be:

• “Weekly status reports by noon on Monday and a status meeting to
  discuss at 10 a.m. on Tuesday”
• “An e-mail if anything seems off, otherwise a monthly meeting”
• “Whatever numbers seem most important to you at the time, and of course
  any changes to delivery date”
• “An up-to-date earned value graph, and control graphs for defects found
  and fixed”

Almost any combination of information is possible at this point. Of course
the overall project manager will have some ideas about metrics that he or she
wants to capture, but this meeting is about senior management needs. The
project manager can collect other data, so long as management’s needs are
met.

Step 2: Plan Sub-Project Metrics

Each sub-project needs metrics to contribute to the overall scorecard. The
sub-project should measure project results to show variance, problems, and
progress. The sub-project manager should come up with ideas about the metrics
that he or she wants to collect. He or she may collect traditional project
management metrics like earned value metrics, budget vs. actuals, and
schedule slippage. Domain-specific metrics are often also useful, like number
of defects opened or closed, feet of wire laid, number of rooms painted,
number of code-reviews completed, and so on. The possibilities are infinite.
The choice of metrics will depend upon the sub-project manager’s experience
with metrics as well as the domain. Some sub-project managers may even decide
to collect no metrics of their own. At this point in the process, that is
fine.

Step 3: Identify Common Metrics

Once the overall project manager has a list from each sub-project manager,
he or she can look for common metrics. Sometimes two managers will call the
same metric two different things, or calculate the same basic metric two
different ways. For instance, a straight budget calculation of actual costs
to-date will usually be greater than an earned-value calculation of actual
costs, due to differences in definition of earned and unearned costs. Note
areas of possible conflict as well as areas of agreement.

At this stage it is also critical to compare the sub-project manager’s
metrics against the senior management needs. The metrics list may meet some
management needs completely, while other management needs have no supporting
metrics. For the next step, it is critical to analyze the results and develop
a strategy to meet management needs.

Step 4: Negotiate with Sub-Project Managers

With a list of common metrics, conflicting metrics, and missing metrics,
it is time to talk to each sub-project manager. There will always be some
changes to negotiate at this phase. Changes at a sub-project level might
include:

• changing data collection methods (i.e. use an earned-value budget or an
  accounting budget)
• adding more data elements to the scorecard
• renaming data elements
• changing data collection/reporting tools to achieve consistent
  results
• resolving differences in definitions (i.e. what does “50% complete”
  mean?)

The overall project manager may need to convince some sub-project managers
to begin creating a scorecard at all. It is not uncommon for people to manage
projects without a scorecard, and the main benefit of a common scorecard is
to the overall project manager and to the overall management team. Some
people may resist the standardization required.

Any sub-project manager who decided to collect no metrics in step #2 must
now agree to collect a set of basic data, in order to participate in the
overall project. These negotiations can be difficult. It is critical for the
overall project manager to have a simple set of basic data needs, and to know
exactly why each piece of data is necessary. Sometimes resistant managers
fight metrics at every step of the way. The overall project manager should be
prepared with advice or even practical help to begin data collection for each
sub-project.

Be aware that gaining their cooperation or acceptance at this phase is not
enough. Sometime resistant managers can misreport or report late during the
project execution, as a form of passive resistance. The overall project
manager needs to understand each sub-project manager; this negotiating period
is a perfect time to get to know the overall project team really well.
Wherever possible, try to generate goodwill and cooperation among the
managers. Avoid creating situations where a single sub-project manager
appears to win every negotiation, while everyone else looses.

Step 5: Set Standards

The overall project manager takes the results of all the negotiations and
creates a list of standards for the overall project scorecard. The standards
should describe who collects what data and when. Responsibilities for each
sub-project should be clear and precise. Measurement methods should be exact.
For instance, for “percent complete” for tasks, describe your method. Sample
methods include:

• Either 0% or 100% – complete or not yet complete
• 0% if not started, 50% when started, 100% when complete
• (100* Act) / (Act + ETC)
• Different percentages for each phase of completion: 0% not started, 10%
  in-progress, 75% almost done, 90% ready for review, 100% passed review

Clear standards are critical for consistent reporting. Even the
best-intentioned managers will vary in the way that they report information.
It is impossible to have 100% consistency for some types of metrics, but a
good standards document increases consistency better than any other single
tool.

While setting standards, beware of vague standards that are hard to
enforce or audit. Ideally, metrics are like the results of scientific
experiments: anyone else could use the same methods, observe the same
phenomena, and get the same results. The overall project manager should be
able to verify the sub-project scorecards, at least within some range of
certainty, like plus or minus 20%.

Networked tools for project scheduling, defect tracking, and work tracking
simplify reporting considerably. Sometimes the sub-project manager does not
even need to report their data to the overall project manager. If the data is
networked, the overall project manager can read and report on the data
independently. This kind of transparency is a huge benefit in multi-project
situations.

Step 6: Create an Overall Project Scorecard

“A scorecard showing counters for UIs, Classes, and Interfaces. Shows number of items coded, tested, required and remaining, with totals. Data appears in a table and a bar graph.” />
Scorecards can combine tables and graphs to show a snapshot of current
status

With a standard document in hand, designing the scorecard is
straightforward. A few key questions will drive formatting and layout:

• What information is most important?
• What order will we discuss this information at status meetings (if
  applicable)?
• How are we delivering the material (on-line, web, e-mail, text-only,
  print, projector, etc.)?
• What are the limits of the physical media (color, font size,
  etc.)?
• Do we need to highlight any key information?
• Do any readers need the data in a particular format (graphical,
  percentage, table of results, etc.)?
• What formatting or aesthetic standards should we follow?
• Is this report too expensive to deliver?

In general, a good scorecard flows clearly from one set of information to
the next, in the order that management wants to see and discuss it. Data is
clear, formatting does not get in the way, and it is easy to scan the page
for critical information. Totals are obvious and meaningful.

Time and time again, I have found that management appreciates a one-page
scorecard. Sometimes this single page just has summary data. Additional pages
can always show the detail. A single-page summary makes it easy to see
project status at a glance.

Step 7: Create Overall Benchmarks

“A spreadsheet shows goals vs. actuals on a monthly basis for classes, UIs, and interfaces. Goals extend into the future, beyond the actual values. Below each actual is a variance, highlighted in green if it is positive, red if negative.” />
Benchmarks show progress over time, measured against concrete goals.
Color helps show that this project is falling behind, and is ready for
new benchmarks.

Certain scorecard metrics will be expected to steadily increase or
decrease over time. These metrics are good candidates for benchmarking.
Imagine benchmarks as a snapshot of expected values over time, with one
expected value for each reporting period. The most common form of
benchmarking is earned value analysis. Managers take the schedule and
budget at the start of the project, and write the expected earned value at
each period throughout the project. Regular status reports show the actual
values compared to the benchmarks.

The overall project manager can use this technique for any metric on the
scorecard. Benchmarks can be extraordinarily useful for schedule-constrained
projects. Management can see with every report whether work is on schedule,
ahead or schedule, or behind schedule, without reviewing hundreds of
tasks.

Some advice for creating benchmarks:

• Only benchmark values that change predictably over time: number of open
  defects might seem like a good value to benchmark, because it must be zero
  at the end, but if the value goes up and down every week, it may be a poor
  choice
• Benchmark a few values that really matter, that predict project success
  and show progress: too many benchmarks are confusing
• Estimate likely progress: take into account vacation time, project
  phases, and seasonal issues, instead of just assuming a linear progression,
  like “2 more each week”
• Publish the benchmarks: benchmarks are a great motivational tool for
  many teams, encouraging team members to contribute to the overall
  goals
• Use sub-project benchmarks: give sub-project managers control over
  their own benchmarks, and total their goals to set the overall project
  benchmarks

The overall project manager must understand the purpose of the benchmarks.
If there is a prize for the team at each met-or-exceeded benchmark, then set
the benchmark goals high. Perhaps the manager sets the benchmarks so that he
or she is only 30% sure that the team will make it each week. Perhaps the
benchmarks show project completion several weeks ahead of schedule.
Communicate these expectations to management and to the teams.

Sometimes benchmarks are a simple monitoring tool. The overall project
manager should then set the benchmark at a spot where he or she expects the
team to perform. Estimating uncertainty in the estimate is very valuable.
Benchmarks can serve as a form of control graph, showing whether results are
in control or out of control. The classic control graph has a single result
expected over time, with horizontal bars for average, high, and low values.
The benchmark graph would have an expected-result line that moves up or down
over time, with control bars above and below it. When using this approach,
uncertainty usually varies over time, so the uncertainty levels may change
with each reporting period. As each goal comes closer in time, the likelihood
of reaching the goal becomes more and more certain.

Sub-project managers may or may not choose to create their own scorecards
for each value. Over the life of a software project, one sub-project might
occasionally find and fix one or two defects, while another sub-project finds
and fixes a hundred defects per week. A defect scorecard and benchmark could
be useful for the overall project and the second sub-project, but not for the
first. Be flexible about applying metrics at the overall and sub-project
level.

Step 8: Senior Management Approval

When developing any reporting tool, it is critical to get approval of its
target audience. A good project manager will involve the audience throughout
the process. When and how to involve senior management always varies by
organization, but with a draft scorecard developed and benchmarks set, it is
essential to present a proposal to management.

This meeting is typically quite formal. The overall project manager
distributes the benchmarks and scorecard ahead of time, then reviews it step
by step with management. A face-to-face meeting is recommended, even if the
project communication calls for no face-to-face, regular status meetings.
People need time to review and discuss the results. Different senior managers
may need different information, and it helps them to hear each other defend
each part of the report. Removing or changing a scorecard section requires
everyone’s approval, and often requires discussion.

Usually there is at least one change needed. Often the organization or
format of the document requires changes. Sometimes the benchmark goals are
too aggressive, or not aggressive enough. Occasionally the metrics do not
meet management needs, and the scorecard goes back to the drawing board.
Usually the overall project manager can note the changes in a follow-up
e-mail and begin using the new report for the next project-reporting period.
Going through this process almost always produces a report that is better
than the tools that came before it. Even if it is not perfect, management
usually is ready to adopt it.

Step 9: Update Scorecards, Evaluate Actuals vs. Benchmarks

With each reporting period, metrics change. The overall project manager
should set a reporting schedule with all sub-project managers, to meet the
reporting schedule required by management. Having everyone report all data as
of the same date is critical to having a useful overall project scorecard.
Assembling and reporting data for a complex project often takes several days.
Even after the sub-project managers have finished their reports, it may take
a day for the overall project manager to file the overall report. A tightly
organized team can improve that time, by meeting or beating reporting
deadlines consistently. Data may be old by the time it is published; balance
the competing needs for timely data against the needs for accurate,
well-reviewed data.

Each sub-project manager may have different reporting responsibilities for
each reporting period. Typical responsibilities include:

• Producing the sub-project scorecard and benchmarks
• Reviewing and approving data in shared databases
• Reporting additional data to the overall project manager, via e-mail or
  phone
• Meeting with the overall project manager to review and confirm project
  results

It is useful for the whole project-management team to get together and
review the overall results, before publishing them to senior management. This
step ensures that no one is surprised by the results in the report. The
overall project manager can reconcile any conflicts between sub-project
managers and can make last-minute corrections to the scorecards.

Benchmark values change every week. The overall project manager fills in
the actuals for the week and compares them against the benchmarks. Any
variances should have an explanation.

The overall project manager publishes the scorecards and benchmarks to
management. The cycle repeats for every reporting period until the project is
complete.

Step 10: Refine and Update

Projects are all unique endeavors, and an overall project is a collection
of many unique endeavors. Even the best project manager will have trouble
predicting the course of an overall project. Benchmarks are usually the first
to change. Changes in staffing levels, unexpected project events (positive or
negative), changes to scope, or changes to estimates all should trigger a
review of benchmarks. Benchmarks loose their value as a management and
motivation tool as soon as they become too easy or too difficult.

Often the project’s metrics change as the project goes through different
phases. The overall project manager should monitor overall project
activities, to see when a particular metric is no longer relevant or when a
new one should be captured. The overall project manager may choose to design
a new scorecard for each phase or to keep a single scorecard, but remove and
add data elements as the project evolves. Either approach can be successful.
Adding new metrics mean that the overall project manager should consider
adding a new benchmark goal. Dropping old metrics sometimes means dropping a
benchmark.

This step goes hand-in-hand with updating scorecards. It may be triggered
anytime after reporting begins on a project. Critical events in the project
almost always trigger it, including changes to scope, budget, schedule, or
phase.

During project close-out, the use of the scorecards and their evolution
through the project often make their way into the lessons learned for the
project. The history of scorecards and benchmarks will help the managers of
future projects develop their own templates and standards.

Putting Scorecards and Benchmarks to Use

Following these ten steps, a project manager can have an easier transition
from managing stand-alone projects to managing projects that contain
sub-projects. These techniques are broadly applicable, useful for a wide
range of industries. There is some key recommendations for people who are new
to these tools:

1. Keep the metrics simple and easy to understand
2. Make sure that sub-project metrics “scale up” to the overall
   project
3. Do not force metrics that work for one sub-project onto all
   sub-projects; only use what works
4. Watch your math, because not all metrics sum up through simple
   addition; standard deviation adds using squares, for instance – see
   recommendation #1 to avoid this problem entirely
5. Explicitly state the uncertainty of the numbers at the start; it is
   next to impossible to collect data completely accurately or to forecast
   benchmarks perfectly
6. When benchmark values and actual values diverge, do not panic; figure
   out if the project is off-course or if the metrics are off-course and take
   corrective action
7. Never, ever adjust the scorecard numbers in order to appear to meet
   goals; not only is it unethical behavior, it is increasing the projects’
   list of problems and threatening the bond of trust with project
   sponsors
8. Do adjust metric calculations and benchmarks as needed; do so openly,
   with the consent of senior managers and sub-project managers
9. Use the literature to select appropriate metrics; academics have
   studied and argued over appropriate metrics for decades, and it is
   expensive to ignore their advice
10. Only use sophisticated metrics after gaining experience with simpler
    metrics, and only if the audience of the reports understand the complex
    metrics thoroughly.

To summarize: FOCUS ON COMMUNICATION. These reports are a tool to
communicate progress and status. Be creative, using color, graphs, charts,
tables, pictures, or anything that helps communicate. An effective report
will generate complete silence around a meeting-room table, followed by
grunts of surprise, grunts of “humph,” exclamations of “wow!” and finally
cries, “This is great,” or “NOW I understand.” The measure of effectiveness
for a scorecard or benchmark is its communication value. It is not enough to
have numbers that accurately reflect project progress, they must also
communicate that progress to the intended audience. Perhaps the ultimate
compliment is when senior management starts telling your peers to use
benchmarks and scorecards, and your peers come to you as the in-house expert
for advice.

Ultimately the scorecards and benchmarks are about seeing the forest,
despite all the trees. Managers who remain focused on communicating the state
of the project “forest” will not get lost in the “trees” of numbers in their
reports. Giving senior management the right level of vision, supported by
accurate, demonstrable facts, is the ultimate goal of the exercise.

Download

Printer-friendly article
(PDF)

Adding all sorts of extra line-break tags. Should we disable wpautop? Can we do it post-by-post somehow? Would be good to leave it on for comments, at least. Also consider installing the Advanced TinyMCE editor plug-in, to add table support, pull in styles from the stylesheet automatically, and to stop converting DIV to P.

Red-Yellow-Green (R/Y/G) reports have become incredibly popular as a way
to quickly show the status of a project or a group of projects. An executive
or senior decision-maker can quickly scan a row or column of colored
indicators and get an overview of status.

One Company’s Approach

Each company may take a different approach to these reports. We will
examine one company as a sample, before reviewing some general
principles.

The report fits on one side of a letter-sized paper. It summarizes the
status of between 15 and 25 projects. Each project gets a row, and each row
has room for four indicators and an explanation for each. The four R/Y/G
indicators are for Planning, Schedule, Budget, and Objectives.

Each indicator has a threshold for each color rating. Many indicators are
tied to our policies and procedures. For instance, budgets with variances
above a certain dollar amount or percentage need to be explained or revised.
Budget indicators become Yellow when a budget is expected to be close to or
over that threshold, and Red when it is definitely going to go over and
action needs to be taken.

The schedule indicator is driven by the number of weeks delay and number
of times the target dates have been revised in the last month. Yellow means
that schedule is at risk for more than two weeks delay or that there has been
a target date changed within the last month. Red means that either the
schedule is at risk for more than four weeks delay, the project end-date is
“TBD”, or the target date has been changed two or more times in the last
month.

Some indicators have guidelines but require individual judgment to come up
with a final decision. Sometimes a Yellow might mean that an objective or a
date is “at risk”. Not all indicators can be based on solid
numerical data and thresholds.

Setting Standards for Your Company

Each of these definitions is based on how one particular company runs its
projects. Each indicator serves a specific need of the senior executive team.
I created a draft report and draft definitions for each indicator, and the
executive team reviewed, modified, and eventually approved the new report
format.

To adopt a new report format successfully at your company, look at what
information does and does not matter to your decision-makers. Define Red,
Yellow, and Green so that they will drive people’s interest and attention.
Isolate the factors that really show project health and project problems.
Design indicators that will dive executives to make decisions and to take
action.

An internal auditor reviewed my report recently and noted that certain
indicators were almost always Green. The report was changed to show other
information that would be different for each project, to show data that
helped people make decisions. Ensure that the indicators you choose will be
useful and relevant. Chart what they would have shown over the past few
reporting periods to see when and how they change.

Remember to always be prepared to change the reports. As an organization
matures, some metrics may become more predictable and under better control. A
Red-Yellow-Green report needs to show differences and variation. Change the
reports as your organization improves its processes.

A Few Tips and Tricks

Use
shapes as well as color for R/Y/G

Here are some other tips to help when designing a report:

• Keep it to one page if at all possible
• Make the list easy to scan (plenty of white space, even columns and
  rows)
• Consider using shapes or text as well as color, in case people must
  print on a black-and-white printer or in case people are color blind (try a
  stop-sign for Red, a yield sign for Yellow, and a diamond for Green)
• Provide room for a text explanation of critical issues and
  indicators
• Focus on the Red and Yellow projects, but not exclusively
• Spend time on projects that are consistently Green, to find best
  practices and to make sure that the team’s status report is truly
  accurate
• Make public the standards and guidelines used to set the indicators, so
  that people know how projects are being judged
• Involve the most senior people who will receive these reports in the
  report design process, to ensure it meets their needs
• Provide drill-down reports to explain the details behind each indicator
  and to avoid the need to clutter the main report with details

Related Articles

Of Benchmarks and
Scorecards: Reporting on Multiple Projects