The Project Manager's Corner: A Big Test—Our Conceptual Design Review
Gary Sanders, TMT Project Manager

May 2006

TMT has had its long awaited Conceptual Design Review.

No matter how good you are, no matter how experienced you are, no matter how hard you work or reflect and question what you are doing and deciding, there is no substitute for getting quizzed and scrutinized and probed by an outside group of experts. These experts must not be members of the project, or advocates for the project. They must be objective. They should have a sense that, by listening and studying and pondering and telling you what they really think, they would improve the project.

TMT just went through this very important review process. The result is a strong positive assessment of the TMT design and plan, and a long list of suggestions that will help TMT in the next steps.

Let me tell you a story about reviews. Seventeen years ago I was the Project Manager for a detector at the ill-fated Superconducting Super Collider. I had made a few trips to a collaborating institute just to the east of East Berlin, in the German Democratic Republic (DDR), as it was known then. Science has always been very international. On one trip in late 1989, I passed through the checkpoint in the famous Berlin Wall. It was the very last day that such checks would be performed. The Wall was about to come down and the guard knew it and I knew it. The stamp in my passport looked like a museum piece the moment it was entered.

I traveled to this institute and met with a very senior leader of the institute to discuss technical matters. We had a good visit and then discussed the uncertainty ahead as the parent Academy of Sciences of the DDR was about to be disbanded. He seemed anxious. Unknown new events lay ahead.

A month or two later, I returned. My car passed over a trench that once was the Wall. My colleague greeted me at the door of the institute. Coal smoke was gone. The air was strangely clear. Already a new gas furnace had been installed to replace the old smelly coal burning plant. The West had arrived. My colleague was happy.

But he had a concern. He ushered me into his office to ask me a private question. He told me that a blue-ribbon committee had decided to make his institute a branch of the West German high-energy physics laboratory in Hamburg. I told him that this was good news, as I had worked at that institution and it was a good laboratory to be part of. He looked anxiously at me and told me that he had been visited by the director of that lab. He said he needed to ask my advice on an important event ahead.

He seemed very anxious. He whispered to me, "Tell me, what is a review?" As terrible as an authoritarian regime had been, this eminent scientific and technical institution had never been questioned, never scrutinized, never measured for excellence.

Even for those of us who have been through several major research or construction projects, who have weathered many reviews and learned that they introduce new perspectives and show you what you have missed or accepted too easily, a review brings anxious moments. Nothing good comes easily!

Reviews are sometimes used as ceremonies. A project that has been reviewed has a box checked off on some imaginary checklist. Announcements are made after reviews. Press releases are issued. Articles like this are written in positive and upbeat tones. Compared to uncovering real insights in to the project, this may seem superficial. But a major project is a public happening. An astronomy project captures the public imagination. It also captures and expends precious public and private resources. As a matter of fulfilling the public and private trust, such a project should stand up and present its work and challenges to open scrutiny. Like a wedding or swearing in, a public review is a ceremonial milestone acknowledging the interest and trust of the supporters and adherents of a project. When generally successful, they are a needed ceremony and a moment of celebration.

But we are talking about an engineering megaproject with serious astronomy goals. Ceremony is hardly what we should dwell upon. More appropriate, a proper review is an essential process to assure the quality of the work. It is a mechanism for improving the outcome. It is insurance against many kinds of failures and a way to see the project's work through fresh eyes.

The value of a review begins the moment that it is defined and scheduled. The date is now known and the members of the project team recognize it as an important test. Work gets organized to assure that important things are done by the review.

The charge to the review panel, the list of questions asked of them, is defined and that list immediately focuses the work of the project team. In fact, the act of writing down the key questions that the review is supposed to address is, by itself, an act of crucial planning by the project leaders. It becomes a strategic goal to pass that review and to address the posed questions. From top to bottom in the project, the charge becomes a definition of success.

The review panel members are selected. To be effective, the reviewers must be true and respected experts. For the review results to be taken seriously at all levels, the professional reputations of the reviewers must be very high. The panel member identities set a standard, a blue-ribbon standard.

The agenda of the review defines the nature of the test. Is it a short one day set of top level presentations done in a public forum? Or is the review spread over several days and does it include small working groups asking detailed questions of the individual team experts?

In designing a project, there are several expected reviews. These are sometimes called Conceptual, Preliminary and Final Design Reviews. They are turned into acronyms, CoDR, PDR, FDR respectively. Engineers let these bits of alphabet soup roll off their tongues in knowing agreement. They know what is meant by each. (Some fields and agencies use slightly different labels, like Critical Review for the Final Review, or Phase A, B, C, etc. But they are all similar to what we describe.)

The CoDR (the one we just had in TMT) is held at the point that the science requirements have been developed into technical requirements on the system, and that one representative design has been studied and defined well enough to see if it can meet the requirements. This means that the design can accomplish the project goals. It is a proof that there is a possible solution. There will be remaining technical issues or design choices ahead, and many details to resolve. The CoDR is held to test the technical requirements and one carefully considered design that promises to meet the requirements.

Once that review is held, the project team has advice on some remaining issues or design alternates to consider. Work proceeds until that single representative design is now fully developed. This means that all of the remaining major design choices and open issues are resolved and the design is now the actual intended design to be constructed. At this point, it has to be possible to argue that the design will meet the requirements. After the PDR is held, the project may receive some remaining homework to complete. After finishing those tasks, the general step is to progress with the specific reviewed design.

A Final Design Review tests whether the project is ready to translate its design into pieces of hardware and purchased items and final software and so on. That is generally the final hurdle before committing to "bending metal" in the words of old time engineers.

TMT just passed its CoDR. This review was very complete. It had a very detailed charge, a blue-ribbon panel that was selected from institutions around the world and it lasted for four days, in which the panel worked in small groups with team members digging into what the project had done. The result is that this respected and independent panel of reviewers pronounced:

"This is a well scoped project, technically challenging, yet within reach. It will enable a new era in astronomy that is seductive and highly motivating."

"With respect to the CoDR charge, the telescope requirements are appropriate for the CoDR stage. The conceptual designs have been extensively developed, with some designs closer to the PDR level. The state of development allowed a good risk assessment, and most of subsystem presentations included a realistic assessment of the risks."

"The TMT conceptual design process has produced an excellent conceptual design for a first light adaptive optics (AO) capability and a comprehensive plan for incorporating AO into future generations of instruments to meet a wide range of science requirements. The philosophy of operating TMT at first light with adaptive optics is highly commendable. The Panel has relatively few findings and recommendations for improving the probability of success of AO at first light."

These are heartening words. But I must confess that there is a significant list of "you should consider the following…" and "more work needs to be done on…" and "a good start has been made on…but…".

All in all, we have all learned a great deal. I told you that a review makes a difference from the moment is scheduled. And the review findings also give specific advice and quality assurance. After the review, however, there is one more positive effect. The team, from top to bottom, has seen what it has done, in one setting, through its own eyes and through the insight of outside reviewers. This is an enormously powerful team building experience and it sets the stage for the next phase of work. Onward.

I wonder how my friends in that institute in the eastern part of Germany have fared. How did their reviews work out over the years?

TMT Project Manager Gary Sanders addresses the audience at the project's Conceptual Design Review in Pasadena, CA, on May 8, 2006.