*************************************************************** **This document is an ASCII formatted version of a printed ** **document. The page numbers in this eletronic version may ** **not be in the same order as the printed document. The ** **printed document may also contain charts and photographs ** **which are not reproduced in this electronic version. If ** **you reqyure the printed version of this document, contact ** **the Office of Inspector General (IG-1), Department of ** **Energy, 1000 Independence Avenue, SW, Washington, D.C. ** **20585 or call Wilma Slaughter at (202) 586D1924. ** *************************************************************** May 19, 1995 IG-1 INFORMATION: "Audit of Administration of Cooperative Research and Development Agreements at DOE National Laboratories" The Secretary BACKGROUND: The Department of Energy (DOE) established policies to ensure that Cooperative Research and Development Agreements (CRADAs) enhance U.S. competitiveness in the world economy, provide a reasonable return on resources invested, and enable successful commercialization of technologies developed. DOE's Office of Technology Partnerships issued the General Guidance Memorandum to DOE operations offices establishing policy goals for technology transfer programs, including CRADAs. DISCUSSION: We found that the efforts to manage CRADAs at three DOE national laboratories (Los Alamos, Oak Ridge, and Lawrence Livermore) did not fully achieve DOE's policy goals outlined in the General Guidance Memorandum. Specifically, the audit disclosed that: (1) joint work statements did not always contain clearly defined information that allowed DOE to facilitate technology transfer or to evaluate a CRADA's potential benefits; (2) CRADA statements of work did not always contain adequate documentation or address potential benefits; (3) the three national laboratories did not have effective mechanisms for continuous self-appraisal or measures of overall program success; and (4) CRADA provisions did not exist to ensure an accurate valuation of partner contributions. Additionally, we concluded that the cooperative efforts envisioned in the CRADA concept have the potential to greatly benefit the U.S. economy. However, the CRADA concept could be undermined if the Department does not implement appropriate management techniques and performance measurement mechanisms to determine the viability of its CRADAs. We believe that such techniques and mechanisms can be implemented in a manner that will not inhibit commercial sector interaction with the Department's national laboratories. -2- We discussed this report with the Associate Deputy Under Secretary for Operations, Office of Technology Partnerships. Management partially concurred with the finding and recommendations in the report. John C. Layton Inspector General Attachment cc: Deputy Secretary Under Secretary Associate Deputy Under Secretary for Operations, Office of Technology Partnerships U.S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL AUDIT OF ADMINISTRATION OF COOPERATIVE RESEARCH AND DEVELOPMENT AGREEMENTS AT DOE NATIONAL LABORATORIES The Office of Audit Services wants to make the distribution of its audit reports as customer friendly and cost effective as possible. Therefore, we are making this report available electronically through the Internet at the following alternative addresses: Department of Energy Headquarters Gopher gopher.hr.doe.gov Department of Energy Headquarters Anonymous FTP vm1.hqadmin.doe.gov We are experimenting with various options to facilitate audit report distribution. Your comments would be appreciated and can be provided on the Customer Comment Form attached to the Audit Report. Report Number: DOE/IG-0373 Western Regional Audit Office Date of Issue: May 19, 1995 Albuquerque, New Mexico 87185 AUDIT OF ADMINISTRATION OF COOPERATIVE RESEARCH AND DEVELOPMENT AGREEMENTS AT DOE NATIONAL LABORATORIES TABLE OF CONTENTS Page SUMMARY ........................................ 1 PART I - APPROACH AND OVERVIEW .......................... 2 Introduction ................................... 2 Scope and Methodology .......................... 2 Background ..................................... 3 Observations and Conclusions ................... 5 PART II - FINDING AND RECOMMENDATIONS .................... 8 Administration of Cooperative Research and Development Agreements at DOE National Laboratories.................................... 8 PART III - MANAGEMENT AND AUDITOR COMMENTS ................ 16 U.S. DEPARTMENT OF ENERGY OFFICE OF INSPECTOR GENERAL OFFICE OF AUDIT SERVICES AUDIT OF ADMINISTRATION OF COOPERATIVE RESEARCH AND DEVELOPMENT AGREEMENTS AT DOE NATIONAL LABORATORIES Audit Report Number: DOE/IG-0373 SUMMARY The Department of Energy (DOE) established policies to ensure that Cooperative Research and Development Agreements (CRADAs) enhance U.S. competitiveness in the world economy, provide a reasonable return on resources invested, and enable successful commercialization of technologies developed. DOE's Office of Technology Partnerships issued a General Guidance Memorandum to DOE operations offices establishing policy goals for technology transfer programs, including CRADAs. Our audit disclosed that efforts to manage CRADAs at three national laboratories did not fully achieve DOE's policy goals outlined in the General Guidance Memorandum. Specifically, the audit showed that: (1) joint work statements did not always contain clearly defined information that allowed DOE to facilitate technology transfer or to evaluate CRADAs potential benefits; (2) CRADA statements of work did not always contain adequate documentation or address potential benefits; (3) the national laboratories reviewed did not have effective mechanisms for continuous self-appraisal or measures of overall program success; and (4) CRADA provisions did not exist to ensure an accurate evaluation of partner contributions. Management partially concurred with the finding and recommendations. Management agreed to take action on those recommendations they concurred with. #=2 PART I APPROACH AND OVERVIEW INTRODUCTION The Department of Energy contracts with management and operating (M&O) contractors to operate its national laboratories. The laboratories are involved in multiple areas of research and development in science and nuclear technologies. This includes efforts to transfer technology from the national laboratories to the private sector. One type of technology transfer effort encompasses Cooperative Research and Development Agreements. CRADAs are cost-sharing agreements between a Federal entity, such as a DOE national laboratory and a private sector partner (partner), to engage in joint scientific research aimed at providing mutual benefits to the partner, DOE, and the U.S. economy. The objective of the audit was to determine whether the national laboratory M&O contractors were managing and implementing their CRADAs consistent with the goals of DOE's Technology Transfer Program and technology transfer legislation. SCOPE AND METHODOLOGY The audit was conducted at DOE's Los Alamos, Oak Ridge, and Lawrence Livermore National Laboratories, and at DOE's Albuquerque, Oak Ridge, and Oakland Operations Offices, from October 1993 through June 1994. The audit included a review of current technology transfer legislation and DOE policies and procedures pertaining to CRADAs adopted by Los Alamos, Oak Ridge, and Lawrence Livermore. We emphasized policies and procedures in place during Fiscal Years (FY) 1993 and 1994. Consistent with the specific audit objective, we: o examined technology transfer legislation, applicable DOE directives, and the prime contracts between DOE and the three national laboratories; o reviewed policies and procedures, memoranda, database information, and correspondence from DOE and the three national laboratories pertaining to CRADA management; and, o interviewed personnel responsible for approving and managing CRADAs. We judgmentally selected 110 joint work statements valued at $408 million for review. A joint work statement is a proposal prepared for a Federal agency by the director of a Government-owned, contractor-operated laboratory describing the purpose and scope of a proposed CRADA. As a proposal, it precedes preparation of the CRADA. We reviewed the joint work statements to determine if each: clearly addressed expected goals and accomplishments; defined tasks and milestones; described potential benefits to both DOE and the U.S. economy; and, addressed fairness of opportunity. In addition, we judgmentally selected 194 executed CRADAs valued at $580 million for examination. Our examination was to determine whether CRADA statements of work, tasks, milestones, and clauses were clear and specific. We also examined the CRADAs to determine whether each contained a means to verify the valuation of partner contributions to the CRADA, and whether the CRADA contained a clear statement of potential benefits to both DOE and the U.S. economy. Finally, we reviewed supporting documentation provided by laboratory personnel to determine whether it was specific and consistent with the tasks or phases outlined in CRADA statements of work. The audit was conducted according to generally accepted Government auditing standards for performance audits and included tests of internal controls and compliance with laws and regulations to the extent necessary to satisfy the audit objective. Accordingly, we assessed the significant internal controls with respect to the management of CRADAs. We relied on computer-processed data in only one limited instance: a computerized list of all executed CRADAs from each of the three DOE national laboratories. Due to time constraints, we did not examine whether the lists actually represented all executed CRADAs at those sites. Because our review was limited, it would not necessarily have disclosed all internal control deficiencies that may have existed at the time of our audit. We discussed the results of the audit with DOE's Associate Deputy Under Secretary for Operations, Office of Technology Partnerships, in November 1994. BACKGROUND During the 1980s, Congress began to focus on technology transfer as a way to correct what it perceived as an unfavorable imbalance in U.S.-international trade. There was concern that the U.S. commercial marketplace was not receiving the potential benefits of science and technology developed at the Federal research and development laboratories. Congress believed that a major gap had emerged between the Federal Government's technology base and industry's ability to commit resources to applying new technologies in the marketplace. In addition, the downsizing of the U.S. defense establishment contributed to a major shift in Federal funding from weapons research and development to commercial technology research. Congress enacted three laws to promote technology transfer and establish it as a specific mission of all Federal laboratories: o The Stevenson-Wydler Technology Act of 1980 required Federal laboratories to take more active roles in technical cooperation with U.S. industry, and established an Office of Research and Technology Application at each Federal laboratory. o The Federal Technology Transfer Act of 1986 authorized Government-owned, Government-operated laboratories to form CRADAs with partners and made technology transfer a job requirement of every laboratory scientist and engineer. o The National Competitiveness Technology Transfer Act of 1989 amended the Stevenson-Wydler Act by giving Government-owned, contractor-operated laboratories the authority to enter into CRADAs directly with private sector partners without triggering many legal conditions used by the Government, such as adherence to acquisition regulations. In addition, this act also mandated technology transfer as a mission for all Federal laboratories. In the early stages of technology transfer efforts, the Executive Branch also wanted to enhance the transfer of Federally-developed technology to the U.S. economy. This interest in technology transfer was prominent within DOE and its national laboratories. DOE believed that effective transfer of its technologies could benefit the commercial marketplace. Secretary of Energy Notice (SEN) 30A-92 stated that "... we can help U.S. industry develop economically and environmentally superior products and processes, create new jobs, and enhance the skill level of the U.S. economic strength and national security." In addition, SEN 30A-92 indicated that the mission of DOE's technology transfer was to help enhance U.S. competitiveness and national security, by expanding and accelerating the transfer of Federally-funded technologies and knowledge into commercial application by U.S.-based industry. In support of this mission, SEN 30A-92 indicated that DOE would increase participation by U.S.-based industry and by DOE, and would accelerate the technology transfer process. As a result, during 1991 and 1992 DOE operations offices began to review and approve a rapidly increasing number of proposed CRADAs between DOE facilities and interested partners. As of September 1994, DOE facilities had entered into 1,000 CRADAs. Moreover, in the last three years, DOE entered into CRADAs valued at $1.3 billion. DOE's share of the contributions to those CRADAs was 44 percent (approximately $572 million). The FY 1995 budget for DOE reserved $279 million for CRADAs. In July 1994, the Office of Inspector General issued a report titled "Inspection of Selected Issues Regarding the Department's Enhanced Technology Transfer Program" (DOE/IG-0353). This report discussed four issues for management's attention. Three issues dealt primarily with budget and accounting procedures, and the fourth dealt with performance measurement. The report stated that the Department had not developed a performance measurement and reporting system as required by the FY 1993 Technology Transfer Crosscut Plan. The Department concurred with the report's recommendation to expedite the development of such a system. OBSERVATIONS AND CONCLUSIONS We recognized during the audit that CRADAs are an important vehicle for DOE national laboratories to transfer "state-of-the-art" technology to the private sector in order to enhance the U.S. economy. CRADAs are an essential tool to effectively use Federally-developed technologies to benefit DOE and the U.S. economy. Further, during the audit, the three national laboratories we reviewed began efforts to improve management of their CRADAs. For example, Oak Ridge National Laboratory was creating a matrix program to address benefit and results of completed CRADAs. Also, Lawrence Livermore National Laboratory was attempting to implement a program to administer CRADAs, and Los Alamos National Laboratory was creating a program to track milestones. Despite these steps, we found that DOE's efforts to manage CRADAs could be improved in four specific areas: joint work statements; statements of work; milestones; and valuation of partner contributions. These areas could be improved through more specific DOE Headquarters guidance addressing management issues identified in this report. Addressing these issues would strengthen and improve the quality of CRADAs and assist in meeting the Secretary's stated goal of implementing total quality management for all DOE processes and products. The Secretary of Energy Advisory Board (Galvin Commission) report, titled "Alternative Futures for the Department of Energy National Laboratories," was published after the completion of our field work. In this report, the Galvin Commission stated that, taken as a whole, industrial competitiveness activities (including CRADAs) at the national laboratories were unfocused and lacked a firm policy foundation. While giving DOE credit for developing criteria for greater focus, the report recommended increased emphasis on measurement of technology transfer performance and more rigorous technical and merit review of CRADAs. The Galvin Commission also noted that improvements in the selection and monitoring processes need not, and should not, increase the time requirements for CRADA negotiations. Our audit results support these conclusions and our recommendations should help to improve the selection and monitoring processes of the CRADA program. In addition, by addressing these issues in a timely manner, DOE would show a positive response to Congressional and others concerns about CRADA accountability. In the Congress, legislation was proposed which was designed to increase accountability over the CRADA program. McGraw-Hill's "Federal Technology Report" reported that during 1994 Senate bill S.473 was passed to cap the maximum amount of funds that DOE national laboratories would be allowed to spend for technology transfer at 20 percent of their budgets until 1999. Thereafter the cap would be 10 percent of their budgets which would effectively cut the funding in half. Additionally, in its passage of S.473 the House Energy and Commerce Committee specified that all CRADAs, where the Federal share is worth $500,000 or more, must include technical milestones, vendor goals, and evaluation criteria. Also, these CRADAs must be evaluated annually against the criteria. The Armed Services Committee and the Science, Space, and Technology Committee passed H.R. 1432 which contained provisions similar to those in S.473. To date, this legislation has not been enacted. DOE and the National Aeronautics and Space Administration (NASA) have shown similar concerns over demonstrating successful CRADA results. The Secretary and Under Secretary of Energy have spoken of a need for more accurate gauges of the effectiveness of interactions between industry and the national laboratories. Also, NASA's Office of Advanced Concepts and Technology is now soliciting information from commercial partners in CRADAs and other technology transfer programs in order to calculate a rate of return on such investments. NASA is also involved in an interagency team examining the feasibility of extending such measurements to technology projects throughout the Federal Government. Our finding relating to DOE national laboratory management of CRADAs disclosed material weaknesses that management should consider when preparing yearend assurance memoranda on management controls. PART II FINDING AND RECOMMENDATIONS Administration of Cooperative Research and Development Agreements at DOE National Laboratories FINDING DOE established policies to ensure that CRADAs enhance U.S. competitiveness in the world economy, provide a reasonable return on resources invested, and enable successful commercialization of technologies developed. However, we found that the efforts to manage CRADAs at three DOE national laboratories did not ensure that these DOE policy goals were met in four areas: (1) joint work statements; (2) statements of work; (3) CRADA milestones; and (4) valuation of partner contributions to a CRADA. This occurred because DOE's Office of Technology Partnerships provided insufficient implementing guidance. As a result, DOE could not demonstrate that CRADAs met their intended goals; measure progress toward achieving stated goals; determine the viability of potential CRADAs and partners in-kind contributions; and, ensure that consistent management guidance was provided to all DOE facilities. RECOMMENDATIONS We recommend that the Associate Deputy Under Secretary for Operations, Office of Technology Partnerships, in coordination with sponsoring DOE program offices: 1. Provide sufficient implementing guidance to DOE's national laboratories for measuring CRADA progress and results. This guidance should: a. ensure that all joint work statements clearly address expected goals and accomplishments and define tasks and milestones; b. ensure that all statements of work contain a detailed description of work, tasks and milestones; c. ensure that all CRADAs and joint work statements clearly state the CRADA's projected potential benefit(s) to both DOE and to the U.S. economy; d. provide for implementation of a formal tracking system that adequately documents and links milestone progress directly back to the tasks contained in the statements of work; and, e. ensure that reports on all completed CRADAs include an explanation of: (1) whether the projected potential benefit(s) to both DOE and to the U.S. economy actually occurred; (2) whether the partner improved its efficiency and effectiveness; and (3) the potential for commercialization and technical success resulting from the completed CRADA. 2. Establish a mechanism to ensure proper valuation of partner contributions to a CRADA. MANAGEMENT REACTION Management concurred with the finding and partially concurred with the recommendations. Detailed management and auditor comments are included in Part III of this report. DETAILS OF FINDING DOE established policies for technology transfer involving its national laboratories through a General Guidance Memorandum. The policies were designed to promote technology transfer that could benefit DOE programs and the U.S. economy. In order to determine whether the policy goals outlined in the memorandum were achieved, DOE needed mechanisms to measure the progress and results of technology transfer programs. The Office of Technology Partnerships issued the General Guidance Memorandum to operations offices establishing policy goals for technology transfer programs, including CRADAs. These goals covered a number of issues, including technology transfer mechanisms, and evaluation and oversight. For example, the section of the memorandum covering technology transfer mechanisms indicated that mechanisms selected should facilitate technology transfer, while promoting a fair return to the taxpayer and benefits to the U.S. economy. Also, the section on evaluation and oversight indicated that each DOE facility shall establish criteria to measure the success of the overall technology transfer effort. It cited four examples of such criteria: (1) documenting technology transfer activities; (2) establishing internal controls, including mechanisms for continuous self-appraisal at all management levels; (3) using measures of overall program success, including milestones achieved and notable deliverables related to the technology transfer activities and tasks; and (4) using standard accounting and auditing procedures with appropriate tracking of funds. Although the General Guidance Memorandum established technology transfer policies, our audit disclosed that DOE mechanisms to measure the progress and results of technology transfer programs were inadequate. For example, DOE issued a "modular CRADA" with 60 pre-approved provisions designed to significantly shorten the process for negotiation and approval of CRADAs. The purpose was to enable national laboratories and their partners to use a "mix-and-match" approach to select appropriate provisions for a CRADA. However, the modular CRADA did not address mechanisms to measure the progress and results of CRADAs. In the absence of such mechanisms, we developed techniques to measure the progress and results of the CRADAs we reviewed. Our techniques were designed to determine whether joint work statements and statements of work clearly stated how the proposed CRADA would be conducted: who would do the work; the purpose of the work; tasks to be performed; proposed milestones; and, how the work would potentially benefit DOE and the U.S. economy. Additionally, we examined CRADA milestone documentation to determine if it explained how a specific task was completed, or determine whether it could be used to measure progress toward completion. Finally, CRADAs were reviewed to determine if they contained mechanisms to verify the valuation of partner contributions. LABORATORY MANAGEMENT OF CRADAS We found that the efforts to manage CRADAs at Los Alamos, Oak Ridge, and Lawrence Livermore national laboratories did not fully achieve DOE's policy goals outlined in the General Guidance Memorandum. Specifically, (1) joint work statements did not always contain clearly defined information that allowed DOE to facilitate technology transfer or to evaluate a CRADA's potential benefits; (2) CRADA statements of work did not always contain adequate documentation or address potential benefits; (3) the three national laboratories did not have effective mechanisms for continuous self-appraisal or measures of overall program success; and (4) CRADA provisions did not exist to ensure an accurate valuation of partner contributions. Joint Work Statements Joint work statements did not always contain clearly defined information that allowed DOE to facilitate technology transfer, or to evaluate a CRADA's potential to provide benefits to both DOE programs and the U.S. economy. Of the 110 joint work statements reviewed: 18 did not clearly address expected goals and accomplishments; 37 did not define tasks and milestones; and 45 did not clearly address potential benefits to both DOE and the U.S. economy. For example, one joint work statement's section on potential benefits stated that DOE would have access to new technology, but did not indicate how this technology would benefit DOE. CRADA Statements of Work CRADA statements of work did not always contain adequate documentation, or address potential benefits to DOE programs or to the U.S. economy. The statement of work should be the CRADA's detailed outline of the actual work to be performed. We reviewed statements of work for 194 CRADAs and found that 109 did not clearly define one or more of the following: the description of work, the tasks to be completed, or milestones assigned to each task. By contrast, an adequate CRADA statement of work would contain specific information that clearly defined the purpose, scope, description, tasks, and milestones for both parties. Potential benefits to DOE or the U.S. economy were not addressed in 170 of the CRADAs. Although 75 CRADAs addressed potential benefits of the research/development, they did not provide enough information to identify how the developed product or process would benefit DOE or the U.S. economy. The remaining 95 did not address potential benefits at all. In commenting on a draft of this report, the Associate Deputy Under Secretary acknowledged that improvements could be made to joint work statements and statements of work. He said that such improvements will be the subject of conversations between the Office of Technology Partnerships, the funding program offices, and the Office of Field Management. Milestones The three national laboratories did not have effective mechanisms for continuous self-appraisal or measures of overall program success. Although all three laboratories submitted quarterly reports to DOE on CRADA progress, these reports did not address whether CRADA supporting documentation linked milestone progress directly back to tasks or phases outlined in the statement of work. This made it difficult to determine if milestones were being met or completed on time. We reviewed the milestone documentation for 60 CRADAs and found that the documentation provided for 51 CRADAs did not specifically tie back to the tasks or phases contained in the CRADA statements of work. The Associate Deputy Under Secretary's response to the draft report also stated that the milestone issue would be addressed by the Department's successful pilot of its Integrated Technology Transfer System. This system, which management believes will soon be implemented, is to provide data for performance measurements of the Department's technology programs. While such a system is a positive step, because it was not in place at the time of our audit, we could not determine whether the system will actually measure milestone progress for ongoing CRADAs. Valuation of Partner Contributions The three national laboratories did not employ standard accounting and auditing procedures with appropriate tracking of funds to verify the value partners assigned to their in-kind contributions. In-kind contributions are noncash contributions that may be in the form of labor, facilities, equipment, etc., that the partners contribute to cover their share of the CRADA. During the audit, we found that principal investigators (laboratory personnel responsible for carrying out CRADA work) generally had access to data showing the level of a partner's in-kind contributions to the CRADA. However, this access was not sufficient to protect the Government's investment in CRADAs because it did not provide a formal, independent valuation of contributions. None of the 194 CRADAs we reviewed contained mechanisms that would allow the Government to determine a partner's in-kind contribution. Although earlier versions of standard DOE guidance allowed for the negotiation of audit clauses, this was not done because DOE officials believed the potential for audit would discourage partner participation. The Associate Deputy Under Secretary stated that the current DOE policy is not to audit partner in-kind contributions. Also, DOE policy allows funds from other Federal programs to be applied to CRADAs, subject to the rules applicable to those other programs. Therefore, it would be difficult to account for or validate partners in-kind contributions. However, our position is that in the absence of the audit clause DOE should devise an alternate mechanism to independently validate the partner's in-kind contributions. We believe this can be accomplished in a way such that the private sector partner would not view the Government's activity as a disincentive to participation in a CRADA project. A recent Office of Inspector General audit report (WR-B-95-01) noted that DOE established CRADA cost-sharing goals but did not establish controls to provide reasonable assurance of achieving those goals. The report noted that DOE's CRADAs at national laboratories did not include provisions that specified how the partner was to charge its costs or that provided for Government access to the partner's accounting records. Partners were found to be shifting their share of CRADA costs to the Government through inappropriate charges to Federal contracts. This cost shifting reduced the partner's financial risk and possibly the partner's incentive to exert best efforts to achieve project success. DOE's Office of Chief Financial Officer has also expressed concern about the valuation of partner contributions. In the May 1993 report of its compliance review of DOE's Albuquerque Operations Office, the Chief Financial Officer noted that under existing controls, DOE could be paying more than its fair share of CRADA costs. The Chief Financial Officer's report recommended that DOE Albuquerque Operations Office direct the finance staffs of its national laboratories to review partner valuation of in-kind contributions. DOE GUIDANCE ON MANAGEMENT OF CRADAS The national laboratories could not ensure that their CRADAs met the policy goals outlined in the General Guidance Memorandum because DOE Headquarters did not provide sufficient implementing guidance. Headquarters issued a modular CRADA, that contained examples of desirable CRADA provisions, which was designed to expedite the negotiation and approval of CRADAs. However, this modular CRADA did not provide implementing guidance that sufficiently addressed how to achieve the policy goals outlined in the General Guidance Memorandum. Headquarters emphasis on expediting the negotiation and approval of CRADAs may have contributed to the failure to develop and issue detailed implementing guidance. POTENTIAL DIFFICULTIES In effect, the vague statements of work and joint work statements, the failure to link work performed to milestones, and the failure to account for partners in-kind contributions made it difficult to adequately determine the viability of the CRADAs. If these deficiencies are not addressed, DOE could be in the position of justifying large expenditures on CRADAs that were not viable as well as adversely effecting future funding for CRADAs. The failure to provide detailed implementing guidance can also have the following effects. o DOE may not be able to demonstrate that a CRADA was successful. We reviewed final reports on 17 completed CRADAs, valued at $3.3 million, and found that none explained how the completed CRADA benefited the DOE or the U.S. economy. Although final reports were completed, the reports were written from a technical standpoint addressing technical accomplishments. Therefore, it was difficult to determine if the CRADA was successful in actually producing a product, improving a process, or providing a benefit to DOE programs or the U.S. economy. o Without adequate links between work performed and milestones achieved, DOE cannot adequately monitor milestone progress to ensure timely completion of the CRADA. Support for CRADAs may diminish if they are not completed in a timely manner. o Without a mechanism to validate and account for partner in-kind contributions, DOE may commit to a CRADA that has little chance of success because it is underfunded. Also, such a mechanism would assist in strengthening internal controls for preventing and detecting fraud and abuse. o DOE facilities may manage CRADAs in a manner that is inconsistent with following DOE policy and achieving DOE's goals. More detailed DOE-wide management guidance would promote consistency in management approaches and timeliness in implementing CRADAs for achieving DOE's goals. In summary, we concluded that cooperative efforts, such as those envisioned in the CRADA concept, have the potential to greatly benefit the U.S. economy. However, based on the audit results, we are concerned that the CRADA process could be undermined if the Department does not implement appropriate management techniques and performance measurement mechanisms. We believe that this can be done without creating bureaucratic disincentives for commercial sector interaction with the Department's national laboratory system. PART III MANAGEMENT AND AUDITOR COMMENTS The Office of Technology Partnerships concurred with the finding and with recommendations 1.a, 1.b., and 1.e. However, there was no concurrence with recommendations 1.c, 1.d, and 2. A summary of management's comments on the latter recommendations and our comments follow. Recommendation 1.c. Ensure that all CRADAs and joint work statements clearly state the CRADA's projected potential benefit(s) to both DOE and to the U.S. economy. Management Comments. The benefit to the U.S. economy is already sufficiently addressed by the modular CRADA's competitiveness work sheet. This work sheet requires that the CRADA partner manufacture substantially in the U.S. or demonstrate alternative benefits to the U.S. economy. Management also stated that the DOE field organization which approves CRADAs should ensure that this policy guidance is implemented. Auditor Comments. The modular CRADA's competitiveness work sheet contains valid information which pertains to manufacturing in the United States. The issue of benefit, however, can go well beyond the determination of where a product will be manufactured. Most CRADAs reviewed during the audit did not contain clear and specific statements as to what benefits were expected, either to DOE or the U.S. economy. Negotiators are required to ensure that the taxpayers will receive some return on their investment; therefore, descriptions of such benefits should appear in joint work statements prior to CRADA approval. Recommendation 1.d. Implement a formal tracking system that adequately documents and links milestone progress directly back to the tasks contained in the statements of work. Management Comments. The Department has already successfully piloted and will soon implement its Integrated Technology Transfer System. This system will provide data for performance measurements of the Department's technology programs, including CRADAs. This data will be input by the laboratories and reviewed by DOE operations offices. The institution of a formal tracking system for documentation of progress toward milestones in statements of work is not an effective research or project management tool. Regular project reviews and informal, outcome- oriented communication among managers and principal investigators are much better and more cost-effective tools for management of research projects than the building of files of formal reports. The latter measure would divert the attention of principal investigators from technical work and constitute a burden on available time of laboratory scientists and engineers. It might also be viewed as intrusive by CRADA partners. Auditor Comments. Implementing the Technology Transfer System to measure performance is a positive step. However, because the pilot system was not in place at the time of our audit, we could not determine whether the system could measure milestone progress. As to the issue of additional burden, many principal investigators were already preparing some type of informal report to show milestone progress. These reports, in fact, were helpful during the audit in determining that milestones did not always link back to tasks in the statements of work. Formalizing this milestone reporting process should clarify and standardize the reports and should not create an additional administrative burden for researchers. Recommendation 2. Establish a mechanism to ensure proper valuation of partner contributions to a CRADA. Management Comments. The Department has a written policy that it does not audit partner in-kind contributions and the funds from other Federal programs may be applied to CRADAs (subject to the procedures of the process that granted those funds). Further, it is up to the laboratories, by exercising good business judgment, to ensure that partner in-kind CRADA contributions are reasonable. It is the role of the DOE operations offices to ensure that good judgment is being exercised. Auditor Comments. The inclusion of audit clauses in CRADAs is but one suggestion to achieving the goal of ensuring that partners in-kind contributions are validated. To encourage fair cost-sharing arrangements, DOE should devise some mechanism, audit clause or otherwise, to independently validate the partners in-kind contributions. The principal investigator's access to the partner's in-kind contributions is a positive step, but does not sufficiently protect DOE's and the taxpayers' interests. Additionally, advisory management comments from the Director of Defense Program's Office of Economic Competitiveness and the Director of the Office of Energy Research's Laboratory Management Division follow. Defense Programs (DP) Advisory Comments The Director, Office of Economic Competitiveness (DP-14) provided advisory comments stating that the report identified some legitimate concerns for DOE as a whole. Further, the Director stated that he agreed with the comments provided by the Office of Technology Partnerships. Energy Research (ER) Advisory Comments The Director, Laboratory Management Division (ER-80) stated that he wanted to reinforce the comment from the Office of Technology Partnerships that audits are not appropriate for assessing partners' in-kind contributions. Summary Management Comments. The Galvin Commission report identified excessive audits and inspections as one of the problems in the DOE laboratory system. The Office of Technology Partnerships does not wish to be part of or add to that problem by requiring audits of CRADA partners. Auditor Comments. The Galvin Commission report also emphasized that industrial competitiveness activities at the national laboratories are unfocused, need to be more clearly related to DOE's mission, and need better measures of performance. Management's position does not recognize the need for tools for assessing program goals or needed improvements. Audits may not be the sole alternative, but the Office of Technology Partnerships needs to develop and use alternative avenues for assuring that DOE, on behalf of the taxpayers, is investing wisely in each new CRADA. EXAMPLE OF CUSTOMER RESPONSE FORM IG Report No. DOE/IG-03 73 CUSTOMER RESPONSE FORM The Office of Inspector General has a continuing interest in improving the usefulness of its products. We wish to make our reports as responsive as possible to our customers' requirements, and therefore ask that you consider sharing your thoughts with us. On the back of this form, you may suggest improvements to enhance the effectiveness of future reports. Please include answers to the following questions if they are applicable to you: 1. What additional background information about the selection, scheduling, scope, or procedures of the audit or inspection would have been helpful to the reader in understanding this report? 2. What additional information related to findings and recommenda- tions could have been included in this report to assist management in implementing corrective actions? 3. What format, stylistic, or organizational changes might have made this report's overall message more clear to the reader? 4. What additional actions could the Office of Inspector General have taken on the issues discussed in this report which would have been helpful? Please include your name and telephone number so that we may contact you should we have any questions about your comments. Name Date Telephone Organization When you have completed this form, you may telefax it to the Office of Inspector General at (202) 586D0948, or you may mail it to: Office of Inspector General (IG-1) Department of Energy Washington, D.C. 20585 ATTN: Customer Relations If you wish to discuss this report or your comments with a staff member of the Office of Inspector General, please contact Rob Jacques at (202) 586D3223.