!R! CALL BCA; EXIT; THIS OPINION WAS INITIALLY ISSUED UNDER PROTECTIVE ORDER AND IS BEING RELEASED TO THE PUBLIC IN REDACTED FORM ON JULY 24, 1992 ____________________________ DECIDED: May 29, 1992 ____________________________ GSBCA 11736-P, 11745-P DENRO, INC., and WESTINGHOUSE ELECTRIC CORPORATION, Protesters, v. DEPARTMENT OF TRANSPORTATION, and DEPARTMENT OF DEFENSE, Respondents, and LITTON SYSTEMS, INC., Intervenor. William H. Butterfield, Jed L. Babbin, Charlotte F. Rothenberg, and Lori Beth Feld of McGuire, Woods, Battle & Boothe, Washington, DC, appearing for Protester Denro, Inc. Rand L. Allen, Philip J. Davis, Paul F. Khoury, and Eric W. DeSilva of Wiley, Rein & Fielding, Washington, DC, appearing for Protester Westinghouse Electric Corporation. Richard J. McCarthy, John R. McCaw, Jr., and Anthony L. Washington, Office of the Chief Counsel, Federal Aviation Administration, Washington, DC, appearing for Respondent Department of Transportation. Roger B. Sabin and Captain Richard B. Wingate, Office of General Counsel, Defense Information Systems Agency, Arlington, VA, and Clifton M. Hasegawa, Virginia Farrier, and Douglas G. White, Defense Commercial Communications Office, Defense Information Systems Agency, Scott Air Force Base, IL, appearing for Respondent Department of Defense. David V. Anthony and Richard P. Rector of Pettit & Martin, Washington, DC, appearing for Intervenor. Before Board Judges NEILL, HYATT, and WILLIAMS. Board Judge HYATT. These consolidated protests challenge the elimination of Denro, Inc. and Westinghouse Electric Corporation from the competitive range of a procurement for voiceband switching systems,[foot #] 1 leaving one offeror in the competition, intervenor Litton Systems, Inc. The procurement is being conducted by the Defense Information Systems Agency, Defense Commercial Communications Office (DECCO), on behalf of the Federal Aviation Administration (FAA).[foot #] 2 Denro and Westinghouse were informed by respondent that their proposals would no longer be considered for award because of their failure to comply or verify compliance with the request for proposals (RFP). The principal argument raised by both protesters is that the Government's competitive range determination was improper. In particular, the Government effectively treated the results of an operational capabilities demonstration (OCD) as a pass/fail test in eliminating from the competition all proposals that were not ----------- FOOTNOTE BEGINS --------- [foot #] 1 Denro filed its protest on February 25, 1992; Westinghouse filed its protest on February 28, 1992. The due date for this decision has been extended as a result of a delay in completing the hearing in this matter. The delay was necessitated by the unavailability, for health reasons, of the Government's key technical witness to participate in the trial as originally scheduled. [foot #] 2 DECCO and the FAA are referred to interchangeably as respondent in this decision. ----------- FOOTNOTE ENDS ----------- deemed to have performed satisfactorily. Protesters also contend that contrary to the technical evaluation plan (TEP), the Government scored all technical functions not demonstrated at OCD as 0, meaning (under the definition established in the TEP) that the deficiency was virtually unsusceptible of being corrected. In addition, although the Government initiated discussions with all vendors following the OCD, it placed an unannounced higher standard of proof of compliance with RFP requirements on Westinghouse and Denro than it did on Litton, thus making it impossible for either protester to improve its technical score. Based on a review of the extensive record in these protests, we agree that the evaluation process was defective and that the subsequent elimination of the Denro and Westinghouse proposals from the competitive range of the procurement was improper. Experts retained by counsel and engineers employed by protesters offered credible evidence that many of the functions scored as "not demonstrated" by the OCD team were relatively simple to correct. In addition, some discrepancies involved reasonably differing interpretations of specification requirements, with protesters agreeing to modify their systems in order to give the Government what it wanted. Nonetheless, responses generated by Denro and Westinghouse to written discussion items issued after OCDs were almost uniformly rejected as inadequate by the technical evaluators, not because the proposed modifications were technically noncompliant, but because protesters did not supply underlying documentation such as regression studies, test data, software code, and the like. We find that additional substantiating documentation, if desired, could have been requested without engaging in technical levelling. Protesters should not have been disqualified from the competition on this basis. Accordingly, we grant the protests. Findings of Fact[foot #] 3 Background 1. The Integrated Communications Switching System (ICSS) was first identified as an element of the FAA's National Airspace Plan (NAS) modernization program in 1981. The ICSS is a voice communications and voice switching system that controls and transfers communications by and among air traffic controllers to aircraft in flight and on the ground. This system is intended to replace leased electro-mechanical voice switches. Protest File, Exhibit 33; Transcript at 830, 1958-59. 2. Phase 1 of the ICSS procurement was intended to obtain leased "critical" communications services. Critical ----------- FOOTNOTE BEGINS --------- [foot #] 3 The Rule 4 file and supplemental exhibits submitted by the parties in this case are sequentially numbered by Bates stamp designation. This pagination is used in citing to these exhibits. ----------- FOOTNOTE ENDS ----------- communications services are those directly involved in the control over and separation of aircraft. Phase 1 was divided into three types. Type 1 systems were used in "critical" facilities, small aircraft control towers, and Terminal Radar Approach Control (TRACONS) Facilities. Type 2 systems were placed in large and busy airport control towers and TRACONs, also considered critical by the FAA. Type 3 systems were identified for use in FAA Automated Flight Service Stations, facilities that, although essential, are not involved in controller-to- aircraft communications. Transcript at 1818-20. 3. In 1982, both Denro and Litton received awards of contracts to lease systems to the FAA under the Phase 1A program. Transcript at 1821-22. In 1988, Denro received another contract, following a sealed-bid procurement, under the Phase 1A procurement for Type 3 systems. The FAA experienced technical complications with systems received under both procurements. It attributes failures under the Phase 1 program in part at least to untested system software. Transcript at 1824-28. 4. The Phase 1B procurement, which is the subject of this protest, is intended to satisfy critical communications needs at new FAA facilities, including the planned Denver airport, and the new Southern California TRACON. Transcript at 1959. This procurement will also serve as the vehicle to replace the last of the remaining electro-mechanical voice switches in the field. Id. at 1958-59. 5. The ICSS-1B is a voice communications switching system that will provide air-to-ground and ground-to-ground communications capability for air traffic controllers in airport towers and TRACON facilities throughout the United States. Each ICSS-1B system is comprised of position equipment, interface equipment, configuration control equipment, maintenance and diagnostic equipment, power system equipment, cabling, and an administrative telephone system. Protest File, Exhibit 21. 6. This acquisition was originally proposed as the preliminary Tower Communication System (pre-TCS), and called for the procurement of approximately twenty-four systems at critical locations which could not await the development of TCS systems. By early 1990, the FAA had abandoned the developmental approach and the Pre-TCS program was incorporated into the Phase 1B of the ICSS. The request for proposals (RFP) was issued in August of 1990, and contemplated the award of a firm fixed-price requirements type contract for a one-year base period with nine one-year options.[foot #] 4 Protest File, Exhibits 9- 27; Transcript at 1138. The Subject Procurement 7. In 1991, the FAA shelved the TCS program because of the risk associated with a developmental voice switch program and the urgent, immediate need to replace obsolescing switches in the field. Transcript at 830, 1831-33. As a consequence, the quantities of switches to be procured under the Phase 1B procurement were increased, by amendment 004 to the RFP, from 24 to 200. Protest File, Exhibit 5; Transcript at 1004-05, 1138-39, 1962-65. 8. The Acquisition Plan for the Phase 1B program states that "[t]here are known sources that meet the requirements of this acquisition." The plan further notes that other sources, , had expressed an interest in the procurement. Protest File, Exhibit 33. 9. Although the Phase 1B program seeks systems that are substantially similar to those already in the field, the FAA did modify the purchase description to some extent in order to incorporate "lessons learned" in fielding, operating and maintaining the previously purchased systems. Transcript at 1823-24, 1848-53. One FAA witness testified that the functionality of the fielded systems is about 95% of what the 1B specification is seeking to obtain. Id. at 970. 10. To minimize the risk associated with developmental systems, the Phase 1B procurement initially provided that: [T]he Contractor shall use Commercial-Off- the-Shelf (COTS) items to meet ICSS functional requirements. For purposes of the procurement, the term COTS is used to designate material available from a variety of sources requiring little or no developmental effort. ----------- FOOTNOTE BEGINS --------- [foot #] 4 Under an interagency agreement, the procurement was conducted for the FAA by the Defense Commercial Communications Office (DECCO). FAA used DECCO's services in this procurement because of DECCO's expertise in the telecommunications field. Transcript at 1957. DECCO is responsible for award and management of the contract. It also evaluated cost proposals. The FAA, as the ultimate user of the equipment, through its subcontractor, Arinc Research, Inc., was responsible for the technical evaluation of offers. Protester's Exhibit 17. ----------- FOOTNOTE ENDS ----------- Protest File, Exhibit 20 at 428. In amendment 004, this language was changed to state instead that: "The Government prefers, but does not require, that the contractor use commercial off-the- shelf (COTS) items to meet ICSS functional requirements." Id., Exhibit 5. The purpose of this change was to reflect the Government's actual objective to obtain non-developmental equipment, while recognizing that it was not absolutely necessary to obtain equipment that has been commercially sold. Transcript at 1965-66. After the issuance of amendment 004, no mandatory requirements remained in the RFP. Id. at 1144-45, 1916, 2002-03. 11. In addition to an operational capabilities demonstration (OCD) to be scheduled prior to completion of evaluations, the RFP provided for a series of post-award tests of systems to be purchased. These include first article tests and factory acceptance tests. Protest File, Exhibit 20. The first article test is for the purpose of demonstrating "full compliance of the system" with the requirements of the purchase description. The factory acceptance test is to "ensure that the system is equipped as required and that all functions are performed" according to the purchase description. Finally, once shipped to the site, each system is subject to an installation test intended to "ensure that the system is fully operational and ready for cutover." Id. 12. Prior to the issuance of amendment 004, vendors, had submitted proposals to supply switches. Protest File, Exhibits 38, 40, 41. Following issuance of the amendment, The Technical Evaluation Process 13. Section M of the RFP provided that technical evaluation factors would consist of three general areas: (1) technical equipment, (2) technical services and, (3) management evaluation. The evaluation of technical equipment was further subdivided into four areas: (1) performance requirements, (2) functional requirements, (3) compliance with standards and, (4) physical requirements. Protest File, Exhibit 19, M.2.b, M.2.c. 14. The Technical Evaluation Plan (TEP) established the procedures to be followed in evaluating proposals. Protest File, Exhibit 37. The TEP provided that technical evaluations would be accomplished by the technical evaluation team (TET). In general: The TET supports the SEB [Source Evaluation Board] by performing an in-depth evaluation of all offerors' proposals through a comparison of each proposal with the RFP requirements. This evaluation results in the assignment of numerical scores to the submitted proposals and in written technical and quality assurance findings. Id. at 000871. 15. The TET was composed of groups of evaluators, established in accordance with Section M's proviso that offers be technically evaluated in three general areas. The individuals assigned to evaluate proposals consisted of employees of Arinc Research, a consultant retained by the FAA, and various FAA employees. Each area had a team leader, who was an employee of Arinc. Protest File, Exhibit 37. 16. Each area leader was responsible for briefing the group on procedures for evaluating proposals. The area leader also worked with the group to attempt to reach a consensus for each assigned proposal area reviewed by individual evaluators. Protest File, Exhibit 37; Transcript at 60-63. 17. An employee of Arinc Research, with the title of senior principal engineer, served as the area leader for the technical requirements group. Transcript at 49. This individual does not possess an engineering degree, but has accumulated nearly 1000 hours of training with the Bell System in voice switching and transmission and in data communications. Id. 18. The TEP provided for preliminary detailed reviews of written proposals by the TET. Upon completion of this detailed preliminary evaluation, OCDs would be conducted. Following completion of the OCDs, discussion item sheets were to be prepared requesting clarifications and notifying offerors of discrepancies found during the OCDs. 19. Table IV of the TEP Protest File, Exhibit 37. 20. In light of the impact of amendment 004, see findings 7, 10, 12, and the magnitude of changes to submitted proposals, the Government decided to begin the evaluation process anew. Transcript at 1004-06. Detailed preliminary evaluations were conducted as provided in the TEP. The identification of ambiguities and deficiencies resulted in the issuance of approximately 200 discussion items (DIs) to the offerors. The DIs requested clarifications of ambiguities or areas that were not sufficiently addressed, and sometimes sought further detail or amplification of proposal points. Id. at 1006-07. The technical proposals and responses to the first round of DIs formed the basis of the technical scoring of proposals. Id. at 1007-10. 21. On the basis of the initial detailed evaluations, all proposals were included in the preliminary competitive range determination. Transcript at 1139. As of that time, based on the written information provided by the three companies, the majority of technical subitems in each of the proposals had been rated as . Protest File, Exhibits 154-57. Operational Capabilities Demonstrations 22. The solicitation addresses the conduct of operational capabilities demonstrations in several provisions. Section L of the RFP states that "[r]esponsive offerors shall be requested to perform an OCD as detailed in Attachment J-6." Protest File, Exhibit 18. This section further provides: The OCD is designed to validate the Offeror's capability to satisfy ICSS Phase 1B requirements. . . and to enable the Government to assess the schedule and technical risks associated with the offeror's proposed use of COTS and other than COTS hardware, software, and/or firmware. The Government reserves the right to perform detailed examination of all prepared components at the OCD. . . . " Id. Section M similarly describes the nature and purpose of the OCD: The OCD provides the offeror an opportunity to demonstrate to the Government the adequacy of the technical information contained in the proposal. Specifically, the OCD affords the offeror the opportunity to demonstrate an ability to meet the ICSS Phase 1B requirements. The OCD results will be considered in the technical evaluation process and the technical evaluation scores will be adjusted accordingly. In addition, the extent that the offeror displays during the OCD commercial off-the-shelf (COTS) hardware, software and/or firmware to meet the ICSS 1B requirements will be considered in determining the technical and schedule risks . . . . The determination will be expressed as high, medium, or low risk. Protest File, Exhibit 19. 23. 24. Attachment J-6 to the RFP defines the requirements for conducting the OCD. It indicates that offerors are "expected to provide a system substantially similar to that proposed for sale to the Government under the ICSS Phase 1B." Protest File, Exhibit 25. In addition: The OCD system shall be composed substantially of COTS components. It shall also correspond directly to the description provided in the Offeror's technical proposal and the OCD baseline. The OCD system should be assembled in a production configuration (i.e., use of breadboarding and similar techniques should be strictly limited). Id. Attachment J-6 further states that "[d]emonstrations will not be considered valid unless they have been witnessed by a representative of the Government." Id. 25. Attachment J-6 also explains that the OCD team will assess the OCD demonstrations as follows: 1) "fully demonstrated," 2) "partially demonstrated," or 3) "not demonstrated." Protest File, Exhibit 25. 26. Prior to the conduct of each OCD, the team conducting the OCD held a conference with the vendor to answer questions. The technical evaluation team (TET) leader, who attended all the OCDs, said that he "explained to each of the offerors that our intent in the OCD was to provide them an opportunity to validate the proposal . . . . I explained to them that if any feature was demonstrated, whether they intended to or not, the Government reserved the right to follow that to its logical conclusion." Transcript at 1015. Offerors were instructed that the Government wished to view a fully functional system and that for informational, or notational purposes, system abnormalities would be regarded as "demonstrated" as well. Id. at 1015-16. 27. The TEP included an OCD checklist to be followed in the conduct of all OCDs. Protest File, Exhibit 37. Attachment J-6 called out all the provisions of the purchase description (PD) that would be tested during the OCD. Id., Exhibit 25. As part of their proposals, offerors submitted their own OCD plans, reflecting the requirements enumerated in Attachment J- 6.[foot #] 5 28. The ICSS-1B OCDs were conducted and witnessed by the Government's OCD team in the summer of 1991. They were held in the following order: . Each offeror was allotted three and one-half days to complete its OCD, including any redemonstrations that an offeror wished to perform. Protest File, Exhibit 151. In addition to Arinc evaluators, the Government's OCD team included several FAA air traffic controllers, who operated the systems in an attempt to replicate typical air traffic control operations. Transcript at 497, 889. 29. The evidence of record indicates that prior to the actual conduct of the OCDs, all offerors anticipated that they would be demonstrating the equipment to the Government team, who would essentially act as observers. Transcript at 496-99, 1056- 57, 1508. Based on the assumption that the OCD was to be their "show," prepared "scripts" to be used in demonstrating features of their systems. E.g., Transcript at 481, 1056-57. in particular, would have prepared differently for the demonstration had it anticipated that the Government intended an interactive exercise of the system that tested many requirements at once in a random manner. Transcript at 2366-68. 30. Although the purchase description contains a provision for "dynamic testing" of equipment by the Government, this occurs in the conduct of post-award quality testing. That PD paragraph is not called out in Attachment J-6. Compare Protest File, Exhibit 21 with Exhibit 25. 31. The Government OCD team arrived at the offerors' plants expecting to see a 1B system that could be wheeled out the door and plugged in. Transcript at 2180. One member of the OCD team testified that he viewed the OCD as part of an "embedded strategy" in the procurement to "truncate" post-contract testing and use the OCD to "provide the Government an opportunity to ensure that this equipment, as advertised, worked." Id. at 1838- 40, 1888-89. In keeping with their philosophy that the system to be demonstrated should essentially be one that is ready to ----------- FOOTNOTE BEGINS --------- [foot #] 5 Id. __ ----------- FOOTNOTE ENDS ----------- install, the team undertook to "stress" the systems being demonstrated to test their ability to withstand the active air traffic control environment. See Transcript at 889. 32. The OCD team decided that if a feature received an assessment of fully demonstrated or "F", that mark would not be reversed, even if subsequent concerns arose. System deficiencies were contemporaneously identified, however, and discussed with each offeror as they arose during the OCDs. In addition, a daily summary of deficiencies and concerns was provided orally to offerors in a debriefing held at the end of each day. Protest File, Exhibit 151; Transcript at 1023-24, 2186-87. Once an item was marked as fully demonstrated, the OCD team did not alter that score on the checklist. Whether or not an item received an "F" for fully demonstrated, if concerns continued, the offerors were informed and were told that the concerns took precedence over the fully demonstrated score. Respondent's Exhibit 10; Transcript at 186-90. Offerors were also told that all areas deemed "not demonstrated" would be the subject of further discussions. Id. at 189. 33. The Government witnesses testified that the OCD was intended simply to be a demonstration, not a test. That is, there was no attempt by the Government witnesses to hook up gauges or otherwise measure the equipment for compliance with specifications. E.g., Transcript at 351. As such, the Government witnesses consistently testified that the OCDs were not intended to be a pass/fail exercise, either. E.g., id. at 141-42. The Denro 400 OCD 34. The OCD for Denro's model 400 unit was held on July 23- 26, 1991. The OCD got off on the "wrong foot" at the outset, with a disagreement over who would operate the equipment and who would follow the script and observe. Following some discussion of the issue, Denro acquiesced and permitted the Government to take the lead and operate the equipment. Transcript at 1025, 1508. 35. The Government's report of the OCD reflects that of the 118 items that were assessed, 95 were fully demonstrated by Denro, 1 was partially demonstrated, and the remaining 22 were deemed not demonstrated. Protest File, Exhibit 151. 36. Denro took issue with many of the "not demonstrated" assessments, which apparently were a result of misunderstandings and miscommunications between the Denro and the Government OCD teams. For example, one member of the Government team questioned the use of a hard disk by Denro in the demonstrated system, believing that the proposed system used only floppy disks. This individual testified that he felt that he had been deceived by that offeror on this issue. Transcript at 873. Denro received two "N's" for not demonstrating the system proposed. Protest File, Exhibit 151. In fact, Denro had proposed a hard disk and should have been scored as fully demonstrated on this item. Denro Supplemental Protest File, Exhibit 21. 37. A similar misunderstanding arose with respect to demonstration of mechanical requirements. There, Denro received an erroneous "N" for failing to demonstrate the actual racks that would be used to hold equipment. Protest File, Exhibit 43. Denro in fact used the equipment that it proposed at the OCD, but the Government somehow received the impression that it planned to use something else. In response to a DI, Denro confirmed that it intended to use the equipment proposed and demonstrated. Id., Exhibits 96, 97. 38. The items regarded by the OCD team as "not demonstrated" included the following: Protest File, Exhibit 151. 39. Following its OCD, Denro wrote a follow-up letter to the Government, responding to items that had arisen in the OCD. Protest File, Exhibit 56. The letter noted where "Ns" had been assigned due to varying interpretations of specifications (seven items), where Denro believed new requirements had been defined by the OCD team (three items), and where the OCD had observed a feature that 1) performed incorrectly (four items), 2) was judged as performing poorly (three items), or the team believed demonstration hardware differed from Denro's proposal[foot #] 6 (two items). Id. The letter included a table of all such items and indicated how they would be addressed, i.e., by modifying or repairing the system. At the time the letter was written, Denro assured the Government that many of the issues had been rectified already. Id. ----------- FOOTNOTE BEGINS --------- [foot #] 6 See Findings 36 and 37 above. Thus, three of the N's on Denro's OCD checklist were simply erroneous. ----------- FOOTNOTE ENDS ----------- Litton's OCD 40. The OCD for the Litton system was held on July 30 through August 2, 1991. This OCD, too, started off inauspiciously. According to the TET team leader: Initially it was rough. Initially Litton had a poor understanding of what the OCD was about. Initially the demonstration was not proceeding in a way that would allow [the TET] to actually look at their system. Transcript at 1028. See also Transcript at 931, 1056-57. 41. Litton did not produce any witnesses at the hearing, but indicated in a response to an interrogatory that after its expression of concern to the TET leader about the Government's interaction with the system, the situation was resolved by allowing "Litton's demonstration of a function to be completed without interaction and then to let the Government to check out anything it wished." Westinghouse Supplemental Protest File, Exhibit 6.[foot #] 7 42. ----------- FOOTNOTE BEGINS --------- [foot #] 7 Two Government witnesses contradicted this statement by asserting that Litton observed and the Government operated the equipment. Transcript at 1056-58; 1233-35. It is, nonetheless, difficult to reconcile fully this testimony with the written response given by Litton to the interrogatory question, thus casting some doubt on the Government's contention that the OCDs were all conducted in entirely the same manner. ----------- FOOTNOTE ENDS ----------- The Westinghouse OCD 43. The OCD of the Westinghouse system took place from August 6 to August 9, 1991. Protest File, Exhibit 151. Since Attachment J-6 did not spell out specific configurations, interfaces, or maps that the Government expected to see at the OCD, Westinghouse assumed that it could choose options permitted by the purchase description. It did not prepare its system to demonstrate every conceivable variance of configuration that might be demanded in actual usage. Transcript at 496-98. In addition, Westinghouse construed the OCD requirement to permit it to present the requirements in a serial fashion -- that is, one at a time. Transcript at 2366. It did not anticipate that the Government's OCD team would engage in an interactive exercise of its system that tested many requirements at once in a random manner. Had Westinghouse realized that this was the intent of the Government, it would have prepared differently for the OCD. Id. at 2366-68. As it was, Westinghouse's engineers testified that the random nature of the testing made it difficult, and in some cases, impossible, to trace or duplicate the TET's results. Id. at 499. 44. Because Westinghouse was not prepared for the type of OCD intended by the TET, it received a checklist at the conclusion of the OCD with which it did not agree. Approximately items received an "N" as "not demonstrated" at the OCD. Protest File, Exhibit 73, Transcript at 499-501. Among features where problems were noted were the following: Protest File, Exhibit 151. 45. Westinghouse protested the conduct of the OCD at the Board, but withdrew its protest, without prejudice, based in part on its understanding that it would have an opportunity to address the issues raised by respondent at the OCD in post-OCD discussion items.[foot #] 8 Westinghouse incorporated this earlier protest into its current protest at the Board. Westinghouse Complaint, 17. 46. Following OCDs, in expectation of receiving discussion items, Westinghouse to remedy[ ] the areas deemed not demonstrated by the OCD team. Transcript at 502; Protest File, Exhibit 41. The purpose of was to help Westinghouse achieve a better understanding of the FAA's expectations and concerns with respect to such equipment, and to help demonstrate to the FAA that the modifications made to the system would be effective. Transcript at 501-02, 544. In its response to the DIs issued subsequently, Westinghouse submitted that nearly all of the areas questioned by the OCD team had been corrected by the time DIs were issued and responded to.[foot #] 9 Protest File, Exhibit 41; Transcript at 690-704. 47. In responding to the Government's DIs, Westinghouse included the types of information that it anticipated the Government would want to see, including explanations of the cause of problems and their resolution, schematics and specific details where appropriate. Protest File, Exhibit 41.[foot #] 10 Westinghouse did not provide, however, evidence of regression testing, adverse trend analysis, test data, or software code, for the simple reason that in the numerous discussion held before OCDs, the Government had never suggested that such data should be submitted. In addition, Westinghouse expected that the Government evaluators would in evaluating the responses. Transcript at 544. ----------- FOOTNOTE BEGINS --------- [foot #] 8 Indeed, in its motion to dismiss the protest as premature, the Government represented that the results of the OCD were not conclusive as to possible contract award and that each discrepancy between the OCD results and the technical proposal would be discussed with Westinghouse. Respondent's Motion to Dismiss and Brief in Support Thereof, GSBCA 11403-P. [foot #] 9 The remaining areas requiring correction, and issues raised in the January 1992 DIs, in March 1992, prior to the hearing. Transcript at 702- 07. [foot #] 10 The Government never compiled the Westinghouse DI responses in a single protest file exhibit. Thus, responses are scattered throughout the Westinghouse proposal, Exhibit 41, and some portions of the DI responses appear not to have been included in even that exhibit. For convenience, Westinghouse compiled all the responses in Exhibit 143 of its trial notebook. ----------- FOOTNOTE ENDS ----------- The Denro 400D OCD 48. The Denro 400D OCD was scheduled to be conducted from August 20 through August 23, 1992. Protest File, Exhibit 151. On August 19, Denro's in-house attorney contacted DECCO to request a short delay to permit corrections to be made to an audio switching failure in the 400D model that had been prepared for the demonstration. Respondent considered this request and denied it on the ground that to grant an extension would be unfair to the other vendors who had gone ahead as scheduled. As a consequence, Denro elected to lose the first day of the OCD while it repaired its system. Transcript at 364-65. 49. Once underway, having forfeited the first day of the demonstration, the 400D unit lost another full day due to what has been characterized by the Government members of the OCD team as loud, and potentially harmful, noises in the headsets. Transcript at 1037. As Denro points out, however, the particular volume level in issue was measured at -19 dbm at the OCD by Denro. Id. at 387, 1282, 1286. This volume level is more than two times lower than that which the FAA specified to be fielded by Denro under its Phase 1A contract. Denro's Supplemental Trial Exhibits D-24 and D-25. The Government was reluctant to proceed with Denro's proposed solution for working around this problem because it involved foregoing the use of headsets, which was where the noise was heard. 50. As a result of the truncated time for demonstration of features, at the conclusion of the OCD, Denro had fully demonstrated only 40 of the 118 items to be assessed. Nearly all of the 40 items that Denro managed to "fully demonstrate" were s h o w n o n t h e l a s t h a l f - d a y o f t h e demonstration.[foot #] 11 Transcript at 385-86, 1434. Of the 78 items that received an "N" for "not demonstrated," some 48 were never even attempted for lack of time. Protest File, Exhibit 58. 51. In a letter dated September 20, 1991, Denro followed up this OCD with its explanation of problems encountered in demonstrating the 400D. Denro advised the Government that it would submit changes to its technical proposal as a result of these problems in the following areas: ----------- FOOTNOTE BEGINS --------- [foot #] 11 Denro suggests, in its post-hearing brief, that the ability to demonstrate some 40 items on the last half- day, if anything, shows that it had the ability and capability to meet the PD requirements. ----------- FOOTNOTE ENDS ----------- Protest File, Exhibit 58. Some of these changed functions involve problems similar to those observed at the Litton OCD, and which Litton was apparently able to remedy almost immediately. See finding 42. Denro's expert, an electrical engineer, testified that the deficiencies identified in the 400D at OCD were basically minor in nature. Transcript at 1601. 52. Although not as mature as the 400 unit, Denro's 400D system is a COTS system in the sense that the unit has been produced and sold to different customers. Transcript at 360. Post-OCD Events 53. After completing OCD's on all four systems, the TET prepared a report and revisited the technical evaluations, concluding that as a result of the demonstrations, many of the technical area subitems[foot #] 12 that had previously been evaluated as must be downgraded to . In reevaluating, the TET proposed, rather than to the scores for subitems, . [foot #] 13 Protest File, Exhibit 15; Transcript at 181. The rationale was that if a significant feature did not work the entire technical area could be rendered nonfunctional. The TET also assigned a schedule risk rating to the various functions and features that were individually rated. Risk was assigned as low if the TET considered that the item could be fixed by the time of a period of . If the correction would require , a moderate risk rating was assigned; a high risk rating was given if an element was considered to require more than to fix. Transcript at 54-55. ----------- FOOTNOTE BEGINS --------- [foot #] 12 These subitems generally corresponded to paragraphs or subparagraphs of the purchase description specifications. [foot #] 13 The technical area team leader testified that the evaluators had been initially instructed not to . Transcript at 181. After turning in a technical evaluation report to the SEB, however, the SEB requested that . The TET was never instructed to "do anything" with other than enter them on the form, however. Id. at 182. __ ----------- FOOTNOTE ENDS ----------- 54. Following the conduct of the OCDs, the Selection Evaluation Board (SEB) met on October 8, 1991, to discuss the next course of action. They discussed the possibility of forming a new competitive range as a result of the perceived inability of both Denro and Westinghouse to validate their proposals at OCD. After much discussion, the SEB decided to proceed with a round of written discussion items with regard to systems tested at OCD. This was designed to permit a reevaluation of technical requirements as required under the TEP. The SEB could not conclude that the proposals "couldn't get better" without considering responses to the outstanding issues. Thus, the SEB did not believe it could determine at that point that the proposals did not have a reasonable chance for award. Protest File, Exhibit 159; Transcript at 1176-77, 2060-68. 55. At that same meeting the SEB decided to delete the term "mandatory" from Section M of the RFP. The provision of Section M requiring that each proposal" must meet all mandatory requirements" was revised to indicate that each "proposal must address all solicitation requirements." Protest File, Exhibits 8, 159. This amendment was issued on October 17; offerors were given until November 8 to submit revisions to their proposals. None of the vendors submitted changes in response to amendment 008. Transcript at 2002-03. 56. Considerable discussion was devoted at the October 8 meeting of the SEB to the form and content of DIs to be issued. By letters dated October 17, 1991, respondent sent DIs to Denro concerning its model 400 and DIs concerning problems at the OCD of the model 400D. DIs were addressed to Westinghouse with respect to the unit it demonstrated at OCD. Finally, DIs were directed to Litton with respect to issues concerning its OCD. Protest File, Exhibits 97, 114, 124, 139. The DIs related to features that were graded as "N" or "not demonstrated" at the OCDs. Id. 57. Responses to the DIs were received from the offerors on October 31, 1991. Protest File, Exhibits 41, 96, 113, 123. 58. The TET reviewed the responses to the DIs and prepared the "Technical Evaluation Team Final Report" on November 20, 1991. That report was reviewed by the SEB in a meeting held on November 26. The SEB was not satisfied with the report and directed the TET chairman to revise the report to include additional information to support the findings and conclusions reached by the team. Transcript at 1049, 2005-07. 59. Another meeting of the SEB was convened on January 7, 1992. At this meeting the discussions focused on the revised technical report, the scoring of technical equipment, the issuance of requests for best and final offers (BAFOs), and the need for additional DIs to the vendors. The SEB directed the TET to issue additional DIs to the vendors for all items that had been marked "F" or fully demonstrated at the OCDs, but for which the TET had noted reservations. This was important especially where a weakness was remarked upon in the technical report. The SEB was concerned about the propriety of eliminating competitors from the competitive range without affording them the opportunity to address deficiencies. Protest File, Exhibit 159. 60. The team leaders also explained their scoring methodology to the members of the SEB. Transcript at 2025-26. Again, where any single item of importance in a sub area was scored as . The SEB requested the TET to as well. When scores were , disqualified vendors received technical scores of between . After some discussion, however, the TET's approach prevailed and the [protesters's] technical proposals were evaluated as unacceptable. Protest File, Exhibit 159. 61. The SEB also considered the question of sending BAFO letters to all of the offerors. They were concerned about this course of action because the two Denro and the Westinghouse offers were rated as technically unacceptable based upon the TET's report of its evaluations. Nonetheless, the SEB preferred to avoid a competitive range of one. The SEB thus tentatively decided to send BAFO requests to all offerors and provide yet another opportunity to submit additional data in support of proposals. Protest File, Exhibit 159. 62. Ultimately, the BAFO letters were not sent because the SEB became concerned that this course of action could be deemed technical leveling. That is, the SEB was concerned that successive rounds of discussions would be tantamount to an effort to bring the deficient offerors up to the level that Litton had already achieved. Protest File, Exhibit 159. 63. Prior to the next SEB meeting, a limited number of DIs were prepared, addressing the items which had been marked as fully demonstrated at the OCDs, but which the OCD team regarded as performing inconsistently. Transcript at 1151-52. The hope was that if more complete answers were provided than had previously been submitted from Denro and Westinghouse, discussions potentially might be reopened. Transcript at 2100. Accordingly, DIs were issued to Denro on the model 400, DIs were addressed to Westinghouse, sent to Litton, and DIs were issued with respect to Denro's model 400D. Protest File, Exhibits 93, 120, and 134. 64. The TET reviewed answers submitted in response to the January DIs, but did not modify any technical scores. Protest File, Exhibit 149. 65. Again, in preparing its final evaluation, the TET assigned scores of to an entire technical area whenever one sub-item in that area considered to be essential to the air traffic control function was scored . Thus, scores were not . Protest File, Exhibit 149. 66. At a meeting held on February 6, to discuss the results of the second round of DIs, the SEB decided to rescind its earlier determination to send BAFO letters to all offerors. Instead, Denro and Westinghouse would be eliminated from the competitive range. A BAFO letter would be sent only to Litton. Protest File, Exhibit 159. Notification Letter on the Denro Model 400 67. By letter dated February 14, 1992, DECCO informed Denro that its offer of the model 400 switch had been eliminated from the competitive range because the proposal "failed to comply or verify compliance with the RFP." The letter also noted that responses to discussion items were "insufficient to permit a comprehensive technical assessment and verification of the causes of the problems or the proposed solutions." Finally, the letter listed specific areas deemed technically unacceptable in terms of compliance with the purchase description set forth in the RFP. These consisted of: Protest File, Exhibit 28. 68. At the OCD, several "Ns" were issued for areas, where Denro had implemented specifications with unchanged wording from the prior procurements in the same manner as in previously fielded systems. It learned at OCD that the Government had a different interpretation. See, e.g., Transcript at 203-305, 210-11. Denro agreed to modify such items in accordance with the Government's interpretation, as explicated at OCD, and explained in its DI responses how this would be accomplished. Protest File, Exhibit 96. 69. With the exception of (a), all of the items enumerated in the letter were the subject of DIs issued either in October or January. Denro responded to each DI with an explanation of how it would modify or correct the system to ensure compliance with the Government's interpretation of the purchase description. Protest File, Exhibits 92, 96. 70. The TET reviewed Denro's responses and uniformly rejected each one as providing insufficient information to permit the TET to assess the validity of the proposed modifications. This was so regardless of how simple or complex the item might have been. As a result, virtually all of the items marked as not demonstrated at the OCD received a score of and many were deemed to be high risk in terms of impact on the schedule. Protest File, Exhibits 49, 156. 71. Both Denro's program manager, a degreed engineer with considerable experience working with this unit at Denro, transcript at 195-96, and Denro's expert [describe credentials] agreed that the majority of the fixes required to achieve compliance with the purchase description were minor in nature and did not present a risk of introducing errors in other parts of the system in their implementation. Some items queried in the DIs, such as had actually been rectified at the OCD, but apparently had not been so noted by the Government. Id. at 254, 256. Many other items required only a small modification to COTS hardware or a minor adjustment to software code involving only a few lines. Id. at 203-39, 257-312. Notification Letter on the Denro Model 400D 72. A similar notification letter was issued, also on February 14, 1992, informing Denro that its 400D model had been eliminated from the competitive range of the procurement. Again, Denro's responses to DIs were considered generally insufficient. The letter listed the following instances of failure to comply with the RFP: Protest File, Exhibit 29. 73. Denro's program manager and its expert, an electrical engineer with more than twenty years of technical experience with variety of data and voice communications systems, see Transcript at 1552-70, testified about the deficiencies and discrepancies identified at the OCD and addressed in discussion items. Denro's expert stated that in his opinion the statements proffered by Denro in responding to discussion items on this model were sufficient "for a person who is experienced in distributed systems to be able to make a rational judgment that the changes were minor in nature." Id. at 1602. Notification Letter on the Westinghouse Proposal 74. Also by letter dated February 14, 1992, DECCO informed Westinghouse that its proposal had been eliminated from the competitive range of the procurement because it failed to comply or verify compliance with the RFP. The following specific areas were listed: Protest File, Exhibit 30. 75. All of the items enumerated in this letter were the subject of DIs issued either in October or January. Westinghouse responded to the DIs with an explanation of how it had or would modify its system to achieve compliance with the purchase description as interpreted by the Government. In addition, Westinghouse provided Protest File, Exhibit 41. 76. The TET reviewed Westinghouse's responses and uniformly rejected each one as providing insufficient information to assess the validity of the proposed adjustments to the system. This was so regardless of how simple or complex the matter giving rise to the query might have been. As a result, virtually all of the items marked as "not demonstrated" at OCD were given a score of zero and many were deemed to be high risk in terms of impact on the schedule. Protest File, Exhibits 149, 155. 77. Two Westinghouse engineers, both with considerable experience in voice switching systems[foot #] 14, and Westinghouse's expert engineer[foot #] 15, agreed that most of the issues raised in these discussion items involved slight adjustments to the system and a minor amount of effort to correct the problems. Because of the modularity of the system, most software changes would involve few lines of code and would not impact other modules in the system. These witnesses offered persuasive testimony that the nature of the system is such that by isolating changes to individual functional areas, changes are unlikely to have an adverse impact elsewhere in the unit. Transcript at 666-67, 1747-49, 1753-54, 1794-95. 78. In assessing the issues raised in the post-OCD DI's issued to Westinghouse, the Westinghouse expert witness credibly testified that in no case was a major amount of redesign required to correct a problem. Transcript at 1743-45. In his opinion, the conclusions reached by the technical evaluators were unreasonable: I don't know how else to say it except that I don't understand how they could come up with their conclusions. I think that, in most cases, these issues were -- a lot of them were very minor and should not, in any respect, be given a markdown; particularly, they shouldn't be given a high risk. I think in most cases they are very simple changes. As I say, even the most significant one, I would categorize as a ----------- FOOTNOTE BEGINS --------- [foot #] 14 The Westinghouse manager of switching and control systems has a degree in electrical engineering and approximately ten years of engineering experience in such systems and related technologies. Part of his experience included a stint with Arinc, during which he worked on the ICSS program. Transcript at 470-75. Another Westinghouse engineer, also with a degree in engineering science, had approximately fifteen years experience in software engineering for voice switching systems. [foot #] 15 The Westinghouse expert has an undergraduate and doctoral degree in electrical engineering. His specialty is in computer applications of electrical engineering and, in particular, in telephone switching systems and telephone transmissions systems. He has authored a book entitled Digital _______ Telephony, covering all aspects of digital technology for _________ telephone systems. Transcript at 790-95. ----------- FOOTNOTE ENDS ----------- moderate change. I don't understand how they could be given the risk assessment they were given. Id. at 1758. SSA Briefing 79. The decision to exclude the offers received from Denro and Westinghouse from the competition was briefed by the contracting officer to the source selection authority (SSA). His approval was not deemed necessary, and he was not requested to ratify the decision. Pricing was not addressed at the briefing, although it had been discussed on several occasions previously. . Protest File, Exhibit 158. Discussion Both Denro and Westinghouse challenge the contracting officer's determination excluding their proposals from the competitive range of the ICSS-1B procurement as improper. They contend that under the circumstances shown to have occurred in this procurement, decision contravenes the legal standards traditionally applied by the Board and the Comptroller General. In particular, the decision to exclude these offerors cannot withstand the stricter scrutiny given to such determinations when the result is to leave only one offeror remaining in the competitive range. The applicable legal principle is well stated in Falcon Systems, Inc., B-213661, 84-1 CPD 658, where the General Accounting Office (GAO), overturning such a decision, observed: [A] proposal must be considered to be within the competitive range unless it is so technically inferior or out of line as to price as to render discussions meaningless. This is particularly the case if there otherwise would be only one offeror remaining. Thus, while the initial determination of whether a proposal is in the competitive range, particularly with respect to technical considerations, is primarily a matter of administrative discretion which will not be disturbed by our Office absent a clear showing that the determination lacked a reasonable basis, we will closely scrutinize any determination that results, as here, in only one offeror being included in the competitive range. Id. at 3 (citations omitted). Accord, Birch & Davis International, Inc. v. Agency for International Development, GSBCA 11643-P (Feb. 27, 1992), 1992 BPD 63; Apt Corp., GSBCA 9237-P, 88-1 BCA 20,411, 1987 BPD 284. The major flaws in the procurement leading up to this defective determination lay in the conduct of the OCDs, the re- evaluation of technical scores based on OCD results, and the conduct of virtually meaningless post-OCD discussions with Denro and Westinghouse, in which differing standards of credibility as between protesters and Litton were applied to similar type responses to discussion questions. The Conduct of the OCDs Westinghouse, in its protest of the adverse competitive range determination, revived its earlier protest of the conduct of its OCD. In essence, the objection to the OCD was that the approach adopted by the OCD team was never articulated in the RFP documents so as to apprise Westinghouse of the nature of the test that would be conducted. Indeed, Westinghouse deemed the exercise to have been an "ambush." The principal difficulty manifested at the OCDs was that all offerors had anticipated, based on the information set forth in the RFP and in Attachment J-6, that the OCD team would passively observe while the offeror demonstrated the functionality of its equipment. To this end, all offerors had prepared a script, which was ultimately scrapped, by which the selected purchase description features would be exhibited to the Government. The Government's approach thus presented some difficulties for all three at the outset of the OCD process. Findings 34, 40, 43. The members of the OCD team affirmed in their testimony that they viewed the OCD as a form of quality assurance test. They were looking for systems that could be "wheeled out the door" and installed in the field.[foot #] 16 Finding 31. The ----------- FOOTNOTE BEGINS --------- [foot #] 16 Although an appealing idea, these expectations were not necessarily realistic under the circumstances. Nominally described as "off-the-shelf" items, the ICSS systems are the equivalent of a roomful of interrelated equipment, as illustrated by Westinghouse Hearing Exhibit 1. This is a picture of the Westinghouse ICSS equipment used for the OCD. Each system consists of a number of position consoles, a switching cabinet, a computer and other peripherals and accessories. Although similar items are indeed fielded at various FAA and other air traffic control facilities, it is uncontroverted that this procurement involved some tailoring of each offeror's existing equipment. It is doubtful that spare systems of this size would be routinely maintained at the manufacturing facility for demonstration purposes. Thus, it is not altogether surprising that, in assembling and preparing equipment for the OCDs, the offerors put (continued...) ----------- FOOTNOTE ENDS ----------- technical evaluation area leader, a member of the OCD team, suggested that he believed the team's approach was justified by the purchase description's provision for quality testing, which explicitly provides for "dynamic exercise" of the equipment, and sets forth a pass/fail standard. The difficulty is that this is not a provision cited in Attachment J-6, which lists the purchase description items to be tested at OCD. Nor does the language of the RFP in any way communicate this intent to the offerors. A review of the solicitation language discussing the OCD reveals that the expectations of Denro and Westinghouse, that they would follow their OCD plans and operate the equipment themselves, with the Government witnesses as passive observers, were entirely reasonable. Section M states that "[t]he offerors shall conduct an . . . OCD. . . . The OCD provides the offeror an opportunity to demonstrate to the Government the adequacy of the technical information contained in the proposal." Finding 22. Attachment J-6 contains several similar statements. Findings 22, 24. A fair reading of these provisions leads quite reasonably to the conclusion that the offeror, not the Government, would demonstrate the system's functions. This is particularly the case where Government hands-on testing is provided for elsewhere. Based on this expectation, we give some credence to the proposition that the surprise element may well have contributed to the poorer than expected performances. In criticizing the conduct of the OCDs, protesters are not suggesting that it is improper for an agency to require such procedures.[foot #] 17 They are, however, quite properly objecting to the process as it unraveled here. There is convincing testimony that had Westinghouse realized the Government's intent it would have prepared differently for the OCDs. Under the circumstances, however, it would have required ----------- FOOTNOTE BEGINS --------- [foot #] 16 (...continued) together a system that would demonstrate the functions and features called out in Attachment J-6, but which might not otherwise have been fully compliant with the entire purchase description, particularly since a controlled exercise was anticipated. [foot #] 17 Litton points out that the Federal Information Resources Management Regulation (FIRMR) prescribes the use of such capability and performance validation techniques as are feasible and necessary in procurements of automatic data processing equipment (ADPE) to reduce the risks associated with such acquisitions. 41 CFR 201.24-215 (1990). Of course, no one is contesting the propriety of conducting OCDs, or of the Government's need to verify the technical proposals. The issue raised by protesters is the inherent unfairness of the way in which the OCDs were conducted, and the subsequent failure to give any credence to proposed technical solutions regardless of the significance of the technical issues at hand. ----------- FOOTNOTE ENDS ----------- "Delphic powers" to have anticipated, based on the solicitation provisions, what actually occurred. Cf. Compuware Corp., GSBCA 8869-P, 87-2 BCA 19,781, 1987 BPD 55. This situation is readily distinguishable from the issue raised in Systemhouse Federal Systems, Inc., GSBCA 10409-P, 90-2 BCA 22,738, 1990 BPD 51, a case relied upon by respondent and intervenor in opposing these contentions. In Systemhouse, the protester wanted to substitute an entirely different software package in demonstrating its proposed database management system, contrary to an express requirement that offerors shall demonstrate the hardware and software proposed and only that hardware and software. Id. at 114,144-45, 1990 BPD 51, at 5. This led the Board to observe that the Government quite reasonably is entitled to test the system it is buying and not something else.[foot #] 18 Here, the offerors intended to demonstrate the system they would ultimately offer, at least to the extent required in Attachment J-6. Notably, however, according to that document, the Government was testing only selected paragraphs of the purchase description, and not the entire panoply of features in the system to be purchased. Protesters could not have anticipated in this case that the Government expected to see a "fully compliant" system that it could "roll out the door" and start using at OCD. This is particularly the case since the solicitation language is not mandatory -- that is, the OCD is not stated as a hurdle that must be passed to be considered for award.[foot #] 19 More fundamentally, however, the OCD is nowhere characterized as the pass/fail exercise it ultimately became. Indeed, the Government has steadfastly maintained that OCDs were not intended to be a "test" at all. The conduct of the OCDs, although questionable under the circumstances, is not by itself the driving factor in these protests, however. Even Westinghouse recognized and expected, in withdrawing its prior protest without prejudice, that defects or misunderstandings in the OCD process could be cured or resolved through the subsequent conduct of adequate discussions. The discussion process, and the concomitant post-OCD technical evaluations in which protesters ----------- FOOTNOTE BEGINS --------- [foot #] 18 It should be noted that in the instant matter, substitutions were not altogether prohibited, in contrast to the solicitation in Systemhouse. The FAA solicitation language ___________ requires demonstration of a "substantially similar" system to that proposed. It also discourages breadboarding and the like, but does not altogether prohibit such techniques. Finding 24. [foot #] 19 It is, of course, perfectly possible, if any agency wishes, to make it mandatory to pass an OCD to be eligible for award. See Systems, Terminals & Communications Corp., GSBCA ___ __________________________________________ 10578-P, 90-3 BCA 23,043, 1990 BPD 151. Respondent did not use such language in this solicitation, however. ----------- FOOTNOTE ENDS ----------- were never given a meaningful opportunity to recover from the OCDs, however, were equally flawed. Conduct of Evaluations and Discussion Process Having suffered, by their own testimony, severe disappointment in the OCD results of the Denro and Westinghouse systems, the evaluators turned to post-OCD evaluations. This exercise, which was performed during the period following completion of OCDs, and which ended shortly after submission of responses to the initial round of post-OCD DIs, resulted in virtually all of the functions rated as "not demonstrated" at OCD receiving numerical scores of zero. This was coupled with assessments of "high risk", meaning that, in the judgment of the evaluators, a fix would take more than eleven months to complete. Under the facts established in this case, it is clear that the conduct of post-OCD discussions was an exercise in futility.[foot #] 20 Virtually every response, proposal page change, and explanation was rejected out of hand for failure to include "adequate substantiating information" such as test data, regression studies, schematics, and the ----------- FOOTNOTE BEGINS --------- [foot #] 20 It is clear that the OCDs jaundiced the evaluators' opinion of the abilities of these offerors to produce a compliant system. The team leader for the technical evaluation stated, for example: The post-OCD DI's, as I said before, were intended as a way to clear up any ambiguities or any misunderstandings -- that arose as part of the OCD. They were not an attempt to allow an offeror to fix his system. I mean his chance to validate the system was the OCD. I mean his validation was done or not done. Transcript at 1044. In explaining his rejection of the DI responses this technical area team leader declared: Well, the problem with this whole issue is that Denro did not validate their technical proposal as required in the OCD. Transcript at 117. Accord Transcript at 137 (did not accept DI ______ responses because "Denro did extremely poorly at the OCD"), 167 (DI response indicates they have identified a solution, but "again, the OCD showed they could not" perform this function). ----------- FOOTNOTE ENDS ----------- like.[foot #] 21 The explanation proffered by the Government technical evaluators was that after witnessing the unsatisfactory performances of the Denro and Westinghouse systems, the evaluators were no longer content with written explanations of technical fixes -- something more was required to persuade them that the systems would work.[foot #] 22 They, and apparently the SEB members, were reluctant to come out and ask for such back-up information, however, for fear of technical leveling. This concern was unfounded. The concept of technical leveling precludes the Government from using the discussion process to assist an offeror to improve its proposal by pointing out weaknesses through successive rounds of discussions or, alternatively, from eliminating "technical discrimination" such that price dominates in the selection decision. E.g., Tidewater Consultants, Inc., GSBCA 8069-P-R, 85-3 BCA 18,459, 1985 BPD 86 (offerors received explicit suggestions about how they might improve their proposals). Neither situation exists in this case. Lack of information is a "deficiency" not a "weakness" that may properly be addressed through discussions requesting additional information. See System Automation Corp., GSBCA 8204-P, 86-1 BCA ----------- FOOTNOTE BEGINS --------- [foot #] 21 [foot #] 22 Although undoubtedly well-intentioned, this concern about the ability of the offerors to field systems that would work seems far out of proportion to the matter at hand. The FAA itself recognized that Denro's 400 unit is a viable system -- to suggest at this point that the model cannot be made to work, or cannot be made to work in a reasonably short period of time because of the various issues raised at OCD, strains credulity. ----------- FOOTNOTE ENDS ----------- 18,654, 1985 BPD 158.[foot #] 23 Asking Denro and Westinghouse to provide software charts, regression test results or similar documentation would hardly be the equivalent of offering advice on how to improve their respective switches. Even absent the additional test data that the TET desired to convince it that proposed revisions and solutions were viable, we are persuaded by the testimony of the experts in this proceeding that the detailed explanations and such supporting data as were provided were sufficient in many cases to have demonstrated that the response was adequate. E.g., finding 78. Moreover, although we agree that Westinghouse could not use to "validate its system in the same manner as intended by the OCD, should not have been totally disregarded by the TET. The deserved some weight as providing support for the validity of solutions proposed and implemented by Westinghouse. It is clear, however, that this supporting documentation was completely ignored by the evaluators. This brings us to the other point made by the protesters -- the unreasonableness per se of the evaluation scoring, in which items not demonstrated at the OCD were automatically assessed as unacceptable and high risk, regardless of how trivial or minor the malfunction may have been, and without regard to any responses offered with respect to their correction. As Denro points out, even the lay person must view with skepticism the pronouncement of the evaluators that it would require from seven to eleven months to make a hold button wink, to turn off or to fix ----------- FOOTNOTE BEGINS --------- [foot #] 23 See also Falcon Systems, Inc., 84-1 CPD 658, ________ ____________________ at 8, which is closely on point: [W]ith the elimination of [protester], only one firm remained in the competitive range, thereby effectively creating a sole-source procurement. We think that under such circumstances, it is incumbent upon an agency, in the interest of maximizing competition, to keep an informationally deficient proposal in the competitive range unless further discussions based on that proposal would be truly meaningless. . . . We are not saying that an agency may never reject a proposal for informational deficiencies when to do so would leave only one offeror in the competitive range. However, such instances are rare, and the deficiencies would have to be so irremedial that the agency had no reasonable alternative but to proceed with a sole-source procurement. ----------- FOOTNOTE ENDS ----------- .[foot #] 24 The TET's complete disregard of even uncontroverted information provided in response to discussion items was not reasonable. As the Board has noted previously, the provision of some information, even if not regarded by the agency as enough to merit a high score, may be enough to invalidate a finding of unacceptability. See National Capital Systems, Inc., GSBCA 10823-P et al., 91-1 BCA 23,525, at 117,961, 1990 BPD 390, at 16. Viewed in light of the totality of the evidence in the record of these protests, the evaluation scores assigned to the Denro and Westinghouse proposals following OCDs and the post-OCD discussion process fail to accord with the methodology propounded in the technical evaluation plan. It is simply not credible that every non-demonstrated item connoted a technical approach that was "seriously flawed" and would present an "overwhelming" task to achieve an improvement. Nor is it credible, in light of the evidence in this record, to have concluded that the systems as a whole were so deficient that it would present an "overwhelming" task to improve them. This is not consistent with the Government's own statements recognizing that feasible systems already exist and are fielded, nor is it consistent with the FAA's position that the systems it seeks are not developmental and require no technological breakthroughs. Finally, the decision to assign a score of to an entire area based on the assessment that a rating of on one or two sub-items, if those items were sufficiently critical to the functioning of the system in the eyes of the evaluators, is also suspect in this case. It is clearly an unreasonably severe approach when protesters were virtually precluded from improving their ratings on these items through post-OCD discussions. Competitive Range Decision Respondent and Litton rely heavily on a Comptroller General decision to support their contention that the competitive range determination was proper in this case. This decision, Aydin Corp., B-224354, 86-2 CPD 274, also involved a competitive range determination resulting in only one offeror remaining in contention for award. There, the protester alleged that 1) the Government had improperly considered the results of an OCD on a pass/fail basis; 2) that the seven deficiencies noted at OCD were insufficient to support the finding that the proposal was technically unacceptable; and 3) that the Government failed to conduct meaningful discussions on deficiencies. Rejecting these contentions, GAO stated: While we have criticized the strict application of pass/fail test criteria that lead to the automatic and final exclusion of a potentially acceptable proposal, . . . ----------- FOOTNOTE BEGINS --------- [foot #] 24 See Transcript at 101, 106, 119. ___ ----------- FOOTNOTE ENDS ----------- these cases generally involve situations where the offeror has been unable to demonstrate compliance with only one out of a number of mandatory requirements and is eliminated from the competition solely for that reason. Here, the technical evaluation board identified seven specific deficiencies in Aydin's system, as demonstrated, and determined that these deficiencies rendered the proposed system technically unacceptable. Under these circumstances, we do not think the agency's actions amounted to the conduct of the OCD on a pass/fail basis. We turn then to the question of whether the deficiencies identified by the agency in Aydin's system as a result of the OCD are sufficient to justify the determination to exclude the protester from the competitive range. . . . We have found that while one or more deficiencies may not, of themselves, be sufficient reason for rejecting a proposal, it is possible that as a totality, they can justify a contracting agency's decision to eliminate a proposal from the competition because major revisions would be required to make the proposal acceptable . . . . This is essentially what happened here, and based on our review of the record, we find that the [agency's] conclusion was reasonable, particularly since several of the deficiencies were major ones that were not easily correctable. Id at 4, 9. Respondent and Litton maintain that, in a similar vein, the totality of the "massive" numbers of deficiencies noted at the OCDs fully justified the decision to exclude the protesters from further competition. As protesters point out, however, Aydin is factually distinguishable in several important respects. First, the procurement was for a battleground command and control station, and the deficiencies involved non-transportability of the system, incompatibility with a wide range of Marine Corps radios, and inability to survive enemy attack. The opinion states that correction of these deficiencies would have necessitated major revisions in Aydin's proposal. In contrast, the deficiencies in the Denro and Westinghouse proposals, following discussions,[foot #] 25 were largely informational. Findings 70, 76. The preponderance of the testimony in the record establishes that contrary to the contentions of the TET team leader, the technical fixes, while numerous, were not particularly difficult, and were not of a major significance to the overall functionality of the equipment. Findings 71, 73, 77-78. The concerns of the technical area team leader were largely with proof that the proposed modifications would be effective. He was not concerned with the specific engineering imperfections he had discerned in proposed solutions. The fact that Litton was able to fix many of the same or similar-type problems on the spot suggests, if anything, that the TET's conclusion that such problems would present major, or high, schedule risks for Denro and Westinghouse was questionable indeed. The record in these protests establishes that the evaluation of proposals following the OCDs was unfairly and arbitrarily conducted, and that the discussions held with the offerors were inadequate and meaningless. The "massive" deficiencies cited by the FAA and Litton as justifying the competitive range determination appear to be attributable as much to confusion over what the OCD was intended to encompass as to any real inadequacies in the systems themselves. The subsequent conduct of discussions, in which virtually every response was discounted, was, in essence, a sham. As such, the protests must be granted. Litton's Hearing Exhibit 3 Before turning to a discussion of the remedy requested by protesters, we are compelled to address a subsidiary issue raised by Litton at the hearing. This concerns a document, included by Denro on its list of trial exhibits, but ultimately introduced into evidence by Litton, as its hearing exhibit 3, which Litton contends is improperly in the possession of Denro and requires Denro's disqualification from the competition. The document appears to be a copy of a slide presentation prepared by Arinc in its role as a consultant to the FAA in connection with voice switching programs. The document is entitled "Terminal Voice Switching Program Orientation." Denro indicates in its post- hearing reply brief that the document was given to it by Arinc. The Government has been silent on the subject of how Denro might have come into possession of these materials. ----------- FOOTNOTE BEGINS --------- [foot #] 25 We are unimpressed with Litton's contention that Aydin stands for the proposition that the FAA had no _____ obligation even to conduct post-OCD discussions with protesters. In point of fact, the TEP expressly contemplates such discussions, finding 18, and the leader of the OCD team informed protesters that discussion items would be issued with respect to items not demonstrated satisfactorily at OCD. Finding 32. ----------- FOOTNOTE ENDS ----------- Counsel for respondent alleged at the hearing that the document contains procurement sensitive information, but has not pursued the matter in its post-hearing submissions. Litton urges that Denro's mere possession of this document, which Litton claims it has never seen, is an impropriety requiring Denro's immediate disqualification from this procurement. In support of this position, Litton relies upon several cases involving real and apparent conflicts of interest in which decisions of the contracting officer to disqualify a vendor have been upheld. See NKF Engineering v. United States, 805 F.2d 372 (Fed. Cir. 1986) (contractor hired contracting officer's former technical representative and thereafter made a substantial downward adjustment to offered price); Compliance Corp. v. United States, 22 Cl. Ct. 193 (1991), aff'd, 960 F.2d 157 (Fed. Cir. 1992)(table) (contractor engaged in improper business conduct by attempting to obtain proprietary information of a competitor). On the information available to us, we cannot reach such a conclusion. First, Litton's assertion that Denro has committed an impropriety in obtaining possession of these materials is entirely speculative. There is absolutely no evidence in the record that Denro came by this document illicitly.[foot #] 26 Second, the information concerning the ICSS 1B program appears to be largely obsolete, as the document in question is dated September 28, 1990, and appears to be a global briefing, discussing the program as it existed prior to the issuance of amendment 004, in March of 1991. This amendment substantially changed the quantities of the procurement and the FAA's approach to satisfying its program needs for voice switching equipment. Thus, although there are cryptic references to ICSS 1B funding and anticipated costs in the document, its relevance to the procurement as presently structured is problematic. As such, the possibility of any competitive advantage being conferred upon Denro is remote at best.[foot #] 27 The Remedy ----------- FOOTNOTE BEGINS --------- [foot #] 26 Moreover, the Government, which would be the party in the best position to make this contention, has not done so. [foot #] 27 In any event, if the document was erroneously, but innocently, given to Denro by an Arinc employee, it would seem that the fair solution, should the Government be persuaded that some competitive advantage may have been conferred, would be to provide the document to the other competitors to restore the "level playing field." The cases cited above involving disqualification include a showing, or reasonable inference, of improper behavior on the part of the recipient of the sensitive information. Such a showing is wholly absent here. ----------- FOOTNOTE ENDS ----------- Having disposed of this contention, we reach the matter of the proper remedy. Denro and Westinghouse both express concern that mere restoration of their proposals to the competitive range would result in less than meaningful relief. Westinghouse asks us to amend the FAA's delegation of procurement authority (DPA) to require that further technical evaluations 1) not give any weight to the results of the improperly conducted OCDs and 2) exclude the participation of individuals who participated or have knowledge of the OCDs. Both Westinghouse and Denro advocate a second round of OCDs, with the terms and conditions of that exercise set forth explicitly in advance. Litton opposes further evaluations and OCDs on the ground that such relief would unfairly deprive it of its deserved award based on its superior performance in the OCD. According to Litton, a rerun of the OCD at this point would eliminate any technical competition and would create a de facto auction of the ICSS-1B procurement. Thus, at most, we should, if inclined to grant the protests, award only attorney's fees and proposal preparation costs (starting with costs incurred after the OCDs) to protesters. We are not convinced, however, that a second round of OCDs would necessarily have the effect of nullifying technical differences in performance of the systems. Respondent, throughout the hearing, expressed the need for expedited treatment of this procurement, emphasizing the urgent need to get the ICSS 1B equipment up and running. Indeed, respondent's witnesses intimated that given the need to install this equipment rapidly, the only resort would be to award to Litton, since they believed the other offerors presented undue schedule risks. As Denro points out, however, even under expedited schedules presented at the hearing, the earliest installation of a system would be nine months from date of contract award. Thus, it need not necessarily present an undue schedule burden on the FAA to schedule another OCD, this time making it clear what the terms and conditions are. If the FAA is concerned about receiving fully functional systems, it may require a pass/fail test to insure that the systems are indeed compliant as the preponderance of the evidence in this record now reflects. We also agree with the concerns of Westinghouse that new evaluators should be assigned in order to ensure that the results of the original OCDs do not infect further consideration of the protesters' systems. Accordingly, we revise the FAA's DPA to require the restoration of the protesters to the competitive range of the procurement. In accordance with applicable statutes and regulations, the FAA shall decide whether to conduct new OCDs, under such expressly stated terms and conditions as the agency deems appropriate, and reevaluate based on those demonstrations, or whether to reevaluate based on existing information without reference to the results of the original OCDs. Either approach shall use new personnel.[foot #] 28 Decision The protests are GRANTED. The suspension of respondent's delegation of procurement authority lapses by its terms. ----------- FOOTNOTE BEGINS --------- [foot #] 28 Westinghouse has asked the Board to declare that its proposal costs expended in participating in the OCD be awarded to it as having been wasted. Given the relief ordered, however, this request may be premature. Accordingly, we defer this issue for resolution in any cost petition filed by Westinghouse with respect to this decision.