Description
JUSTIFICATION FOR AN EXCEPTION TO FAIR OPPORTUNITY 1. Contracting Activity: Department of Veterans Affairs (VA) Office of Procurement, Acquisition, and Logistics Technology Acquisition Center 23 Christopher Way Eatontown, New Jersey 07724 2. Description of Action: This proposed action is for a firm-fixed-price delivery order issued under the National Aeronautics and Space Administration (NASA) Solutions for Enterprise-Wide Procurement (SEWP) V Governmentwide Acquisition Contract (GWAC) for brand name Citrix software licenses, associated maintenance and technical support to include Citrix Technical Account Manager (TAM) services for both software and hardware, Citrix Managed Services as well as consulting and training services. 3. Description of the Supplies or Services: VA Office of Information and Technology (OI&T), Infrastructure Operations (IO) has a requirement for the renewal of maintenance support for existing brand name Citrix software licenses and hardware, Citrix analysis software, Citrix TAM services for both software and hardware, Citrix Managed Services as well as consulting and training services. Citrix software is currently installed throughout the existing VA virtual server infrastructure and provides virtual application and virtual desktop access to endpoint devices inside and outside the VA network. VA s current use of Citrix provides virtual secure accessibility to servers hosting Computerized Patient Records System (CPRS), Veterans Benefit Management System (VBMS), and Burial Operations Support System (BOSS) databases, as well as over 400 other VA applications published to users off existing VA Citrix data center locations. This allows VA to centrally deploy and manage software applications while providing secure, on-demand access to critical business resources accessed 24 hours a day, 7 days a week by internal and external employees, contractors, as well as key external partners such as Veterans Service Organizations. In order to ensure continuity of operations for these critical systems and databases, VA OI&T IO requires the renewal of 30,000 existing Citrix Virtual Apps and Desktops Premium Edition software licenses. Additionally, VA requires an optional 20,000 net new Citrix Virtual Apps and Desktops On-Prem Platinum Edition licenses in order to accommodate a potential surge in the number of remote users due to the ongoing COVID-19 pandemic. VA currently owns 110 Citrix Application Delivery Controllers that will require continued maintenance. VA requires brand name Citrix software licenses and maintenance for the 110 existing VA Citrix Application Delivery Controllers that are dedicated to Citrix application backend environment used for high availability in order to provide more than 95% uptime. The hardware maintenance will be a pooled capacity model to allow for scaling of capacity. In addition, VA requires the renewal and maintenance of 240 Citrix analysis and monitoring software licenses to monitor, manage, and troubleshoot end to end from the client to Citrix servers and Citrix Application Delivery Controllers. The associated maintenance support for both software and hardware includes downloads, minor software upgrades, troubleshooting features, and 24 hours per day, 7 days per week, 365 days per year technical support for all required Citrix software licenses and hardware. Finally, VA requires technical support which includes dedicated Citrix TAM services for both software and hardware. VA also requires consulting and training services to support the existing and new Citrix software licenses and hardware products installed throughout VA. This shall include full-time dedicated software and hardware TAM services to assist VA staff in ensuring high availability of the Citrix environment while utilizing the least number of servers. This avoids unnecessary cost in underutilized hardware and software and ensures business operations are not negatively impacted by any unscheduled downtime. The TAMs have access to key internal Citrix resources that only Citrix or an authorized reseller of Citrix products and services can provide and assign correct priorities to any technical case VA opens to mitigate system issues. Because VA implements only a subset of Citrix s hardware and software capabilities, customized training classes are required to teach employees and contract staff maintaining Citrix environments in order to better maintain and improve their operation for end users. It would be inefficient to train VA staff in Citrix features that VA will not be utilizing. Only Citrix Education can provide customized Citrix course content, including the ability to skip standard course content that public training centers are required to teach, which allows VA to reduce the time commitment of standard Citrix courses from 40 to 30 hours of training per attendee. Citrix Education provides training specific to the technology they offer and the curriculum stays current to the latest technology versions. They offer the ability to hold dedicated sessions to only VA staff and schedule training sessions according to the VA s needs. Only authorized Citrix resellers offer Citrix Education vouchers. The Citrix consulting services are required and provide for architecture, design, and implementation support for the 30 Citrix environments located throughout the continental United States at VA data center locations. The consulting and training services are procured via Citrix Consulting Vouchers which are redeemed as the training classes and consulting engagements are scheduled. For Citrix Managed Services, the Contractor shall provide specific Citrix monitoring, expert Citrix specific ticket support guidance and collaboration with existing Citrix consultants, support, and education contacts assigned to the VA 24 hours a day, 7 days a week, 365 days a year. VA requires Managed Services to initially support VA s 30,000 Citrix concurrent user base and requires optional tasks for additional Managed Services, which can be exercised monthly, in increments of 10,000, and up to 20,000 additional concurrent users per month to support COVID 19 surge efforts. Additionally, VA requires optional tasks of Citrix Consulting services to provide for architecture, design, and implementation support for the COVID-19 telework surge for both the on-premise and cloud Citrix environments located throughout the continental United States at VA and cloud service provider data center locations. Capabilities of Citrix Consulting services include support of architecture design and build, both on premise and on cloud, as well as rollout and end user support. Citrix Consulting services have Citrix specific knowledge on how to best deploy Citrix systems both on premise and on cloud as well as the knowledge to provide ticket optimization and end user support techniques. The consulting and training services are procured via Citrix Consulting Vouchers which are redeemed as the consulting engagements are scheduled. In summary, the Citrix software and hardware maintenance and support and managed services will help VA meet the challenges of supporting up to a potential of 50,000 concurrent Citrix users and managing and responding to the large number of tickets outstanding in the queue due to the COVID 19 surge of remote users. The Citrix Managed Services will also provide sustainment support for any future tickets as well as regular maintenance of increased infrastructure. Citrix Consulting will assist VA in optimizing the infrastructure to support the influx of remote users as VA needs have accelerated due to COVID 19. The base period of performance shall be March 29, 2021 through March 28, 2022 with one 12-month option period for the continuation of Citrix brand name software licenses, associated maintenance and technical support, including TAM services. In addition, there are optional tasks for Citrix Managed Services, consulting, training support, and additional software licenses that may be exercised at any time during contract performance as the need arises. The total estimated price of the proposed action is $ inclusive of all options and optional tasks. 4. Statutory Authority: Pursuant to Federal Acquisition Regulation (FAR) 16.505(a)(4), the following brand name requirement is being justified and approved as an exception to fair opportunity under the statutory authority of Section 41 U.S.C. 4106(c)(2) as implemented by FAR 16.505(b)(2)(i)(B), entitled Only one awardee is capable of providing the supplies or services required at the level of quality required because the supplies or services ordered are unique or highly specialized. 5. Rationale Supporting Use of Authority Cited Above: Based on market research, as described in Section 8 of this document, it has been determined that limited competition is available among authorized resellers for the required brand name Citrix software licenses, maintenance, technical support, and Citrix managed services. Only brand name Citrix items and services meet all of VA s requirements, specifically compatibility and interoperability with VA s existing virtual infrastructure and integration with critical business systems including CPRS, VBMS, and BOSS, support of personal Windows and Macintosh endpoints, two-factor authentication exemptions, and two-factor authentication with corporate workstations who connect through the VA Citrix Gateway. Due to Citrix s patents, copyrights, proprietary database structures, proprietary communication protocols, and proprietary software application programming interfaces, no other source is able to provide the software and support required for maintaining these high availability information systems. The Citrix software licenses provide the underlying core functionality necessary for application and desktop virtualization currently being utilized in support of VA s datacenter consolidation efforts. Additionally, VA s current remote access infrastructure, comprised of Citrix software licenses and hardware, is configured to only support Citrix software through Citrix Application Delivery Controller appliances deployed in VA s internet gateways. In response to the COVID-19 pandemic, VA OI&T was originally requested to support 500,000 teleworkers, 290,000 of which would be supported through the Citrix Gateway solution. It has since been reduced to support up to 50,000 concurrent users since concurrent licenses have not exceeded 25,000 since the summer of 2020. Use of any other brand software or hardware would result in serious interoperability and compatibility issues with the current infrastructure as the solution is reliant on custom web pages and scripts that are currently supported by Citrix. Specifically, the current Citrix infrastructure consists of over 5,000 servers spread across the VA enterprise both on premise and on VA Enterprise Cloud that can only work with Citrix software inclusive of the solution providing parsing ability for proxy server sessions to assist in troubleshooting, network analysis and end-to-end performance for Virtual Apps and Desktop sessions to include historical data. The Citrix solution provides the capability of enabling traffic visibility; inclusive of management software, which is based on user Application flow statistics in measuring and determining latency on the client side, network side, and application side of the connection. Citrix has proprietary constraints within its software. No other brand name software would be able to communicate with these servers. If another brand name software was to be utilized the current infrastructure would have to be completely replaced with a non-Citrix based solution which would require a similar number of servers to be procured or provisioned, along with all the associated software and remote access software. This solution would need to be deployed parallel to the existing Citrix solutions to ensure there was seamless coverage. The Government estimates that this effort would take over one year to design, implement and test before it would be available to support production user load and would cost in excess of $82M to restructure the entire infrastructure. This estimate is based on previous procurement history. Only Citrix is utilized for remote access to over 400 applications, which include CPRS, VBMS and BOSS. Users from non-VA furnished endpoint devices are required to utilize Citrix for remote access to these applications. Approximately 100,000 unique remote access users use Citrix on a monthly basis from non-VA issued endpoints thus avoiding the need for VA to purchase, deploy, and maintain an equivalent number of VA issued endpoint devices and associated software licensure for these users at a sizable capital and operational expense to OI&T. During the COVID-19 pandemic, VA added 5,000-7,000 Citrix remote access enabled users a day and has approximately 190,000 users enabled as of December 2020. The purchase of VA issued endpoints and associated software licenses for 100,000 new endpoint devices for remote users would cost VA in excess of $200M in duplicated cost. This duplicated cost estimate is based on previous efforts in obtaining VA issued endpoints and associated software licenses. VA uses Citrix Application Delivery Controller (ADC) appliances to satisfy several security and functional requirements. No additional Citrix ADCs are being purchased at this time. These appliances need to be maintained and are the basis for VA s utilization of the Citrix Virtual Apps and Desktops Premium Edition software licenses. Specifically, VA utilizes Citrix Application Delivery Controller appliances in its Trusted Internet Connection gateway in providing remote accessibility from internet endpoints to 15 internal data center locations across the enterprise since 2009 and two VA Enterprise Cloud Service Providers. Load balancing features must be able to provide specific monitoring probes for Citrix XML service and the ability to provide data to a VA Application Delivery Management system for real-time client and network latency metrics, historical reports, end-to-end performance data and troubleshooting, to ensure that individual server software or hardware failures do not negatively impact user access to Citrix from inside or outside the VA network. VA s existing Citrix brand name software licenses and associated hardware require technical support in order to retain the use of the software, for future software version upgrades, and to ensure that software and hardware issues are addressed in a timely manner. Citrix has proprietary rights to the software licenses and associated technical support. Therefore, the required associated technical support must be compatible with the existing software licenses. Only Citrix, or an authorized Citrix reseller, has the rights to perform the associated technical support. Any other source must have access to the proprietary rights to ensure all the associated technical support provided is properly configured to both the software and infrastructure. Without access to the proprietary rights, another source would not be able to push through updates. Due to VA s large, complex Citrix environment, customized training is required. Only authorized Citrix resellers offer Citrix Education vouchers. Standard Citrix Authorized Learning Centers (CALCs), offered by many Citrix Certified Instructors, are not permitted to customize proprietary course content. Citrix Education is the only authorized provider of Citrix Customized Training and is the only source that can offer the appropriate tailored training for VA, through a variety of methods, such as customized lectures and tailored hands-on labs to meet VA s unique technical training requirement in support of VA s Citrix environment. The tailored training, to include the lectures and hands-on labs that only Citrix Education can provide, will prepare VA staff to maintain and improve VA s complex Citrix environments in a manner that could not be achieved by taking generic, off-the-shelf training. Additionally, only through Citrix Education s customized training can VA reduce the length of a standard Citrix offering from 40 to 30 hours a week allowing staff to continue to support existing Citrix environments part time, even while in training. A Citrix training partner is required to teach the full 40-hour course exposing VA staff to material that they will not use as VA often uses a reduced subset of Citrix s product capabilities due to the uniqueness of VA s requirements. Managed Services provided by Citrix will allow immediate support for VA s 30,000 concurrent users with maximum flexibility to accommodate increases on a monthly basis during this unprecedented time of the COVID 19 pandemic. Only Citrix Managed Services provides for 24x7x365 coverage backed by Citrix Technical Support. Only Citrix Managed Services offer VA the capability of bridging Citrix Product Engineers, Professional Services, and Technical Support thereby reducing troubleshooting and remediation times. Citrix Managed Services ensure detailed familiarity of the environment from design through support with internal feedback loops which will allow for a proactive approach which focuses attention on key areas to identify and remediate potentially harmful issues prior to user impact. Citrix Managed Services provides direct lines of communication with Citrix Engineering for necessary hotfixes as well as Citrix Product Management. Direct access to Citrix technical support services is critical to the success of ensuring not only the expansion of VA s remote access infrastructure occurs timely without interruption but also in assisting VA in eliminating its ticket backlog and putting appropriate mechanisms in place to ensure tickets remain at a manageable level. Citrix Consulting services, which are only available through Citrix or its resellers, provide VA with the required Citrix specific knowledge on how to best deploy Citrix systems both on premise and on cloud as well as the knowledge to provide ticket optimization and end user support techniques. VA is currently utilizing these services for consulting and training support and requires continuation of these services in order to ensure expansion efforts are successful as there can be no interruption in remote access as critical VA staff, including clinicians, rely on the CAG environment to perform their duties in the midst of the COVID 19 pandemic. The impact of not renewing the Citrix software licenses and associated technical support would result in a loss of the ability for VA to migrate to a new operating system on endpoint devices being implemented with zero disruption to users, downtime, or loss of data provided by Citrix application and desktop virtualization solutions. Further, the impact would require VA to offboard, rehire, onboard, and retrain approximately 25 staff members all over again, which would impact the ability to support the expanded infrastructure and increase of tickets from the telework surge. If Citrix software licenses were not renewed, VA s entire remote access infrastructure would need to be re-engineered and revalidated or VA would have to purchase, deploy and maintain 100,000 additional endpoint devices. Failure to renew Citrix software licenses would impact 24x7 application accessibility from outside the VA firewall to key critical and business applications including CPRS, VBMS and BOSS utilized by a large number of medical, benefits and burial personnel daily. In addition, Citrix virtual desktops are currently a key component of VA s telework and continuity of operations strategy and is utilized daily for both remote network and system access needs of staff and in emergent situations when weather or other disasters impact abilities of VA personnel to report to physical office spaces. Thus, technical support of existing Citrix software licenses is vital to ensure the security and availability of VA s information technology infrastructure for over 160,000+ Citrix users, 100,000+ of which access Citrix remotely. If VA does not renew technical support on existing Citrix software licenses annually, VA will be required to purchase brand new licenses each year at a substantial cost in order to use the supported version. 6. Efforts to Obtain Competition: Market research was conducted, details of which are in the market research section of this justification. This effort did not yield any additional sources that can meet the Government's requirements. However, the undersigned VA technical representative determined that limited competition amongst authorized Citrix resellers is available for this brand name acquisition. In accordance with FAR 16.505(a)(4)(iii)(A)(1), this justification will be provided with the solicitation to all appropriate NASA SEWP V GWAC holders. Furthermore, in accordance with FAR 5.301, 16.505(b)(2)(ii)(D), and 16.505(a)(4)(iii), the award notice for this action will be synopsized on the Contracting Opportunities website and this justification will be made publicly available within 14 days of award. 7. Actions to Increase Competition: In order to remove or overcome barriers to competition in future acquisitions for this requirement, the Government will continue to perform market research to determine if there are any new products or maintenance services available in the marketplace that would enable future actions to be fully competed. 8. Market Research: As recently as December 2020, the Government technical experts reviewed similar software and technical support and determined that no other products or services would be fully compatible or interoperable with VA s existing virtual infrastructure consisting of Citrix hardware and software. The software reviewed by the Government technical experts included Microsoft Windows Virtual Desktop and Amazon Workspaces both of which provide similar capabilities, however, neither of these solutions are compatible or interoperable with the existing architecture established within VA necessitating the building of a parallel infrastructure if VA were to migrate from Citrix to one of these alternatives. Citrix has proprietary rights to the software licenses and associated hardware currently being used by VA throughout the virtual infrastructure. Given that only Citrix has the proprietary rights, there are no other authorized vendors that can provide the technical support services without voiding the licenses currently in use. Only Citrix, or an authorized Citrix reseller, has the right to provide Citrix license technical support renewals, and only they can provide the required training, consulting, and dedicated technical support. In January 2021, the Contract Specialist conducted market research to ascertain the extent of limited competition among resellers of the required software, hardware, and associated maintenance and technical support. Through the use of the Provider Look-up tool on the NASA SEWP V GWAC website, 16 Service-Disabled Veteran-Owned Small Businesses were identified as resellers of the required Citrix software, hardware, and associated maintenance and technical support. Therefore, limited competition is anticipated. 9. Other Facts: None. 10. Technical and Requirements Certification: I certify that the supporting data under my cognizance, which are included in this justification, are accurate and complete to the best of my knowledge and belief. Date: ________________________________ Service Line Manager, Signature: ____________________________ IT Operations 11. Fair and Reasonable Cost Determination: I hereby determine that the anticipated price to the Government for this contract action will be fair and reasonable based on anticipated limited competition. Additionally, NASA SEWP V has already determined the prices on contract to be fair and reasonable and more competitive pricing may result from competition. Finally, the successful quote will be compared with the Independent Government Cost Estimate. Date: ________________________________ Procuring Contracting Officer Signature: ____________________________ 12. Procuring Contracting Officer Certification: I certify that this justification is accurate and complete to the best of my knowledge and belief. Date: ________________________________ Procuring Contracting Officer Signature: ____________________________ Approval In my role as Head of the Contracting Activity, based on the foregoing justification, I hereby approve the acquisition of brand name maintenance support for existing brand name Citrix software licenses and hardware, Citrix analysis software, Citrix TAM services for both software and hardware, Citrix Managed Services as well as consulting and training services on an other than fair opportunity basis pursuant to the authority cited in Section 4 above, subject to availability of funds, and provided that the property and services herein described have otherwise been authorized for acquisition. Date: ____________ Signature: __________________________________ Head of the Contracting Activity Technology Acquisition Center Office of Procurement, Acquisition, and Logistics