The US Government is continuing to move rapidly to ensure US competitiveness in the area of Artificial Intelligence (AI). The FedRAMP Program Management Office (PMO) published the Emerging Technology Prioritization Framework (ETPF) in January 2024. The ETPF is designed to help accelerate the availability of FedRAMP accredited Gen AI cloud solutions for federal agencies and users. The FedRAMP PMO is soliciting feedback and comments due by March 11, 2024 on the the proposed prioritization framework. Please read our blog to learn more about FedRAMP.
US Government agencies including DOD and Federal Civilian agencies spent nearly $3B on AI solutions based on a 2023 report from Stanford University. This spending is expected to grow rapidly as agencies and public sector organizations start production deployments in the near future. To ensure the safe and secure deployment of AI technologies and to drive the development of standards, the Department of Commerce announced the creation of the NIST AI Safety Institute Consortium (AISIC). With over 2,200 companies that have AI related solutions funded by various Government R&D programs, there is a recognition that mission critical AI solutions must reach the hands of the end-users in Federal agencies. Since most of these commercial AI solutions are delivered using cloud services like Google Cloud, AWS and Microsoft Azure, these AI solutions must be FedRAMP accredited.
To jumpstart the availability of AI solutions, the FedRAMP PMO published a DRAFT prioritization framework that spells out the initial categories of Generative AI solutions as well as the benchmarks that will be used to drive selection. The initial focus is on Generative AI solutions used for (1) chat interfaces, (2) code generation & debugging tools and (3) prompt-based image generation. In order to provide an objective criteria for prioritization, the following benchmarks are proposed (reproduced below from the FedRAMP document).
Benchmarks for Chat interfaces
Technical Characteristics: The offering should be capable of discerning meaning from open-ended user inputs and provide an appropriate response.
| Benchmark | Leaderboard | Benchmark Goal | Source/Creator |
| WinoGrande | Leaderboard Paper | Common sense reasoning | Allen Institute for AI |
| ARC challenge | Leaderboard Paper | Common sense reasoning and scientific reasoning and knowledge-based question answering | Allen Institute for AI |
| HellaSwag | Leaderboard Paper | Sentence completion | Allen Institute for AI |
| OpenBookQA | Leaderboard Paper | Reading comprehension and question and answering | Allen Institute for AI |
| MMLU (5-shot) | Leaderboard Paper | Reading comprehension and question and answering |
Allen Institute for AI |
| HumanEval | Leaderboard Paper | Scientific reasoning and knowledge-based question answering | OpenAI |
| MBPP (3-shot) | Leaderboard Paper | Programming language understanding and code generation | Google Research |
Benchmarks for Code generation and debugging tools
Description: A tool used by software developers to help them with creating and debugging software.
| Benchmark | Leaderboard | Benchmark Goal | Source/Creator |
| HumanEval | Leaderboard Paper | Scientific reasoning and knowledge-based question answering | OpenAI |
| MBPP (3-shot) | Leaderboard Paper | Programming language understanding and code generation | Google Research |
Benchmarks for Prompt-based image generators
Description: A product that takes text or photographic input and generates new images or videos based on those inputs.
| Benchmark | Leaderboard | Benchmark Goal | Source/Creator |
| CLIPScore | Leaderboard Paper | Evaluating the alignment of text and images in multimodal models | Allen Institute for AI |
| X-IQE-Overall | Leaderboard Paper | Assessing the quality of image quality enhancers through a comprehensive evaluation framework | University of Waterloo |
Preparing for FedRAMP ATO
If you are an AI company with a compelling solution that has the ability to meet the benchmarks described above and have a desire to sell into a rapidly growing US Government, Department of Defense and Public sector markets, then a FedRAMP strategy is critical. Please review some of the resources below to assist in your journey.
How to Prepare for FedRAMP – Free WhitepaperHow much does it cost to prepare for FedRAMP – Free Blog How long does a FedRAMP ATO take (prioritization can help reduce the time!) – Free Blog
We hope you find these resources helpful. Please contact us to schedule a free consultation and planning discussion. If you are located in the Bay Area then please join us at the FedRAMP & AI Symposium 2024 on March 08, 2024.
