Managing Generative AI Risk and Meeting M-24-10 Mandates on Monitoring & Evaluation
OMB’s memo M-24-10 (5c. Minimum Practices for Safety-Impacting and Rights-Impacting Artificial Intelligence) is prescriptive (and timebound): No later than December 1, 2024 and on an ongoing basis while using new or existing covered safety-impacting or rights-impacting AI, agencies must ensure these practices are followed for the AI: D. Conduct ongoing monitoring. In addition to pre-deployment testing, agencies must institute ongoing procedures to monitor degradation of the AI’s functionality and to detect changes in the AI’s impact on rights and safety. Agencies should also scale up the use of new or updated AI features incrementally where possible to provide adequate time to monitor for adverse performance or outcomes. Agencies should monitor and defend the AI from AI-specific exploits, particularly those that would adversely impact rights and safety.  E. Regularly evaluate risks from the use of AI. The monitoring process in paragraph (D) must include periodic human reviews to determine whether