SUBJECT
Title
LAUNCHING A COMPREHENSIVE APPROACH TO EVIDENCE-BASED POLICYMAKING AND ESTABLISHING AN OFFICE OF EVALUATION, PERFORMANCE, AND ANALYTICS (DISTRICTS: ALL)
Body
OVERVIEW
Evidence matters in public policymaking. By highlighting what is working and what is not, evidence can guide and inform policy and budget decisions to be more effective, more equitable, and more focused on delivering the best outcomes for all San Diegans. An evidence-based approach prioritizes high-quality research, data, and program evaluation over and above anecdotes, ideology, and inertia towards the status quo.
Today’s actions outline a bold new approach for the county grounded in national and international best practices and sets us up as a leader among Counties and local governments. This comprehensive approach goes way beyond mere data collection and reporting obligations but signals a significant shift in the organizational culture and attitude towards evidence and evaluation. These actions will allow the County to deepen a culture of iterative learning, where we can pair pilot policies with rigorous evaluation to bring effective approaches to scale. It also allows the County to be nimbler, collecting timely information on what is working to make mid-course corrections and policy adjustments. Finally, the County’s new approach will be strategic. Rather than evaluating all programs and initiatives, we will strategically focus program evaluation efforts on the service areas and program models that are most in line with County priorities and have the greatest potential to influence major policy and funding decisions.
This approach is rooted in proven practices developed by national and international experts in program evaluation, evidence-based policy, and good governance, including United Stated Agency for International Development, University of California San Diego’s Global School of Policy, PEW Charitable Trust’s Results First Initiative, the Urban Institute, and the Brookings Institute.
We urge you to join us in supporting this groundbreaking initiative.
RECOMMENDATION(S)
SUPERVISOR TERRA LAWSON-REMER AND CHAIRNATHAN FLETCHER
1. Direct the Chief Administrative Officer to establish an Office of Evaluation, Performance, and Analytics which will provide leadership, coordination, and capacity-building to oversee the County’s efforts around evidence-based policymaking, program evaluation, data sharing, collection, metrics. This Office will serve an enterprise-wide function and report directly to the Chief Administrative Officer. Among the required duties will be developing an Annual Strategic Research Plan that identifies and aligns strategic research goals in coordination with the County’s annual strategic planning process. The Office will be led by a Chief Evaluation Officer and staffed with appropriate expertise to carry out this policy shift towards a comprehensive approach to evidence-based policymaking.
2. Direct the Chief Administrative Officer to conduct a national search for a Chief Evaluation Officer. Once hired, the Chief Evaluation Officer should evaluate all County data collection, management, sharing and analysis functions and align them within an appropriate staffing and governance structure to advance the objectives of this policy initiative.
3. Direct the Chief Administrative Officer to competitively bid for a consultant with deep expertise in evidence-based policymaking, program evaluation, and data sciences to assist in establishing the Office of Evaluation, Performance, and Analytics including, but not limited to, evidence-based strategies, data infrastructure, staffing, and resource development.
4. Approve the introduction (first reading), read the title, and waive further reading of the Ordinance:
AN ORDINANCE AMENDING THE SAN DIEGO COUNTY CODE OF ADMINISTRATIVE ORDINANCES RELATING TO ESTABLISHING AN EVALUATION ADVISORY COMMITTEE
If, on May 18, 2021, the Board of Supervisors takes action as recommended, then on June 8, 2021, consider and adopt (second reading):
AN ORDINANCE AMENDING THE SAN DIEGO COUNTY CODE OF ADMINISTRATIVE ORDINANCES RELATING TO ESTABLISHING AN EVALUATION ADVISORY COMMITTEE
5. Direct the Chief Administrative Officer to provide a written progress report on the above actions to the Board in 60 days, again in 120 days, and return to the Board by March 1, 2022 with a third progress report and a detailed plan for further implementation in FY 2022-23.
6. Refer to FY2021-22 Budget $14,500,000 to establish the Office of Evaluation, Performance, and Analytics.
FISCAL IMPACT
Funds for this request are not included in the Fiscal Year 2021-22 CAO Recommended Operation Plan. If approved, the request to establish the new Office of Evaluation, Performance and Analytics will result in one-time costs of $14,500,000 based on General Fund fund balance. There will be a change in net General Fund cost of an estimated $14,500,000, and 3.0 additional staff years, based on program design, organizational structure, and selected classifications. This represents an initial first year investment. Once the Office of Evaluation, Performance, and Analytics is fully established, ongoing costs of an estimated $4,500,000 will be based on General Purpose Revenue for this office with future costs to be determined.
BUSINESS IMPACT STATEMENT
N/A
Details
ADVISORY BOARD STATEMENT
N/A
BACKGROUND
Evidence matters in public policymaking. By highlighting what is working and what is not, evidence can guide and inform significant policy and budget decisions to be more effective, more equitable, and more focused on delivering the best outcomes for all San Diegans. An evidence-based approach prioritizes high-quality research, data, and program evaluation over and above anecdotes, ideology, and inertia towards the status quo.
This effort is more important than ever today as we face deep challenges to public health, racial justice, and economic inclusion. We can leverage evidence-based policymaking as a potent tool, putting tested solutions on the table and bridging divides using sound analysis and data. Research from the PEW Charitable Trust and the Urban Institute has linked evidence-based policymaking with the following benefits:
• Focusing on proven programs that deliver better outcomes for residents
• Strengthening accountability and transparency
• Encouraging sound stewardship of public resources
• Promoting a culture of learning and innovation, continuously testing new approaches, and adapting to new situations.
The movement towards evidence-based policy has grown in recent years and is changing how governments work for people. Momentum continues to build as the Biden Administration recently issued a “Memorandum on Restoring Trust in Government Through Scientific Integrity and Evidence-Based Policymaking” that directs all federal agencies to put scientific integrity and evidence at the center of federal policymaking.
There is much at stake for the County of San Diego, which designs, funds, and implements thousands of critical services and programs that serve the over 3.3 million residents. Today’s action outlines a bold new approach for the county grounded in national and international best practices and sets us up as a national leader among Counties and local governments.
This comprehensive approach goes beyond mere data collection and reporting obligations but represents a major shift in the culture and attitude that we have taken towards evidence and evaluation. Together, these actions will allow the County to deepen its culture of learning, where we can rapidly pair pilot policies with rigorous evaluation to bring effective approaches to scale. It also allows the County to be nimbler, collecting timely information on what is working to make mid-course corrections and policy adjustments. Finally, the County’s new approach will be strategic. Rather than evaluating all programs and initiatives, we will strategically focus program evaluation efforts on the service areas and program models that are most in line with County priorities and have the greatest potential to influence major policy decisions.
This approach is rooted in proven practices developed by national and international experts in program evaluation, evidence-based policy, and good governance, including United Stated Agency for International Development, University of California San Diego’s Global School of Policy, PEW Charitable Trust’s Results First Initiative, the Urban Institute, and the Brookings Institute.
Establish the Office of Evaluation, Performance, and Analytics
This newly formed Office will report to the Chief Administrative Officer with responsibilities that will include, but not be limited to the following:
• Development of enterprise-wide standardized definitions around terms like “evidence-based”, “research-based”, and “promising practice” to create common terminology and clarify what constitutes differing levels of evidence.
• Development of an enterprise-wide program inventory to review program fidelity and the extent to which each program meets the definitions of evidence as described above.
• Development of an Annual Strategic Research Plan that includes identifying relevant policy questions and identifying key service areas and/or program models to focus program evaluation efforts. Further discussion of the Research Plan is included below.
• Implementation of in-house program evaluation efforts.
• Project management of third-party independent evaluators.
• Focus on establishing County-wide data infrastructure and governance, with data-sharing agreements across law enforcement, jails, health and social services.
• Improvement of data collection, visualizations, and reporting on County programs, outputs, and outcomes, including tracking progress toward attaining strategic organizational goals including Live Well San Diego goals.
• Identification of strategies and opportunities to strengthen County capacity for strategic planning, performance monitoring, and program evaluation.
• Collaboration with Department of Procurement and Contracting and Group departments to strengthen and expand the use of performance-based contracting, focused on the delivery of outcomes.
• Collaboration with the County Technology Office to strengthen and expand data management systems and develop advanced technology solutions to meet the enterprise-wide and department-specific data-driven needs that support monitoring and evaluation and performance management.
• Build internal capacity and drive innovative data analytics and collection to assist departments with scrutinizing data and drawing conclusions to help solve identified problems.
Suggested roles and responsibilities when determining the governing structure of the new office:
• Chief Evaluation Officer: This high-level management position is responsible for planning, directing, organizing, coordinating, and evaluating the overall activities and operations of the Office of Evaluation, Performance, and Analytics, including the development of high-profile County project such as the Annual Strategic Research Plan. Directs the establishment and implementation of policies and procedures related to County-wide data collection, analytics, performance dashboard, and program evaluation. Incorporates data and program evaluation policy throughout County policies, procedures, strategies, and operations.
• Principal Investigator Research Manager: This role is responsible for leading the development and implementation of program evaluation activities, participating in all phases of the complete evaluation process, including problem definition, planning, implementation, analysis, interpretation, and reporting of findings
• Data and Evaluation Manager: The role is responsible for managing and overseeing the implementation, monitoring, and reporting of analytics, metrics, and evaluation data. The position will work closely with staff within each Group, including the Data and Evaluation Analyst Liaisons.
• Performance-Based Contracting Manager: This role provides technical assistance to program staff and Department of Procurement and Contracting staff to evaluate and improve performance-based contracting reporting metrics and ensure that all County contracts are aligned with County’s evaluation framework and national best practices.
• Data and Evaluation Analyst Liaison: The role serves as a critical bridge between the Office of Evaluation, Performance, and Analytics and each respective group. The position will develop deep programmatic, service area, and data collection expertise, and coordinate with the Data and Evaluation Managers within the Office of Evaluation, Performance, and Analytics.
• Data Scientist: The role works in collaboration with the County Technology Office and departmental staff to integrate the collection, storage, analysis, use, and sharing of data to support the realization of the County’s vision around evidence-based policy and integrated data management.
Integration with other county data collection and analysis functions
Analyze and assess whether it would be more effective and efficient to integrate all data collection and analysis functions across County Departments, such as the Data Performance Analytics Unit, to align with this policy for a new approach to evidence-based policymaking for maximization of resources and outcomes.
Annual Strategic Research Plan
The Office of Evaluation, Performance, and Analytics will develop an Annual Strategic Research Plan that includes identifying the most important research and policy questions and identifying key service areas and/or program models to focus program evaluation efforts. Development of the Annual Strategic Research Plan should be incorporated into the County’s annual strategic planning process. The research areas should be selected based on existing County priorities, the need to test innovative new pilot programs and models, and the potential for program evaluation to influence major policy and budgetary decisions. The Annual Strategic Research Plan should include the identification of research methods and program evaluation staffing models, including the use of third-party independent evaluators where appropriate. The Office of Evaluation, Performance, and Anlaytics should consult with the Evaluation Advisory Committee, as well as the public, department subject matter experts and the Board prior to finalizing the plan.
Establish an Evaluation Advisory Committee
The Office of Evaluation, Performance, and Analytics will be supported by a newly created Evaluation Advisory Committee that will review, analyze, and make recommendations on the County’s Annual Strategic Research Plan, and provide on-going strategic advice on evaluation activities, program designs, and data sources, and data collection methodologies. The Advisory Committee shall be comprised of highly experienced, respected experts drawing from academic institutions, think tanks, and independent researchers with deep subject matter expertise. The Evaluation Advisory Committee shall represent a diversity of evaluation methodologies, philosophies, and research areas.
LINKAGE TO THE COUNTY OF SAN DIEGO STRATEGIC PLAN
The requested action is strongly aligned to the Live Well San Diego Initiative in the County's 2020-2025 Strategic Plan and its vision for a San Diego that is Building Better Health, Living Safely and Thriving. By focusing on evidence and evaluation, the County will deploy resources and develop programs proven to lead to the best life outcomes for all residents.
Respectfully submitted,
TERRA LAWSON-REMER NATHAN FLETCHER
Supervisor, Third District Chair, Fourth District
ATTACHMENT(S)
AN ORDINANCE AMENDING THE SAN DIEGO COUNTY CODE OF ADMINISTRATIVE ORDINANCES RELATING TO ESTABLISHING AN EVALUATION ADVISORY COMMITTEE
AGENDA ITEM INFORMATION SHEET
REQUIRES FOUR VOTES: ☐ Yes ☒ No
WRITTEN DISCLOSURE PER COUNTY CHARTER SECTION 1000.1 REQUIRED
☐ Yes ☒ No
|
PREVIOUS RELEVANT BOARD ACTIONS: January 12, 2021 (11) Framework for our Future: Realigning Policies that Govern County Resources and Contracting |
|
BOARD POLICIES APPLICABLE: N/A |
|
BOARD POLICY STATEMENTS: N/A |
|
MANDATORY COMPLIANCE: N/A |
|
ORACLE AWARD NUMBER(S) AND CONTRACT AND/OR REQUISITION NUMBER(S): N/A |
|
ORIGINATING DEPARTMENT: Districts 3 and 4, Board of Supervisors |
|
OTHER CONCURRENCE(S): |
N/A |
CONTACT PERSON(S):
|
Jeff Yuen |
|
Stephanie Gioia-Beckman |
|
Name |
|
Name |
|
858-285-0403 |
|
619-531-5544 |
|
Phone |
|
Phone |
|
Jeffrey.yuen@sdcounty.ca.gov |
|
Stephanie.Gioia@sdcounty.ca.gov |
|
E-mail |
|
E-mail |