S.B. 324 Outcome-based Investment Grant Pilot Framework: transcript

Okay. Um, The bill, this bill represents the next step in accountability policy making. Over time, public budgeting has evolved. First, we focused whether funds were spent lawfully, then we required activity reporting what was done with the money. In other words. More recently, we've emphasized on measurable outcomes. Think of your RFAs. The modern evaluation goes one step further. It asks a simple but important question. Did the funded intervention actually cause the intended outcome? Did the intervention cause the outcome? As we studied evidence-based policy, we found that nationally evidence is often described in tears. The is level, you see randomized controlled trials and results are statistically significant and, and replicable. At the moderate level, you see quasi experimental, um, designs and match comparison groups. At a promising level. You'll see pre and post measurement with emerging data. Each has value. But the key shift in moderate policy making is not just labeling something evidence-based. It's requiring clarity and rigor, rigor being the key word before funding is awarded. When I first began exploring this issue, I considered whether we should divine evidence-based in statute, even create a tiered scaled, uh, uh, scale and code. But as we consulted and worked with experts and reviewed national bottles, it became clear that research methods evolve in different policy areas requiring different levels of rigor. Instead of starting with a fixed definition in code, the bill focuses on creating an adaptive framework that can mature over time as we learn from implementation. So what this bill does, it requires clarity upfront. Before a grant is awarded, applicants must define the specific outcomes they intended to achieve. They must identify how those outcomes will be measured. They must describe the evaluation methodology, including comparison and con counterfactual approaches were appropriate. Second, the it, uh, it requires independent evaluation. That's the most key portion of this. The entity implementing the project does not score its own performance. That's what we've been doing with grants and independent evaluator measures whether the intervention produced the intended outcome. Third, the office of legislative auditor general establish. This is also where I talked about evidence based evaluation. This is the key. The legislator, auditor general will establish the standards and review the methodology rigor, so the separation preserve Independence and credibility. Um, in the substitute, and I do have a substitute online here, you'll see first that it clarifies the framework and applies when the legislature explicitly directs its to use to the appropriation. This is a targeted pilot, not a universal mandate. Um, I wanted to go over specifically some of the items on that, on that sub. Um, it adds two appropriations. Uh, the, for the first grant program, the use, uh, the use of the new framework. The framework that this pilot program would create. It, it, there's 4.5 million to the Utah State Board of Education. And under that, the administer a grant for up to two LEAs to implement evidence-based strategies modeled on the strategies by the Houston Independent School District, implemented beginning in 2023. Number one, hold teachers and principals accountable for improving student outcomes. Number two, hire and retain high quality educators and school leaders through enhanced compensation coaching, extended planning time, instructional support, and targeted professional development. Second, there's another 4.5 million in the Utah Board of Higher Education administer grant designed to improve upward mobility through the use of digital data inventory to measure improvements and in individual's educational or eco economic outcomes. The, the initial bill was to set up the structure and then the eyes got on what, what I was doing in this bill and, and thought that these two particular, uh, funding items or grants would be good test cases to put in this to this pilot grant program. It's a far more rigorous, far more, uh, methodologic methodological process, uh, with independent analysis. Instead of the grant recipients coming back and reporting how good it was, we have independent analysis on that. And the structure, again, being set up, uh, separately so that there's that process of, of gauging whether it was successful or not, can be done by an independent analysis. With that, I can, uh, that concludes my presentation.

It's not, the grant program's just gonna be to a specific program. Not So, so again, yeah, this is a pilot program. I suppose it could be implemented wherever the legislature sees fit. If there's a grant program, they would like to see funneled through this program, uh, pilot program to, for its analysis. They could, but unless it's funneled through meaning funded into this, then there it wouldn't, it wouldn't be widespread. You, you wouldn't apply it to every grant program out there. We would have to say, as a legislature, we want this grant program to go into this pilot program for that analysis.

Yeah. So as I, I started as I kind of give the background evidence policy, making that whole kind of tiered process of what, what it might, what it might structure. Um, we're gonna let them come up that structure instead of in, you know, in code dictating what that we'll let them freedom to be able to structure that and then do the analysis or have independent analysis done. That way we can have some flexibility to adjust it as the program goes forward. There's a Sunset in the pilot program. They can make adjustments as they see how these first two programs go through. And, and I, I suppose just as I'm, I'm not gonna hide the ball here. If the pilot program is successful, if it works, then we, we as a legislator would decide, do we want to expand this and have this greater rigor, kind of a kind of, uh, analyzing the outcomes of each of these grants going forward? We could spread it further, but that will be up for us to decide in the future.

That, that's right, Mr. Chair. And if I might, it not only will set up the framework for analysis, uh, after the, after the program's in place to report back, does it do what, what did the outcome, did it produce the outcome that it said it was gonna produce, but it also set up the framework so that the grantees or the individuals speaking the grant will know, they'll know, all right, this is the stringent kind of, uh, criteria of which I'm gonna be judged on. They'll know going in, um, what that is. And then that framework will be applied and then we can assess as to whether or not we want to continue the grants. It's a far more rigorous, it's gonna be more difficult. There's no question about it. There's gonna be more accountability. It's gonna be independent auditing. This is far more rigorous than what we've ever done before. This is, this is somewhat of a game changer when it comes to granting Yeah. Money. So, yeah.

Okay. You, you called that Mr. Chair? Yes, I do. Uh, Nate Talley, deputy Commissioners, uh, CFO for the Utah System of Higher Education. I'm here to speak to the first sub and the appropriation that impacts, uh, Yushi. Uh, I sh I I wanted to note that the MC Gardner Policy Institute under the umbrella of the University of Utah has been working with Dr. Raj Chetty of Opportunity Insights to explore new and in innovative ways to measure educational workforce program effectiveness, uh, with respect to employment and wage outcomes of participants. If you haven't heard, uh, Dr. Chetty is a Harvard PhD economist who's been massively cited for seminal work in socioeconomic upper mobility, which is the same evidentiary basis and analytical foundation that's behind the claim that so many of us have heard and, uh, have witnessed firsthand that Utah is leading the nation in socioeconomic upward mobility. As these con conversations have continued, they expanded to include many of, uh, your colleagues, the fiscal analyst office, the bill sponsor, uh, legislative auditors, and, uh, now this appropriation in this bill and, and how that came about. Uh, the elevator pitch, of course, is exactly along the lines of, uh, what the center has described, and that's to leverage, um, uh, emerging analytical power and data to, uh, uh, exercise matching al algorithms to identify counterfactual observations in participating program associated with individuals participating in education workforce programs to be able to actually speak to program effectiveness in a way that, uh, we haven't been able to do before and will be able to do moving forward with, uh, with this pilot. Uh, moreover, it creates this framework that is potentially scalable to other state policy interventions and other state policy priorities.

Thank you, Mr. Chair. I'll be very quick so you can cut it off. But Michael Parker with the City Strong Foundation, uh, we hosted Dr. Chetty in November with Zion Bank and, uh, would just like to speak to the overall effort here to essentially upgrade what the state's ability is to analyze outcomes. I often work with private capital coming to public good problems and private capital, whether in an enterprise or foundation, asks very tough questions of what the outcomes are in a setting. And having a tool like this for a state partner is really critical. So I speak for your support.

Thank you Mr. Chair. I would make a motion that we delete and title and Body Senate bill 3 24 replace it with first sub 3 24

Uh, I would make a motion that we favorably recommend first substitute Senate Bill 3 24.

Uh, uh, I would just, I would just say, Um, hold on just a second.

Man, I should have waved. It'll be good. I, I Know with apologies that I arrived late and so was trying to catch up quickly. Uh, in this fiscally tight year, can you help me understand where $10 million goes, how it goes? So, um, understand, I I didn't catch the last part, Like $10 million. What? It just, it's direct, like what it's going toward in a, in a brief two sentence deal,

4.5 million to Utah State Board of Education, 4.5 million to the Utah Board of Higher Education For, For the Utah State Board of Education administered two, a grant for up to two LEAs to implement evidence-based strategies modeled under the strategies that the Houston Independent School District implemented beginning 2023. Number one, it holds teachers and principals accountable for improving student outcomes. Number two, higher, higher and retain high quality educators and school leaders through enhanced compensation coaching, extending uh, extending planning instructional supports to targeted professional

Development. Okay, so it's the grant and the administration of the grant, the both pieces of it kind Of Yeah, the, uh, the, the, They go, yeah, they go back, if I understand it right, it goes back to 23.

Look at what, what happened, what the results were going forward making. It's, it's implementing what they did in 2023. And then this, this framework that I've created, these grants will go through this framework instead of our standard framework. So what I've created is a framework that is a little more rigorous, not a little bit, it's a lot more rigorous than our standard framework. So these two grants could have just gone through our normal process and would have, but instead of going through that normal process, they're gonna go through this pilot program, which is far more rigorous than anything else we have.

Okay. Thank you for that. Get me up to speed piece. Okay. Yeah. Okay. Yeah, I had another light.

Thank you. Um, Mr. Chair, um, I'm gonna give you a little bit of background real quick. The bill represents the next step. I believe in accountability policy making over time, public budgeting has evolved. First focused, uh, whether funds were spent lawfully, and then we required activity reporting what was done with the money, in other words. And more recently, we've en emphasized, um, measurable outcomes. You think of that when you, when you apply for, uh, funding, the modern EV evolution goes one step further. It's at, it's at ask the simple question. Um, did the funds, or did the funded intervention actually cause the intended outcome? Um, as we studied at evidence-based policymaking, we found that nationally all evidence is often, uh, described in tears. At a high level, you have randomized controlled trials, and then the, at the moderate level, you have quasi experimental designs, and then a promising level. You have pre and post measurement. Each has value. But the key shift in moderate policy making is to just, is not just labeling something evidence-based. It's acquiring clarity and rigor. Uh, before funding is awarded. When I first began to kind of explore this, uh, topic and consider whether it be we should be, uh, defined evidence based in code, but as we consulted with experts, and I talked with the experts we have here, uh, realized that it became clear that maybe something that, um, more of a targeted program would be a more appropriate. So what the bill does, um, before a grant is awarded, applicants must define the specific outcome they intend to achieve. They must identify how those outcomes will be measured. They must describe the evaluation methodology, including comparison and counterfactual approaches when appropriate. Second, it requires an independent evaluation. This is the key point. We don't require independent evaluation. These grantees, we give them grant money, and I'm not, I'm not beating up on the grants. It's just what, how we do it. And then they who comes and reports back the grantee on how well they did, of course, um, if they're wanting to continue. So the idea is to have independent, the entity implementing the project does not score its own performance and independent value measures whether intervention produced the intended outcomes. Third, the office of legislative auditor general established evaluation standards and may review, review methodology, uh, rigor, the separation preserves independent credibility. Um, first it clarifies that this framework only applies when, well, actually, I should say, um, we did have a substitute and, and the key language in there that, um, I, I think some were concerned we were gonna apply this to all grants. This is a pilot program. It only works for grants that are plugged into it. That's it. And there's two grants we've plugged into, into this after that framework or that the bill was created. Um, with that, uh, I'm gonna just clarify what I think is most important, what I just outlined. Um, we have two grants out there that would, would, may have been funded through, uh, our normal request process. By plugging these two grants into this program, we actually get independent analysis and a structure set up by LAC to decide whether or not that outcome, uh, that, that they met the outcome that they said they were going to achieve. Not some ancillary result, but the actual outcome they intended to achieve. And if they didn't, then they report back to us and tell us how that, how that went. Again, it's a pilot program, it has a Sunset date. Um, and we will look at this framework and see if it's something we want to try and do further outside of, of, uh, these two programs. So with that, I'll, I'll submit to questions that completes presentation. Thank you. Alright,

Thank you, Mr. Chair. Um, would you discuss the fiscal note and has this been funded or would it get, can it get funded?

Yeah. Well, I wouldn't be here if it wasn't, but, yeah. So, uh, representative Moz, I appreciate the question. So again, these were two grants that were likely going to happen through the process. They have been funded, they went through that process. But instead of finishing that process, we plugged into this bill so that there's greater rigor and analysis on the grants instead of, 'cause otherwise, we would simply grant them and then they would come back and report by plugging it into this framework that I've created. We'll have an independent analysis and a greater rigor in that, uh, kind of reporting. So just to clarify for me, this money is already there for that purpose for the grants. It's already in our budget. And this would just put it into a different framework, is that correct? That that's Right. So it Right, It's not new money. Yeah, that's, well, it's, no, it's money that would be approved in this budget cycle.

Okay. Yes. Thank you.

And it, and it already has gone through that. Right. So, okay.

Thank you, Mr. Chair. Just real quick, um, so the independent evaluators, who, who is that again? And is that, is that, are we creating a new marketplace for evaluators or, or where do those folks

Yeah, no, that's, that's, that's fair. That's possible that we'd be creating a, a I suppose that marketplace already exists, but that will be, uh, that, that will be part of, I suppose, a side effect of that. Um, I do have Cade Minchi here. You want Cade, do you wanna come up and just maybe talk about the possibilities there? Um,

Is that okay? Yeah, That'd be great. I brought in the big gun so We, we know, but, uh, just for the record,

Minchi, legislative auditor. So, yeah, a really great question. And there are lots of entities out there that do that. Right now. A lot of it is in higher education or in academics, but there's a provision there to, um, help with the funding of that. And so what we would do is we create the standard by which that evaluation would be done. We want those standards being placed to make sure the valuation is done appropriately and by good standards, then in the bill allows us to go back and look at the valuation just to make sure that those standards were done. So our process here is just putting the standards in place and look at the valuation, but the valuation itself would be done by those that are qualified to do that. And we, and we could in this process, help make sure that those qualified people were selected.

So would there be a randomized way of selecting the people? Because, you know, obviously the concern would be is, you know, we got quality people who have integrity here, right? But let's say 20 years from now, it, does it become a good, good old boy system where, um, my buddy's, uh, you know, and they, and they start getting the business, if that makes sense. That's right. Is is it just like a, there's five qualified and we're just gonna right to rotate through each five? Or how does that work?

It, No. Yeah, A olag is gonna set up the framework for, for this process. Not, not just the process of how it evaluated, but the selection process as well. So that, that'll be done in rule is, I guess what I'm

Okay. That, that helps me understand that. Thank you. Yeah.

Uh, well, it's based on evidence-based policy making and there's standardizations that are kind of, you, you can look at other states and what other states have done, kind of establishing what the different levels of what evidence-based is. I'm gonna allow Ola knowing those structures that are out there already to, to kind of set that standard. I shouldn't say I allow, we've, in this bill created rulemaking authority for them to create the rules of which that standard will be applied. But it's the standards of the, you know, the, the highest level randomized controlled trials, uh, at moderate levels, quasi experimental designs, and then of course the low level pre and post measurement. They, I'm giving really quick kind of, uh, outline of what those are, but oola will take that much deeper. And, but the point is, is you're gonna have an independent assessment about whether the met, whether the metrics that were part of the grant application were met, That that's the key independent analysis. Now that's the key point, right. But there's a lot of framework behind it.

Uh, I'd like to make a motion to pass out for substitute SB 3 24 with a favorable recommendation. The motion is an order. Would you like to speak to it? Yes. Um, you know, I, over the years I've seen a lot of money sent out to a lot of organizations to do grants and I, I do appreciate, um, first of all that this is limited in scope, that we're really just kind of ticking on too at first to see how it goes. Um, but that we are thinking about what those metrics are and, and do we really have a good way to understand outcomes for the investments that the state is making. So, um, I think it's an interesting framework. I like that we're, we're limiting it in scope at first to these two grant programs, um, to, to kind of determine if this is the right process and what we can learn along the way. So I, I think it's good.

I'm only gonna parrot, just so that it's clear what the good representative said. This is a pilot program program. It doesn't function unless we direct grants into it. The legislature has to do that. So it's not, it's not vacuuming up or requiring grant proce a grant valuation from any other grant. Just what we dictate. So,

Thank you Mr. President ProAm. Um, this bill represents the next step and accountable policy making over time, public budgeting has evolved. First, we focused whether funds were spent lawfully, then we required activity reporting what was done with the money, in other words. Um, more recently we've even said measurable outcomes. The modern, uh, uh, evaluation goes one step further. Um, it, it asks a simple but important question. Did the funded intervention actually cause the intended outcome as we studied evidence-based policy and we found the nationally evidence is often described in tiers. At the highest level, you see randomized controlled trials and results that are statistically significant and replicable at a moderate level. You see quasi experimental designs and matched comparison groups at a promising level. You see pre and post measurements with emerging data, each has a value. But the key shift in modern policymaking is not just labeling something evidence-based. It's requiring clarity and rigor before funding is awarded. When I first began explaining this issue, I considered whether it was should be defined evidence-based in statute or even create a tiered scale and code. But as we consulted with experts and reviewed national models, it became clear that the research models methods evolve in different policy areas require different levels of rigor. So with that, what this bill does before a grant is awarded, the applicant must define this specific outcomes they intend to achieve. They must identify how those outcomes will be measured. They must describe the evaluation methodology including comparison and counterfactual approaches. Second, it requires independent evaluation. The entity implementing the project does not score its own performance. And independent evaluator measures whether the interventions produce the intended outcomes. Third, the Office of Legislator auditor General establishes evaluation standards and may review the methodology rigor, the separation preserves Independence and credibility. Um, with that there are two, uh, items that are, uh, grants that are in this program that are, will be test cases that have been funded, um, that I'll submit to. Questions that concludes my presentation. Thank you Mr. President Pro Tem.

Thank you Mr. President Pro Tem. Uh, the sponsors open for questions.

Will you yield senator? I I do.

Thank you. I'm looking at the fiscal note and it has $9 million fiscal note that's coming from the Uniform School Fund being transferred to the public education economic stabilization restricted account. Um, how much do we have in the public education economic stabilization restricted account? And is this a fair use of our uniform school funds to go into that account to pay for this? And why is it $9 million to look at grants to see if they are, um, good or not? That just seems like a really high fiscal note.

So Senator Musselman, The, the grant, the money of the fiscal note are for the grants themselves, not for the creating or the structure of the program. The structure of the program is created essentially to review those grants. Those grants could be funded in or outside of this, uh, pilot program that's independent of whether or not the, this, this program, uh, or the structure of this program. The structure of this program is meant to be more rigorous, hold more accountability and have Independence. So in other words, these grants could have came in other methodologies through other bills or through other, uh, funding mechanisms. But they're being put into this bill specifically because it creates a far more rigorous structure for analysis.

Thank you, Mr. Mr. President Pro tem. Um, when I think about all the grants and I think about all the work that's been doing and all the measuring and assessing and reporting back to us, and then reevaluating, and then I think about $9 million. That's a significant amount. Money that could be used better in our classrooms. And I just, I struggle with all this money coming out of our uniform school funds and then also being taken out of our public education stabilization fund. When we think about those two funds, those two funds were created so that we could have more robust classrooms, smaller class sizes, resources in the class fund rooms. I don't believe that this is where that money should be coming from. And so I I, I, I'm kind of okay with what the bill does. I just don't like the way that it's being funded. Thank you.

I I would just resubmit the previous argument and summation. And that is the, the structure of the bill itself. And granted, um, that was, that was clarified that there may be no problem with the structure. Um, but again, these grants are separate from the structure. It doesn't mean they can't, I don't need to justify them in the program. But these grants may or may not have been otherwise provided in other mechanisms throughout the session, whether it be a, for the, for pure funding request or another bill. The, what what's critical here is that this is a far more stringent structure to analyze any grant. These grants happen to be picked either for greater rigor or, or, or greater, um, kind of Independence and, and analysis. I'm, I, I, I don't know for certain as to whether or not that's what the individuals asked that wanted, that wanted to be the system, but I can tell you for certain that will happen within this bill structure. Thank you. Now, for a motion, I would move that, uh, SB first substitute SB 3 24 outcome based investment grant pilot framework. Be read for a third time And the question is, shall first substitute Senate bill 3 24?

Senator Cullimore, There was a seat now on President Prom. We we're on the second reading calendar. We Are on the second reading calendar.

different, uh, speech as to what I gave the previous. Um, on the second reading, I just want to, I I wanna point out that when we evaluate grants, this is what I've experienced. When individuals come back and report on grants they've received, the individuals are reporting on what their experience was and what they think was successful. The problem is, it's the grantee that's reporting on it. There's no Independence whatsoever in that analysis. What we've created in this bill is a framework which requires independent analysis, an independent audit, um, separate from what the grantee might think was successful. The independent audit will dictate or establish what was successful. Did the grantee meet the, meet the outcome that they intended instead of them reporting in and of themselves. And then, um, we can actually have independent data to be able to analyze as to whether or not the grant was successful and others as to whether or not we want to in the future, continue to grant funding to that program or others. This is a pilot program. Indeed. It's lots more rigorous than anything we've ever, you know, tried to accomplish before it sets up standards that just are, to be quite honest, difficult for these grantees, um, in the auditing process. It's, it's, it, it, it provides us a way to really, truly have data, um, and know how to move forward With that. Mr. President, I'm open for

Uh, call the question Mr. President. Thank you. And the question is, shall first substitute Senate Bill three twenty four pass roll call vote.

Senator Vickers Baldy Lewin Brammer Bus Comore Escamia Fillmore Aye. Grover Aye. Harper a Hankins Aye. Ipson Aye. Johnson. Aye. Juab McKay Mikel Milner Musman Aye. Owens pitch thumb re Sandal Stevenson Stratton Vickers Weiler Wilson Winterton bus Picture notes

Thank you. This bill affirms that students, including former students, have the right to access their own educational data and decide who can receive it. It puts students in control of their transcript, data,

Representative Albrecht Motion to circle Motion is to circle.

You. So every year our state spends millions of dollars, uh, sending out to different organizations to do grant programs, um, through grants to do to run different programs. And, um, it's hard sometimes to know what the outcome of those dollars are. Um, often we ask the organization that we gave millions of dollars to to tell us how it worked rather than doing maybe an independent evaluation of what we expected those outcomes to be and being able to look at, at, um, how those programs actually operated. So what this bill does is it sets up a pilot program. So this is, uh, something we're going to try in relation to, uh, our office of legislative. Um, our OLA office LFA has also helped with the drafting of this bill to, to set up a pilot where we look at the outcomes of some of the grants that we're offering as a state. And so there are two grants in here that we anticipated doing this year that have been put into the program that will be part of our pilot. So it won't impact every grant in the state, it will just impact the two in the bill. And this will be our pilot program to see if this process set up in the bill really does help us understand how tax dollars are best used. So that's the bill I'm open to Any questions?

I appreciate your support. Thanks.

Senator Vickers Baldy Lewin B Brammer bus call. More is your escamia. Aye. Fillmore Aye. Rover. Aye. Harper Aye. Kins Aye. Ibsen Aye. Johnson Aye. Juab McKay Aye. Mikel Aye. Milner. Oh, Oops. Sorry. Musman. Aye. Owens pitcher Plum rebe sandal Aye. Stevenson Stratton. Yes. Thickers er Wilson Winterton President Adams

Thank you Mr. Speaker. I'd like to make a motion to recall Senate bill three twenty four from the Senate.

Yes. There was a additional amendment that needs to be made to the bill.

you like to speak to that motion? Yes. This is the bill that just needs the amendment.

Thank you Mr. Speaker. I'd like to move amendment number one in my name.

Yes. Um, this amendment was at the request of the Senate, uh, sponsor and it didn't make its way over. It made its way over just now. So we're getting it into the bill. Thank you.

Thank you. The, so that's the only change. Um, we passed this bill out earlier, I think yesterday afternoon. Um, so if we could, uh, send it on its way, that would be much appreciated. Thank you.