The St.

Paul City Council added committee to order.

Roll call, please.

CM Coleman.

Here.

CM Johnson.

Here.

Committee member Dilworth.

Here.

Committee member Lostner.

Here.

Committee member McVay.

Committee member Donnelly.

Here.

Chair Naker?

Here.

All right.

Uh, welcome everyone to our June audit committee meeting.

Happy summer.

Um, we have a packed agenda today.

We have two items.

One is the um semi-final report from our last audit committee, uh, pending our thoughts and potential revisions.

And then uh we have we're moving straight into topic review for our next topic.

So we just never never take a moment's pause here at audit committee.

So uh we'll welcome back Dr.

Britt to take us away.

And like we were just saying, I think we're gonna try to get 30 minutes, 30 minutes if we can.

Um, so we'll see what we can do.

All right, sounds good.

Fantastic.

Uh well, thanks, everybody.

Uh, and appreciate your patience with us as it relates to our timeline.

Um, so I have a couple of intentions today.

I'm gonna do the sort of walkthrough of the presentation that you had attached in the agenda.

Um, uh, I'm happy to take questions as part of the presentation, happy to take broader questions.

Um, and then I'm just gonna invite you to continue, oops, sorry, uh, continue that feedback process, right?

So written feedback that you want to offer um through Nia to us.

We'll continue to take that.

Um, Mia has invited us today to have some specific conversations with her and with Greg and with Nicole and with some of the sort of city attorney um team, uh, so that we can get some additional feedback as it relates to the report.

So we're still striving for a June thirty final version, but I think if we need to push that because we're still receiving some feedback to you all, if you're like, you know what, I can't make the 19th, but I can make the, you know, insert date that's between the 19th and the 30th, we'll continue to take that feedback and do that final full pass to make sure what you get is the final edited version of what we've got available.

And as you know, since we're in long-term relationship with each other, we are here and around.

So happy to continue to do additional work that you might need.

So, maybe I'll just open by saying a giant thank you to Nia, as always, and a giant thank you to Greg, who spent a lot of time with us walking through the GovQA system, spent a lot of time with us back and forth as it relates to folks that we might survey, um, did a lot of back and forth other as it relates to um just city rules about getting surveys inside the walls of the city system.

So um, so I think just want to say an incredible thank you.

You all are fantastic team for us to work with.

So, uh, so I'm gonna go ahead and get started, but I will say, you know, pause me along the way as we are as we are making our way through this.

All right, so we're gonna talk a little bit about background and purpose.

Just a quick reminder, those research questions, the methods, so the kinds of things that we did to amass all of the pieces and parts for this.

We'll talk a little bit about that GovQ8 platform itself, some of the things that we know from taking a look at that platform and taking a look at that data.

We're gonna talk a little bit about those conversations with the staff, conversations with peer jurisdictions, and I'll say sort of other folks, League of Minnesota Cities, Office of the State Auditor, the Data Practices Office.

We'll talk a little bit about the data we have from the surveys that we did of department leads and of portal users, kind of the key findings and recommendations that we have, and then just invite discussion.

And I think I'll close by saying, as always, we look to the audit committee to tune and pull up recommendations, drop recommendations down, do that kind of stuff.

You'll saw we had a pretty light executive summary with a set of kind of eight major recommendations in the in kind of the back of the report.

But again, look to you all to say, move this around, adjust this.

We'd like to see this here.

So really sort of looking for your for your directions once you're really setting the mark for what you expect the broader city team is going to do as a as sort of a response to this.

So you all know this.

We were taking a look at that public data request process.

We really had three kind of big questions.

We had lots of sub-specific questions, but kind of the three overarching guiding questions.

Um, how's it operating?

That overall process, what's the experience of the public with that process, and what are the kinds of opportunities that exist to strengthen that process overall.

Uh and so we did lots of things.

We touched lots of pieces and parts, and if you'll remember we had a slightly larger scope, we've sort of took a look at sort of managing the scope so that we could make sure that we're really sort of considering things that would help inform and advise the city as it relates to next steps here.

So, first thing we did was took a look at as many documents as Nia and Greg could possibly find and send to us.

Um we took a look at policies, training manuals, the internal workflows, and some model guidance from the state's data practices office.

We did a literature review as we typically do.

That's kind of looking at peer-reviewed literature, it's looking at gray literature, kind of what's the best practice information that we have out there.

Um we did have conversations with folks from Hennepin, Minneapolis, Duluth, the State Auditor's Office, the League of Minnesota Cities, and the Data Practices Office.

We also have information from the websites or the online publicly available sites from all of the jurisdictions that you asked us to take a look at.

But those are the folks that we actually had sort of virtual interviews with.

We did a survey of the department leads and portal users, and then we had a bunch of your GovQA data that Nia and Greg pulled for us, and we took a look at that at a really high level.

If you'll recall early on, we debated how deep we were gonna go inside that data, and so really trying to keep that sort of data snapshot at a high level, really, just to take a look at turnaround time, what is that distribution look like, volume, those kinds of high-level questions that you were primarily interested in.

Um, and wow, that's a lot of data.

75,000 plus requests over this time period.

So, all right, let's talk a little bit about just kind of the current process and the key data.

So, just the platform itself.

GovQA is the city's kind of platform for receiving, tracking, managing all of those requests across all of the departments, right?

So it is what it's the backbone that the city staff are using.

Um, its key capabilities, it's got that public request portal, so folks can sort of submit for requests.

Um, it's got the ability to be able to route requests across multiple departments.

It's got what are called parent child requests, so I can attach a big request to lots of baby requests and sort of keep those all together.

Um, I can manage some documents, I can think about redaction.

Um, it's got some dashboards, some dashboards that came kind of out of the box, and then some reporting that in particular some of your departments really had kind of built out in particular for some of their needs.

The city uses what we would call a decentralized model, so common backbone, kind of common intake and tracking system, but really each department is doing the things that they need to do to be able to respond to requests in the ways that it fits the best for their particular department.

And so you'll see a little bit in the survey what that distribution of behavior looks like across those departments.

One thing you'll see in the report, um, we did do a comparison just of capabilities.

You'll see a grid in there with next request and just FOIA, which I always think is kind of a funny name, just FOIA, on kind of the other the other kinds of tracking systems and sort of management systems that are out there for possible consideration.

So we do make a recommendation that if at some point it seems warranted, the city might consider examining alternative or possible other systems to use.

These are at least a couple of the systems that are sort of best in class on the market, and so we just did a comparison of those sort of basic capabilities and workflows.

But I think just a caution across all of these, you know, they really are about record tracking and management.

They are not, I'm gonna go into your system and pull stuff out and retrieve it.

That is still a profoundly manual process.

That is still something where staff are searching through emails and folders and documents and department databases, and that would happen regardless of the system choice that you made.

And for all kinds of reasons, especially from a security vantage point, that probably makes sense, but I do think it's worth considering that it is inevitably going to be a set of hours and labor and expectation for city staff to have to navigate as they're managing through what these requests look like.

There is no automated fix that is gonna go in and find all this kind of stuff and pull it back.

It just, you know, sort of really isn't available, at least through the mechanisms that are out there for public data request tracking and management.

So a little bit about volume.

So, you know, we pulled back all of those sort of 75,000 plus.

We looked at 22 through 25.

The 25 data was not the full year of data, um, and you know it often takes a while to wrap up the previous year's requests.

So we're gonna see a little bit of a sort of lag experience as it relates to that 2025 data.

So I think a few key notes, growth year over year.

Yes, you actually are seeing more requests come through.

So I think the stress that the staff are feeling is a real stress.

Um so that volume of requests has ticked up.

Um you've got an average close time that actually has improved over time from kind of a 20-day average to about a 12-day average from 22 to 24.

Um, but you have got a really wide range for what it actually takes for folks to actually be able to accomplish that close.

Um so you've got a median of four days, lots of stuff is pretty quick, and then you've got some requests that are lots and lots and lots and lots of days, right?

So you really do have a volume of complicated requests that are taking the staff a pretty significant period of time to make their way through.

Um, so I would say the the data are distributed not in a normal fashion.

Um, if we were to think about kind of a you know bell curve distribution.

Um, got about half a percent of sort of open requests, meaning you're closing really, really, really well.

So there's not a big volume of those opens.

Um I suspect you're hearing a lot about those opens, but in the volume, you're not seeing that giant open.

Yeah.

Yep, a question from Ms.

Coleman.

Quick question, sorry if I'm jumping ahead.

Um that is a shocking increase over three years.

Uh yeah.

Mm-hmm.

Yes.

Question mark.

Um, are we is that comparable to the increase that other cities are seeing?

Is do we have any idea of what's driving that?

So we don't have specific volume information from the other jurisdictions, but when we did have those conversations, and you'll see a little bit in here and a little bit in the report, everybody is reporting that pretty significant uptick over time, right?

Like just an increasing appetite for access to public available data.

Um, and every place, um, and Audrey was the one doing those jurisdiction calls, every place was like, yeah, we're watching the volume, just jump and jump and jump.

So it is startling.

I think when we finally cracked open and look at the date looked at the data, we were we were surprised.

Um, but it was confirmed by the jurisdictions that we talked to.

Yep, everybody is seeing that that increase.

Can I ask a quick follow up question?

Yeah, which is just um I'm assuming we picked 2022 as a random start date for we were looking at this.

Do we have any?

I guess I'm and maybe there's something there.

We recommended that because it can't we came out of COVID during that time.

So there was a tick.

Okay, and it's because during the digital incident, a lot of not digital, during the um COVID phase of our lives, as you recall, everybody was working remotely.

So there was a lot of interest about connecting with other jurisdictions, and that portal had a lot of interest.

So we thought that was its good starting point and interesting.

Yeah.

Okay.

Thank you.

Yeah.

We go back further.

Yeah.

And you were fully fully online in GovQA, right?

It wasn't a you weren't in the on-ramp period, right?

You were fully stable inside GovQA and its utility.

Yeah.

Correct.

Okay.

Just keep going.

Yeah, so you've got um uh sort of a subset of departments with kind of higher open request rates, and I think it's because it's simply because of the complexity of what's coming through.

But again, we were looking at high-level count and volume information.

We intentionally were not looking at the detail inside what those requests were, and that was really an agreement that we made as it related to our scope initially.

Um, as you can expect, police department is the lion's share of the request, safety and inspections next, um, site administrator fire, um, uh those um that there's sort of a which department and then which type of requests are coming through.

So there's sort of a couple of ways that we think about those variables.

Um, sorry, I just want to make sure we're reading this right.

So the police department is 95% of all requests, and the next highest is 2.2%.

Yeah.

Mm-hmm.

Yeah.

So it's good use of lion's share.

Yeah, lion's share.

Lion's share.

Pretty much everyone.

Okay, keep it up.

Yeah, yeah, yeah.

Um you have seen in your human rights um your HREO area some requests that have sort of grown year over year.

Um, so that's a bit of a jump.

Um, yeah, and just kind of that like like I alluded to on the previous slide.

You've got some complexity in some of the non-police requests that are coming through, which likely result in some of that longer close time.

Um, so I think kind of the the takeaway, this is sort of the population perspective.

Um, improvements are gonna have, especially improvements related to like police response are gonna have the broadest impact because they are again the lion lion's share.

Um, but again, that citywide consistency, and you're gonna see a little bit of that threaded through the recommendations, and we heard that in some of the department feedback is gonna matter across all of the departments, right?

So you've sort of got this one one big department, but given that you're seeing this volume increase, you know, kind of year over year.

Um, I suspect you know, you're gonna start to see that need across all of your departments, especially with those departments that have got those complex requests.

All right, so let's talk about talking to folks.

Um, so just from a staff vantage point, um, effective GovQA is a tool effective for tracking, routing, communicating with requesters, kind of keeping that documentation.

It's got some advantages.

As I said before, it doesn't go do the job for you, it doesn't go retrieve the records, um, it doesn't search email, shared drives, all the kinds of things.

That really is the lion's share of the labor that the staff are pouring into this.

Um, and then repositories that are kind of outside the platform.

Um, so repository that allows you to communicate with the requester, um, it does GovQA itself have some flexibility, and there are departments that have done a little bit of workflow work, um, particularly police have done some workflow work.

Um, but that does mean that you've got some departments that have done some of that, and some departments that haven't done some of that, some departments that behave certain ways, and some departments that behave other ways.

So you've just got some inconsistent workflows across those departments.

So, um, and then I think the age-old, it's got a lot of cool stuff hidden, but if you don't use it regularly or have the time to dig into it, you're just not gonna be able to unleash all of those capabilities.

Um, so sort of that underutilized power of the tool, um, yeah, that that exists.

Uh let's see, kind of challenges across the system.

So, again, manual.

Staff have to do a lot of interpretation, routing, splitting of requests, um, and a lot of stuff that comes in centrally is broken apart into different pieces and rerouted in dots in lots of different ways.

Um, those email requests are really resource intensive.

So, right, I want all the emails related to, right?

That is a lot of volume of information that's got to be sort of pulled back and processed by staff in order to be able to be uploaded.

And I think there was a sense of like we've watched all this volume increase, and we have not had a whole lot of increase in staff to be able to manage and process through this through this volume.

So I think the you know Data Practices Act, I think folks feel like still is highly interpretable.

It is not a zero-one black and white sort of set of expectations, rules, and so I think just that that is something around which folks are sort of feeling.

And then I think there's this sense that like folks who are putting in requests don't appreciate the amount of time that it takes for city staff actually to be able to respond to those.

And while we want folks to have high expectations of the city and turnaround time, the volume of hours that it takes to put in to do that response and to do that close is pretty substantial.

And so that that sort of, I'm gonna move over here, that sort of uh sort of disproportionate sort of match in reality.

Um, I think is is sort of a challenge for folks.

Some insights from the peer jurisdictions.

So I think again, back to this question, consistent themes, requests are growing in volume and complexity everywhere, especially those email-intensive requests.

So those give me all of the emails, insert the rest of the sentence here.

Um again, reinforcing technology helps with that tracking, but most of the work is done outside of the system.

Somebody said 90% of the work in their particular area.

So, and jurisdictions are primarily using GovQA, although there are jurisdictions that are using Excel spreadsheets to track.

Mm-hmm.

Yep, yeah.

Uh clear written procedures, defined staff roles, regular training from peer jurisdictions, kind of the things that are really good differentiators about being able to kind of button up the process, make sure you're navigating sort of effective and efficient programs.

That proactive disclosure, and I know we've talked about this, um, so stuff that could be released public, proactively being released public, so that folks aren't repeatedly asking you for it, is something that can sort of head off at the pass a little bit of this workload and just improve that public access more broadly.

And I think just broadly, St.

Paul's challenges are not unique.

Um, so this isn't a particularly exceptional set of experiences.

This is the set of experiences that that peer jurisdictions are having.

Um, and kind of thinking about that structured intake, how are folks putting in the request, and how are you managing workflows and department expectations are probably two of the things that would help sort of enable just some additional efficiencies.

And so that's something that we heard from from the peer jurisdictions.

All right, so a little bit of uh sort of survey feedback.

So we had 13 department leads give us some feedback.

Um about two-thirds of folks are you know, kind of users for at least two years.

Um, most folks uh are somewhere in the kind of like logging in weekly, they may be logging in less frequently.

Um most of the folks that were responding are processing requests themselves, and then they are also assigning requests to other folks, right?

So it's a little bit of a both and that's happening.

If they we ask them to rate the functionality of GovQA, about half said somewhat functional, about a quarter, very functional, and about a quarter, you know, not functional.

Um, so sort of a distribution.

The things that folks found difficult, understanding what's really needed in order to be able to fulfill a request, kind of unpacking that and having to figure that out, thinking about deadlines and closing requests, so sort of just being accountable to that constantly, that constant list ticked cue that folks are trying to manage, and then kind of running the reports to try to figure out should I be looking at reports in order to be able to manage this, right?

Like I think there's sort of a variable use across departments of kind of that reporting capability.

What would help?

Guidance on best practices and workflows, um, what should I be doing?

Um, request intake form, so again, your public portal doing some work, kind of amping up or elevating that, making that much clearer for folks to put information in so that it's easier when it shows up in the system for city staff to respond to.

Automation, some of those kind of features, seeing if some of that could be enabled where possible.

And then just kind of training.

There was definitely this, I got trained, and I could probably use another training, kind of kind of set of feedback.

And I think just kind of the cross department communication and figuring out who's responsible when a request has got multiple parts that spans multiple departments is just complicated.

And everybody was like, that's just hard, right?

When we're crossing departments, it's just gonna be it's gonna be noisy, it's gonna be tricky, and that's gonna be a hard part because we're gonna go back and forth and back and forth and back and forth.

Nia, I feel like I'm talking about like a day in the life of yours.

Uh again, so and we talked about this last time.

We surveyed a whole lot of portal users.

We did not hear from a ton of portal users, but what we did hear is here.

Um, so uh a lot of folks said it was easy to know which department to submit a request to.

Most folks found getting into that portal pretty easy to do.

Um, lots of folks said it was accessible in the preferred language.

Um I will comment that that's probably a biased set of humans who were responding to the survey.

Um, but you know, sort of we have the data that we have.

Um about three-quarters said filling out that request was easy, um, and about a quarter said, you know what, it's it's somewhat difficult or it's pretty difficult.

Improvements, and again, we offered to folks the option to select particular improvements.

So this was not a I'm coming up with my own idea.

This was really in partnership with Greg and Nia, let's be really thoughtful about what kinds of improvements we were gonna offer for folks to select.

So folks really want to know about their request status and next steps, right?

So some clear information about that.

Instructions and examples of what to include.

So when I put in a request, what should I be putting in?

Um I'd love a little bit more insight about that.

Department roles.

So there's a little bit of kind of pick a department, but like understanding what it is that the departments do.

I will say that's a little bit of if you'll recall the last project that we did, inside out City Speak versus folks who are coming to that portal and may not be as familiar with departments and all of the things that are really kind of insider language versus everybody across the city language.

Um, and then just kind of like simplifying what that request form looks like and some links to publicly available data.

You're gonna hear that theme over and over.

If I don't have to ask you for it, it would be great if I could just have access to it.

Yeah.

Oh my sorry for all the questions.

No, yeah.

So I'm surprised, given that 95% of the requests come in are for the police department.

I'm surprised that 29% of our requesters extrapolating a little bit, but don't necessarily feel clear about which department they should be submitting to.

Again, it seems to me like it would likely be obvious if you were submitting to the police department or not.

Is it that like 95%?

Do you all suspect, understand that we don't have a ton of respondents here, that 95% of people are like very clear they're going to the police department, and it's that 5% that are like, is this DSI?

Is this PED?

Is it public works?

Or do we think people are like, is this DSI or the police department that I should be submitting this request to?

I suspect, I mean, I suspect a couple things are going on, right?

I suspect it really is that like 5% who are like, I am really not sure where this is sort of gonna go.

Um I also suspect, and we sort of had a sense for this, like you've got some repeated requesters, and you have some repeated requesters that are not about police, they're about other things, and so I suspect we also heard from those folks in this, right?

Since we had such a low response rate, I'm like, folks who wanted to respond responded to this, right?

And so I suspect that is a not a uniformly distributed set of humans who are who are responding here.

So a little bit of a both and um in that response.

So, yeah.

All right, so some recommendations.

Alright, um, first one, which I think, again, we heard uh make better use of that GovQA tool, right?

So there's some stuff inside of it, right?

It's kind of a hemi, like actually enabling some of its functionality, um, uh leveraging those features through training, shared practices, cross-department review, doing that stuff.

So really making the best use of the current platform first.

And you'll know it at the end before exploring the possibility of other platforms, right?

So it's kind of like make sure you're really maximizing use of what you've got hands on.

Coming back around to that structured intake process.

So really unpacking what that online portal looks like.

You know, are there some things that you can revise?

That public request form, more specific information, reducing that back and forth clarification burden, rerouting.

Yeah, so thinking about kind of that front end, that input piece.

Citywide procedures, again, you've sort of let every department make a set of decisions for themselves about what makes sense as it relates to that management.

And so really considering the possibility of saying, you know what, across the board, we're gonna have some consistency.

So we're gonna route the same way, we're gonna document the same way, we're gonna communicate the same way, and when we've got those lots of pieces and parts, either parent and child or multi-department requests, we're gonna behave in the same fashion when we manage those.

Um, because right now it's really dependent upon who's kind of the primary actor for the first start, and then that is what dictates how the rest of that request is dealt with.

Staff training and support, so regular training, some refresher sessions, some peer learning, right?

You do have lots of folks that are touching the system, each of whom have some expertise in the thing that they've dug into, so really letting those folks actually get exposed to each other in a peer fashion, just to build that consistency to build that confidence.

So, and again, we did hear from folks who were like, yeah, I got trained by the city clerk and had my first half hour, and then I've been on my own since then.

Okay, right.

So, so just a chance to be able to come back around and really create that like almost community of practice for folks who are using GovQA, which might be helpful.

Taking a look at that reporting data pretty regularly.

Um, so and I don't have a sense for how frequently you're taking a look at it, but just that look, where are we at, where are the bottlenecks, where are we moving things through, who's doing well and can we learn from, who seems to be stuck and having longer close times, and where do we need to potentially lean in and offer some additional support or some additional improvement opportunities?

Um, back to the stuff folks are requesting frequently, make that easier to find.

Um, and I think that's both publicly making that easier to find as well as internally for the internal department staff.

So within departments, stuff that we're gonna get asked for a lot or across departments, stuff that we're gonna get asked for a lot if it's possible to make those easier to access inside the walls or up on, you know, quite literally the city's website, so that staff are just not having to respond to some of the same things.

Um, and then I would say periodically take a look at whether or not technology is meeting the city's needs.

Um if your volume and requests grow the same way they have grown over the next year or two, then it may be, you know, at some point a place where you're like, all right, do we want to consider an alternative to GovQA from a system vantage point?

So let's see, public facing information.

So in addition to the portal itself, just some of that, like again, and I think Nia and Greg already knew this when they were starting to think through what kinds of questions we were thinking about asking portal users, thinking about amping up that FAQ, thinking about some more guidance materials, and thinking about just kind of status communications, just so folks know what is happening, right?

Because I think a lot of the feedback you're getting is just I don't know what's happening.

Um, so just some expectations about what that communication pattern should look like.

Um, and then there's you know, looking farther down the road, um, you know, kind of the can you allow for routing based upon subject?

So actually enable that inside the system.

Um, can you think about more proactive data that's released, right?

So more and more and more of that disclosure that happens without having to have folks put in requests.

And then I know, again, you're thinking about dedicated data practices staff, right?

So, what does it mean to have that set that expectation?

And what load are they gonna pull off of those department heads, and what is that balance gonna look like?

Um, yeah.

So that is the story that we have so far.

You have a big report to make your way through with lots of data in it.

You're also gonna get a technical appendix, which we're assembling, which is essentially everything that peer jurisdictions gave us that they said it was okay for us to share.

And then we're just going to pull lots of stuff from the state just to make that available inside that technical appendix.

And you can see the documents that are going to be in there in the back of this existing report.

So but we really wanted you to pay attention to the sort of primary data collection pieces in the existing report itself.

So, and again, happy to take feedback through Friday, happy to take feedback after Friday, and we are going to have some sit-downs with some of the team, city attorney staff, just to get some additional feedback about things that we should be tuning as it relates to the report.

Great.

Well, a lot of food for thought.

Thank you, Dr.

Brett.

I am curious to see we want to make sure we have enough time for our next conversation, but I want to open it up to my colleagues on the committee just to see what questions or thoughts people have.

I see Ms.

Johnson, I see Ms.

Coleman.

Well, thank you.

Well, first, just thank you for kind of getting this to us so we can look at it.

And I I think from a procedural point, I just have a question around like how far in the weeds you were weeds you were able to get since 95% of the work happens outside of the Gov QA system itself.

Like how far in the weeds, I guess, were you able to get with just understanding the what happens after the request comes in piece?

Yeah, yeah.

So what you're reading in the report is what we were able to get into from a weed vantage point.

Okay.

I will say kind of the sort of the cracking open those individual requests and seeing what they look like from a level down from a detail vantage point is not something that we did, and it's not something that we did on purpose, right?

We really were kind of just trying to look at high level volume types, trends, that kind of stuff.

Um, so but I think the, you know, we interviewed Nia and Greg, and then we did um, you know, sort of the department survey, so you can get a sense for, you know, I'm telling you, most of the work that we're doing is outside of that, outside of that system.

So what you're seeing is about the depth that we got to.

Okay.

Without frankly unpacking every single one of those 75,000 requests at a level down, which we were really intentionally setting a boundary around, at least right now.

This may be something that in the future you say, you know what, in two years, we'd like to look at this again and maybe go a little deeper.

Um, but you're in a particular scenario right now where we put some edges around that.

Okay, thank you so much.

Yeah, yeah, great question.

Miss Coleman.

Um, thank you.

I will also echo the thanks.

It's obvious that this is something that's taking up a lot of city time and resources.

Um, so having just more insight into the process and what's working in isn't is super helpful.

And your last answer might have been the answer to this question as well, but I'm just curious about the recommendation to make frequently requested information easier to find and looking in here and seeing that you know, this a next step for the city might be figuring out what that information is that is frequently requested and then figuring out if we could and how we would make it public.

Do you have anything else to share that or anything else that came up in your all's research as to anything that people were bringing up consistently or that other jurisdictions mentioned they had proactively made public in order to try to work around some of those challenges?

Oh, that's a good um.

Let me talk to Audrey about whether or not we heard that in interviews.

I mean, I think we heard it at a high sort of like general level.

Yeah, as much as you can make public make public, um, that's what the like best practice literature says, as much as you can make public make public.

Um, but we'll take a look at those interviews again and see if there's some specificity about like, hey, here's what we made public.

Um, yeah, so look, we can take a look at that.

Um, and then I will say that potentially is again where we to pull open those requests and do probably a qualitative analysis of those requests, we might be able to share back something like that.

Again, we put some edges around this, but that is a future possibility if you were to say, hey, let's take a look at a random sample so that we could get a sense for repeated requests, maybe generate a possible qualitative list of most frequently what of that could be public, what of that could not be public, right?

Thank you.

Yeah, yeah.

Miss Delworth.

Thank you.

Um, I'm just curious, in terms of the 75,000, you weren't able to determine if they were discrete 75,000.

I mean, some of them could have been repeat.

Requests or could you tell if they were all individual?

Yeah, they were, I mean, it's 75,000 individual.

I mean, they could have been but they could have been repeat um requests.

Somebody could have, yeah, come in repeatedly.

Um, yeah, so and we didn't we didn't have enough information to be able to say we're gonna deduplicate based upon anything, right?

Because we didn't know what we didn't pull was I don't know who requested, right?

We were really trying to sort of only have a sense for just sort of straight out volume, right?

So um, but at least from the staff vantage point, logged into the system, it shows up as if it is 75,000 plus individual, maybe not independent, but individual um requests coming through.

Yeah.

And could you tell if there were um like um peak times?

Who I don't know that we have timestamp in there.

We didn't pull that record.

Yeah, I don't think we pulled timestamp, but that's a great question.

Uh-huh.

Okay.

Time stamp.

Yeah, yep, yeah.

Ms.

Delworth, were you getting at like time of year, time of day?

Time of year, I guess.

Really?

And or what was happening in the world, so to speak, as to what they were requesting, yeah.

Particularly since it's related to it seems like to the police, but I'm curious if it's there's duplication and ebb and flow.

Yeah, yep, yeah.

Yeah, we really just, you know, we have a flag on year, right?

Inside that.

So yeah.

And again, we were trying to be, I think, careful about what we were what we were allowed to take a look at and make available more um publicly, but that's also a great perhaps additional, like, are there some timestamp questions?

Are there some qualitative, you know, sort of repeated public, you know, sort of ask kinds of questions that we could maybe do a next level down analysis.

Yeah, thank you.

I think we are gonna need to wrap here, knowing that there may be other questions.

I would really encourage my colleagues on the committee to reach out to Dr.

Britt, reach out to Nia in between now and Friday.

Send your feedback, um, and in particular things that you would love to see presented when this comes to the council, like additional questions you want to see dug into.

Um, and Dr.

Britt, I also think that it would be a good idea going forward to have um now that we have such a strong partnership with our administration to have you come and do a presentation with maybe the the council members and the mayor and administration after it's presented to the council, so and maybe any relevant members of her staff.

So I'm just thinking about that now.

I want to say that loud.

Um, so thank you so much.

Oh my goodness.

Very impressive body of work.

I have a ton of questions that I'm not asking right now because I'm trying to keep this on time.

Um, but thank you.

Okay, thank you.

Thank you, thank you.

Um, and with that, we are going to uh and we'll look forward to having you back at council a month or so.

We'll see, whenever we can fit it in.

Um, sorry, can you pull your microphone closer to me?

Sorry.

Oh, sorry.

I think we're looking at July.

Yeah.

So we will coordinate, so July 8th was the date we had originally discussed as a possibility.

Okay, presenting of console.

Great.

All right, we are gonna pivot to our next topic, which is new topics.

Um Nia, do you want to take us away?

Or over to community advisor.

There were four of us who met to discuss and ranked the score.

So um, but I think everyone agreed that Jeremy would do a nice job.

Okay.

Well, thank you, Nia.

Yeah, and Chair Naker, um, as Nia said, four of us met.

I guess you could call it our risk assessment subcommittee uh to go over the uh list of proposed projects.

So there were five of them, and I think the the key takeaway is um they're all uh very good audit topics.

Uh they all scored very highly on the uh scoring matrix uh that we used.

Um, and so that affirms uh we have good topics.

Any of them uh would be a good topic for an audit.

Um, so just leading off with that uh and then getting into the details.

So um first off, I think what we uh propose is combining uh two of them, um, which are the business licensing and building code operation audit topics, and the reason for combining these is because they're both within DSI, and my understanding is they make use of the same uh system, which DSI is uh using a new uh system right now, so it might make sense uh to combine those.

Um and then so then the other topics were college savings accounts, district council reform, and street maintenance and potholes.

Um so we went through our rubric scoring matrix, which you have copies uh in front of you, and so we have rankings of low, medium, and high for six different uh questions, and um we had two of the projects tie.

Well, actually, we had two ties.

So the the top two um scoring uh topics were the DSI business licensing and building code operation, which I uh had mentioned first, um, but with a caveat that because of the new DSI system, we would recommend maybe that being um the last topic audited in 2027.

So I think what we're looking at is maybe the remainder of 26, and then also looking at calendar year 27 uh for scheduling these audit topics, and so we kind of looked at maybe there being three slots uh for timing of putting those in.

Um and so if that being the case, potentially because of this new system and the the learning curve and um getting that all in place that that would take maybe the third slot would be our recommendation and that scored the highest, but um tying with that project was the college savings account project, and that one um I think mainly scored highly because of the backing of the mayor's office and the interest in doing that um uh audit um to look at that program kind of a performance evaluation.

Is this uh program running uh as intended?

Um so that would be our recommendation for the next one.

So those two are tied, and then the final two were also tied, but with only one point difference.

Um we nearly had a four-way tie, but then um the final two then were district council reform and street maintenance and potholes.

Um so in thinking about if we have three slots, it might be difficult to decide which one of these four projects now to drop.

I guess it would be a choice between district council reform and street maintenance and potholes, at least uh as it comes from the recommendation from the risk assessment subcommittee.

So any questions about what we did.

We we met it was last week.

Yeah, so it was uh community advisors, uh McVay, Dilworth, and then Nina and myself.

Great.

Well, thank you for the report and thank you for the work.

Um it's probably the most important um part of our job is to figure out what we actually want to look at, and this is a huge part of it.

So thank you for doing this.

Um, just one clarifying question before I open it up to others.

So I'm not seeing the shading on the business licensing one, but I'm assuming that all it was shaded all threes.

Okay.

Um questions from my colleagues or comments.

And again, our charge, our goal is to choose three topics uh today to recommend to the council.

Um would also want to um, as usual, make sure that we're doing due diligence with the relevant department directors and staff on each of those topics prior to bringing them to the council.

It was great to be able to speak with the mayor, but we also want to make sure we understand from the departments that um you know there will be buy-in to these topics.

Um, but for today, we want to get down to three.

So I'm looking to folks for questions and comments.

Madam Chair, Mr.

McVay.

Thank you.

Um, I will start off just by saying how thrilled I am that we have these topics because I think that residents really did such a wonderful job, this audit cycle, and I'm really proud to be a resident of St.

Paul because how many people actually came to the district council meetings and shared their ideas and their concerns with us, and I think that's really valuable, especially in this day and age when there's a lot of um eroding confidence in our public institutions, and I think what we do here on this committee is really important in terms of demonstrating that we're all about government accountability and improving public service delivery for the folks that live here in St.

Paul, all 300,000 plus of them.

So I have a couple comments.

Regarding district council reform, again, this is from the district councils themselves, from their staff, and also from like the officers on their boards, and so it's important to keep that in mind that they're gonna they're gonna have some motivation behind this, is what I'm trying to get at, but still um they really would like us to they would highly encourage us to consider looking at uh their op their organization, their structure, their finances, how they operate, they're your core services, because a lot of them are struggling financially right now.

Like in conversations with some of the executive directors or some of the district councils, they are uh having some difficulties, especially with like indirect costing with um pets still, and I think you know who I'm talking about, madam chair.

Um, and there's also a few of it's not just one of them, though, it's actually multiple that would express this concern.

And I will say that with district council reform, I think given how many of the district councils brought this up in terms of the board members and their staff.

I think that if we don't do this within this audit cycle, it's going to create some problems in terms of our relationship as a committee with the district councils going forward in terms of our ability to work with them to bring the public in and get feedback for future audit topics simply because they may say, like, oh, you brought this idea, we really want you to go with this idea, and you brought it to the full attention of the committee, and I um they may just be like, we don't want to really go through with this process in the future.

Because they are our eyes and ears on the ground here in St.

Paul.

And so that's just something to keep in mind.

I'm not saying necessarily needs to be maybe this year or 2027, but it's definitely something we need to look at at some point, whether it's in the next three years if we don't get to do it this year or next year.

So I just want to make sure the committee aware of that in particular.

The other thing is with college savings accounts, and um I am sorry, I'm starting to lose my voice.

I'm getting cold, so bear with me here.

Just getting emotional about the work of the audit committee.

Oh, yes, absolutely.

I mean, I it happens to the best of us.

I'm so moved right now.

Um, with the college savings accounts, I know this is a really big uh this is something that is being uh the mayor, she is a proponent of this because I know she had a hand in creating this program years ago.

Uh I will say this there is a lot of information out there already about how this is how the program is working.

There's not there could be more information because the problem is is we don't have like depending on what comes out from the office of financial empowerment.

There's different metrics that they're using based off of how you frame the question in terms of like participation and then also like um how many because of the fact we don't even automatically enroll um kids when they're born here in St.

Paul into the program that that that creates some difficulties too, which is unusual because most you know programs are automatic enrollment, but I'm getting out I'm getting to beyond my wheelhouse here.

So that being said, um part of me is wondering if that topic, I think there's merit for us to do an audit of it.

I do think that, but I also wonder if this is almost something that could be evaluated during the budget process as well regarding that program, and that could be something that had between the mayor and her administration and the council.

Again, I'm just putting these ideas out there.

I'm just kind of thinking aloud at this point, trying to determine what makes the most sense for us as a committee to be doing, and I'm just initiating that conversation at this point.

So I'm done rambling, Madam Chair.

Thank you, Mr.

McVay.

Never rambling.

Other thoughts?

Mr.

Donnelly.

Okay.

First of all, great work by the team, sincerely.

Like we have four awesome topics.

I love the rubrics, taking all the information, getting it to this point is a real achievement.

Thank you for doing that.

So wherever we land, it's a good outcome.

Um, so when I worked in a factory, we would do lean events, and we would sort of use criteria to decide what we would work on to improve.

And one of the key points would be like if we did a lean event on this part or this process, could we extrapolate those findings to different parts and processes so that what's gained is not just in this one thing, it's over here.

And that's kind of where I'm at with college-bound, where it's like my counsel to the city is to say if you're just gonna look at college bound and that's the only lens you're looking at this, it's probably not a ton of value.

But if you're gonna say, we really need to look at the programs across the city, how we fund them, how we evaluate performance, grant management, and we're gonna take the findings from college-bound and use it as a basis for improvement across the city for all our programs, then I see a ton of value.

And it's really that's a question whether the city how you're approaching this, what you're thinking about programs, and so that that would be my counsel to say, like, think about if there's a genuine interest to learn about program effectiveness and integrity, and whether there's the bandwidth to to take these findings and put it across the city or not.

And I think that's some of the deciding factor, whether you see value in that topic or not.

But again, good job here.

These are all great topics, and I think you'd find value going for any of them, really.

Thanks, Mr.

Channel.

Ms.

Johnson.

Thank you.

I really appreciate just kind of listening to everyone's feedback.

I think I um and also to the folks that took the extra time to go through the scoring and to actually like review the topics and give your thoughts on your rubric scorings.

I think that that's also really important because we have to, you know, kind of see where folks land.

I know during the last meeting, we talked a little bit of our around our own scoring.

So I'll just simply speak from my vantage point.

Um, I uh like I think for some of this, um, I like to think about like the next steps and like what we do with the information that is given to us.

Um I'm unclear what the audit, like first for um the college savings accounts piece, I think I'm a little unclear of like reviewing whether or not it's work it's doing what it's intended and then trying to understand where we would go from there if we found that it was or even if we found out it wasn't.

And I just so I'm a little like I'm kind of like what would the outcome create and also what would the outcomes be, and that would be something that I would really have uh like I think we should.

If we were to evaluate the college of savings accounts, I think we should have a clear understanding of what that means for the findings, and especially because there is an opportunity, I think.

Being that I believe that it sounds like it was, you know, the mayor's request.

I think that would be what I would want to know is like to what, not to what end, but kind of to what in like what exactly are we aiming to look at and what we're looking for?

Because when I look at the description, it's um like what is what does equity mean um when it comes to like the effectiveness, the inform the equity of the program, that of all of the others, like they're very like look at the organization, the core functions of the district councils and looking at the forming formulas.

Um for the building code operations, there's also kind of like that um specific specificity that I know where they're like who they'd be talking to, where they'd be going with the program evaluation of just like the economy efficiency, effectiveness, and equity of both college-bound and college-bound boost, like that to me requires a little bit more of a follow-up.

So I respect that it had higher scoring, and I think to the committee's reviews, but I think I think that if that's the decision that we go with that it's gonna have a it needs to have a little bit more questions asked.

Because I think the description as written is subjective, not research-based, and that's a little concerning for me.

Um, and then also on the flip side, I'm just like, and depending on what the findings are, I'm just like, well, what does that mean?

You know, so that because it's a program, it's tied to multiple funding streams, it's you know, it does in my eyes it carries actually a little bit more risk on the operational side on the financial side, on the compliance and the reputational side.

Like to me, like those risks matter of what the findings would be if we're not confident in the findings themselves.

And so that would be the question that I would ask.

Um for probably for other administration and also for the for uh folks that are looking through this topic because it you just my brain goes into we have multiple external funding sources pulling into this as well, and I just would want to make sure that we were setting it up for success versus making it like literally creating a public facing documentation to ask questions and things of that nature.

That would be the only thing of concern for me of that topic if we moved forward, if that makes sense.

Yeah, thanks, Ms.

Johnson.

Ms.

Coleman, and then I'm gonna move us towards we have to drop one.

So move us towards that.

Thank you.

Um, like a quick and I will give my proposal for when we drop uh super appreciate the subcommittee that put this together.

It's really helpful.

I think to see that all of our topics are strong and we think could sort of meet the basic requirements that we have set out.

I to my mind the DSI one is a no-brainer, as is the streaming and support holes, um, for reasons that I think are articulated at our last meeting, but those are core functions of the city.

There are things that we need to do really well.

There are things that we spend a ton of money on.

We hear about them from residents every single day.

I think we'd be very remiss to not move forward with those two audits.

So to my mind, it sort of leaves the question um between college savings accounts and district council reform.

I will say I would be a vote for the district council reform as the third topic.

I think both as a mark of appreciation, like uh Mr.

McVay was stating, you know, of how much work the district council's put into getting us feedback on this.

I share some of the concerns uh that council member Johnson raised about the college savings accounts.

The other one that I just think about a lot is we put a lot of trust in our district councils.

Um that's reflected to some extent in our city finance funding of the system, but not necessarily entirely reflected in that since we don't give them very much money.

Uh, but we put a lot of trust in them to be the voice of the community, and I think that they have a great deal of influence in our city, and so understanding how they work, how they're currently working, how we can support them, how we can take advantage of this quite significant role that they play is to the benefit of everybody in the city.

So those would be the three that I would be supportive of moving forward with.

Great.

Thanks, Ms.

Coleman.

Well, I think I'm gonna open it up to kind of assuming that's not a formal motion, but sort of a proposal, um, to see what others I think it mirrors some of what we've heard around the table, and I'm curious how others would respond to the idea of having our three be dropping college savings accounts essentially and having the other three be the ones that we move forward with.

What's your opinion, Councilman?

I'm curious.

Um I actually would rank, I would agree, but because I would rank the first parameter a little bit differently for college-bound.

I think it's funny we we started, we were interested in that topic in part because we were interested in the mayor was really interested.

I think as a result, you want a sort of sweet spot.

You don't want the mayor to be zero interested, but in some ways, um, because the mayor has expressed interest in evaluating this program, and I I has I think also conveyed that to her own staff.

Um, and because there is so much data available, I think the likelihood that I think two is actually a more accurate rating for this one.

Topic has not been studied within the last five years, but City has plans to address the topic.

I think it's more likely than not that that will be looked at.

Um, and I think some of these other topics maybe not so much.

Um, I also had not thought about the responsiveness to the district councils wanting to be studied.

I think I've always thought about this topic as something that would not maybe be welcomed by them, but um, as our community engagement chair is attesting to, um, and also maybe sole member of the committee at times, um, the subcommittee, I I think that is really that sits well with me.

So I would I would support the proposal.

Mr.

Lastetter.

Yeah, Chair Naker.

I think that's a very important point.

The the risk ranking, if if it as um if there are plans in place already to address the college bound, um, you know, that would change uh the the results of our scoring on that, which we were not aware of at the time, but um, yeah, what I'm hearing around the table is you know, there's concern about the scope of that engagement.

We'd be need to be very careful about what that is.

I love the suggestion from advisor Donnelly about, you know, could it be looking at um citywide programs as a whole, like best practices, what's working, what doesn't, what you know, are we gonna do similar programs in the future?

What lessons learned could we apply to those programs?

Um that could make sense.

Um but yeah, the the scoping would be key, and if we would move forward with that to council members Johnson's uh comments, you were saying what questions do we want answered, those would be the draft audit objectives.

Yeah, what questions are we wanting to answer with the engagement, and that would be uh key piece, I'd say.

Um here you and agree, and um, and by the way, I totally would echo that the committee, the subcommittee, I'm not sure that the city for sure has plans to address the topic, so it is a little bit of a maybe 2.5 on that one, but I I think just given what I've what I've heard the mayor say, I'm feeling like it might be more likely than not.

So um no balls were hidden from the the subcommittee by any means.

Um, I also just want to note that the three that we're talking about advancing um are all pretty hefty, um, and so we're gonna have to have some difficult conversations as a group about scoping.

Um, for example, district council reform, revisit the financing organization core functions of St.

Paul's 17 district councils.

Um so I think I that is not to say that we shouldn't advance these.

I think as a group, we could make an argument that it's really worth doing all of these in full, and it just might take more time, and we're talking about phases of each of these, um, rather than just trying to take a small bite off of each one.

But we're gonna have to talk about that.

I doubt that we are gonna be, you know, a three or four month study of all of street maintenance or all of business licensing and building code um at all of our district councils.

So maybe six to nine months each.

I I could easily see, and obviously we need to talk to Dr.

Britt about that, but I just say that as a caveat as we consider these.

So which would mean fewer topic selection meetings for us, that's good.

But yes, Ms.

Dilworth.

I just want to, and and I I've been listening to the conversation, and I I think I I think I want to clarify when we had the conversation about college bound, and that doesn't mean that I'm trying to say that we should be doing college bound, but and these are the description is very um high level, but the conversation that we had was a little more detailed, and it's unfortunate that it didn't, and I'm not saying anything, but it's unfortunate it didn't make its way to this document to explain kind of what we were thinking because when I um as I recall when we were talking about like the economy, it's whether is it sustainable?

What is it, you know?

So there's different there was different top things that we were looking at with equity, and even um uh Mr.

McFay said it's not something that if you're born in St.

Paul, you automatically get it.

That's the equity because if you have to apply, that's inequity for on many levels, because there's a lot of people, parents that won't apply.

So, and it's not because they don't want to, they may not it gets past them.

It gets, you know, there's different reasons why people don't apply.

We have all sorts of scholarships out in the world that people don't apply, students don't apply for.

So if it's if this is really was what they wanted to do, then they would get it automatically as soon as you were born in a St.

Paul hospital, whatever you got it.

Um, and so then and effectively an efficiency, I think.

So there was there's a little more depth to the conversation that we had just um in terms of what we were thinking.

But I also, as as Mr.

Donnelly was talking, was saying, you know, I don't know if it's a full audit that needs to happen for this particular um program, but uh, and I'm I don't know if I'm splitting hairs, but a review of some sort that helps to understand where it is, and if that's something that we can um talk to the mayor about of them doing that, I think that would be great.

Because I do think there needs to be something that answers some of these questions that the two gentlemen are even are bringing up, and that we had when we had the discussion that if we just say we're not gonna audit it or not have it on the audit or move it forward, it may not get answered.

That's just my point.

So, first of all, Ms.

Dilworth, I really appreciate that point, and I um I think it's there's there is a middle ground between we're auditing it and we don't think it needs any consideration whatsoever.

I wonder if maybe there is a communication that comes from this body, for example, to the mayor, especially given that this topic came up you know in our discussion with her to say um I think that would be actually really good closing the feedback loop of this is we ended up not deciding to audit this topic for these reasons that don't mean we don't think this program needs attention.

Um, and I really appreciate the points um that Mr.

Donnelly made and Mr.

Laseter echoed about, and those are likely things that could be considered applied to future programs or current programs when we're deciding whether or not to start or continue existing programs, not just this one.

So, if the committee, if we do decide to not move ahead with that audit, I would certainly support some sort of communication to that effect, and maybe even, and I want to recognize that um Director Owens is here who directs the office of financial empowerment.

Um, maybe even a conversation with the team as they are getting into the role and looking at the program about um what they're finding, what their plans are, um that's something that this body can certainly invite and engage in.

And we've done that in the past, inviting department directors to come and talk to us about programs separate from auditing.

Um, so Mr.

McVay.

I just have a quick question, Madam Chair.

Is it possible that we could um work in concert since we have the director here?

Could we actually work in partnership with OFE?

Perhaps they do review and they come back to us as a committee after they've done the administration does their um internal evaluation and report back to us, and then we can if we say if the council needs to make some legislative changes or whatever to the program.

I don't know if it's set up in city code or not.

I mean, I imagine it must be.

Correct me if I'm wrong, I don't know.

Right setting up changes to the like if there needs to be any reforms to the program after having that kind of dialogue with the administration, if they do their own internal review and then they come back with their um evaluation to the committee and what have you.

Because I think um uh Ms.

Dillworth brings a valid point, and we did have a conversation about like, for instance, like the fact that how the program, some of the account, I think how the accounts are structured versus like for um how like under um with section 529 and all that too.

So was that part of our conversation?

I can't recall for freshman.

Was that we had we had a robust conversation about this particular like 20 minutes, I think, right?

Is that we're all over.

Yeah, we were all over.

Can I have a clarifying question?

Yes, okay.

Just a quick clarification question.

I think just so I can know where to the timing for we're picking three topics because we feel like we have the timing to complete it with the topics that we have, or like do you guys think could you just talk about the how many like how we're picking the topics, how many topics, and then also just like is that because we think that that's something Mao will be able to get done in the course of those of the course of the end of 2027.

Is that why we're picking three at this time?

So great question.

We are um we are we typically complete two audits a year, and rather than spending a lot of time choosing just one topic when we often have one very close runner up.

What we decided as we're always looking to continuously improve and become more efficient, even as not a committee, was that we would choose two topics at the same time.

So we have one selection process that yields two results, one takes about half a year, the other one takes about half a year.

We are inevitably running a little bit behind schedule, and so rather than being at the start of 2026 right now, these are becoming sort of the end of 26, beginning of 27 audits.

The idea here is to send three recommendations to the council, have the council choose two, which would be the next two.

And as I was saying earlier, maybe not even the next two for the next year, but two that might take even longer than that.

Um but that's the thinking.

Okay, I I I think with that, then my follow-up comments would be.

Typically, I'd be like, come on up directly, have a conversation about it.

But I think like I have a couple questions about just really where they, you know, to be clear and also um with this particular piece.

I think it's the question for me is more if this is around program evaluation of the actual program itself, or if it's around auditing what has been done and like with the intent of correcting or like taking the information that's given to do something with it.

And um I think that they are distinctive things, and that's actually part of my hesitation around this topic, and so I'm supportive of what is being considered.

I just I do think that those three topics together, like if I had to choose between the potholes and the DSI um piece, the building co-quick piece, because those are two substance, the college savings accounts for me would make the most sense because I don't think it would take as long as the district council reforms going to be.

So I just I'm just those three together, and knowing that we're only doing two, I'll just say there are three heavy ones that probably will take a long time.

Like going through a system that's been in place for decades is gonna take a minute, and so I just realistically don't know if they'll be able, I would really value Dr.

Britt's um input on just what she thinks will be like realistic, tangible pieces by the time this gets to council, because that I mean I am curious to know how long they think it's gonna take to go through 17 systems.

Um anticipate it taking quite a bit of time.

So I'll just offer that and simply be done.

But um, and those will be my follow-ups for the administration and for the director of like what would be the goal of this study because I see the description, here are the conversations that matters to me about whether I would want to like depending on what the results would be, I could that would maybe change my vote today.

But so I have a crazy suggestion based on what you just said, Chair, especially because we get to make our own rules in the audit committee and set our own schedule.

Um we could also, rather than deciding between these today, we could do the follow-ups with the department directors that I mentioned before we make a final decision, and that could include talking to Dr.

Britt about each of these topics and getting her estimate of what we're really talking about in that in terms of a scope and timeline, and we could then come back in July to with all of that information and decide based on all of that, what which are our three?

Is that something that would help people in making?

I don't want to rush us unnecessarily.

And I'm seeing some heads nodding, and I'm seeing some okay.

I like process.

I want, I mean, as long as we're gonna do that consultation anyway.

I think it does make sense to kind of have that.

Why not?

Um, so okay.

And I I would also just say that if to the extent that the subcommittee talked about things that really weren't reflected here that you want to share either in an email afterwards or just other considerations about specifically college savings accounts, for example, feel free to do that.

Like this doesn't have to be the only record of the conversation that was had.

Um, Mr.

Lasteck.

Yeah, Chair Naker.

I think we were kind of uh swirling around a new idea too that I didn't want to get lost.

Um, the idea of maybe willing departments doing a self-assessment that maybe that could be an option.

Um, you know, we could develop the questions kind of like what I was saying before the um the audit objectives questions this committee would like answered about a program, and if uh department is willing and you know has the capacity, they could do a self-assessment and report back, and that could be a deliverable, and then this committee, you know, reviews those and you know, decides is that sufficient?

Would this be ripe for an audit in the future?

Uh, but maybe that's you know, you've said we're in innovative committee, that could be, you know, something new that we look at as a possibility.

Yeah, so maybe sort of an audit light of a self-assessment, and then that frees us up to do something else, but still um provide that support.

Still look at a worthy topic.

To worthy topics, but okay, I like that.

Thank you for bringing that back home so we did not lose it.

With federal departments on with on sub programs within federal agencies at the government accountability office.

That wouldn't be a bad idea to actually adopt as a practice here at the audit committee for our city.

So we could actually have a more thorough um understanding of things going on in city departments on a regular basis.

Okay.

So I'm hearing, I want to wrap us up.

This is great.

That we are going to no, this is this is all it's all good.

Um, we are going to talk to departments and Dr.

Britt about these topics.

We are also going to raise the possibility of a self-assessment, um, sort of in the guise of were you already planning on doing this anyway, or would you like our support in developing some questions and doing this on your own without outside consultants doing it?

Um, and then we will bring that back in July and we will have a real decision before us at that point.

Sound good?

Thanks everyone for the robust conversation.

Sorry, went over, but I think it was worth it.

We are adjourned.