Readiness for the AI Era: What state and local agencies should do now

Welcome everyone and thanks for joining us for today’s webinar on AI readiness for state and local government. I’m Deedee De La Cruz, director of marketing here at PayIt and I’ll be moderating today’s webinar. I’m joined by two great speakers who spend a lot of time thinking about AI. Richard Garvey is chief technology officer at PayIt where he’s responsible for the company’s technology strategy. Ben Dickie is research director at Info-Tech Research Group where he develops a comprehensive framework for enabling IT and business leaders to build world-class experiences for their customers. Today we’ll dig into what AI readiness means for an agency. Richard and Ben, thank you so much for being here. Let’s get started. Everyone’s talking about being AI ready, but it means different things to different people. What does readiness actually look like for a state or local government agency?

Ben, we’ll start with you. It’s a great question. I think the first thing to look at is having a cogent realization of what your objectives for AI actually are. We see a lot of scenarios where folks charge headlong into AI without really thinking through what are our organizational objectives, what are the key use cases that we want to enable or support using AI. So for me a huge part of AI readiness is actually doing the foundational work around what are we looking to achieve with this technology so that you can therefore target your investments target changes to people process technology in an appropriate fashion and I think again that’s the foundation that then allows you to operationalize different components in terms of data hygiene the technology stack itself linking all those things back to what are we looking to achieve with AI at the end of the day.

Yeah, I would add on to some of what Ben said. You know, the great agencies that I see uh are focusing on like three pillars. Governance, data cleanliness and training. So, like when we look at governance, it’s going to be, you know, model provider selection and tooling. Understanding you know and establishing governance around what data is being used and how that data will interact with the AI vendors approval processes for that tooling monitoring of these controls and sort of overall goals uh that ensure safety and compliance and transparency right uh when we talk about data cleanliness it’s do I have accurate data is that data fresh and complete right and and if not how am I filling those gaps in those various areas And then finally, you know, training both hands-on, right? And then having a basic theoretical understanding of how the models work for the workforce. And those things pay huge dividends in long-term efficiency.

What’s the most common misconception leaders have about getting ready for AI?
Ben, we’ll start with you. Yeah. So I think the biggest misconception that I see is that it’s all about the technology without paying full attention to parallel processes around how you’re implementing it, how you’re doing the change

management. Exactly as Richard said, the data hygiene piece, the data quality piece, how you curate knowledge, that’s all critically important. Obviously, the technology, I mean, that’s the the vehicle that drives forward your AI strategy, but there’s all of these adjacencies that have to take place from a governance and data hygiene and change

management and knowledge curation perspective that if you don’t adequately address those, you’re not going to be able to fully realize your strategy. So, I think there’s this misconception was like, I’m going to go out, I’m going to buy a chatbot, and that’s my AI strategy. It’s like, okay, well, what’s the actual use cases you’re targeting with that chatbot? How are you looking to drive forward outcomes for your constituents, for your citizens, right? That needs to be taken stock of alongside the base investments in the technology itself.

Richard, what would you add on there? And I’m also curious in terms of if there’s anything associated with cost that would cause any type of or any type of cost that would, you know, be a misconception or sticker shock or anything like that.

Yeah, absolutely. I would double down on what Ben said about, you know, ROI. You know, some leaders mistakenly treat AI as this hammer, right? And that everything is a nail. And so we can just peanut butter spread AI across all of our problems and the world would be great. And maybe someday that will be true. Uh, but that day is not today, right? So there’s, you know, very important being objective about what are we trying to do and being realistic that AI might not always be the cost-effective solution for every scenario. And that kind of gets back to a little bit of the necessity of training staff to understand token usage and context management to prevent wasteful spending when they could probably use the tool in a more cost-effective manner.

Richard, I just want to dovetail off that. I couldn’t agree more. You see a lot in the technology press these days about that sticker shock that organizations are starting to run into. Yeah, it’s funny to me in a way. It’s almost as though organizations are realizing AI actually can cost a lot of money potentially, right? So being targeted with those use cases again going back to what I mentioned right at the start of our discussion identify the top priority outcomes you want to deliver and target AI that’s critically important for, you know, avoiding some of that sticker shock that you’re hearing more and more about these days. So I think that’s a fantastic point.

That’s great. Let’s talk a little bit more about use cases. So first appointment that struggles can create some skepticism that’s you know pretty hard to undo. So, how should a leader who isn’t technical think about choosing a first use case that’s likely to succeed?

Richard, we’ll start with you this time. Yeah, I mean, I think an incremental rollout is always going to be your best strategy, right? So, you know, for example, if we’re thinking about a use case of we’ll say an AI chatbot to answer questions for residents, right? Maybe it replaces an FAQ page, right? It’s a perfect use case. You should see great success. But like get backs to get back to maybe something that both Ben and I talked about earlier, data quality, right? So, you know, you could spend a long time uh trying to make sure that your data is perfect, right? But if you maybe start by saying, “Okay, we’re going to point an LLM at our existing knowledge base.” And then as our existing staff is answering questions, they’re taking their question and putting it into the chatbot and seeing what it comes back with, right? And so you can then get some sort of incremental feedback and it also limits the sort of initial use cases uh and and scope and so you can get real quick feedback without huge investment.

Well, that’s a more of a crawl walk approach there.

Yeah. Yeah. Absolutely. So when I think about the use cases for AI, there’s really two broad spheres they fall into, right? The first is improving constituent engagement. So delivering better outcomes for your stakeholders across different public sector use cases, right? The other is improving internal operational efficiency for your own employees. And I think a lot, particularly in the public sector, a lot of organizations gravitate first and foremost to that constituent engagement piece, right? Because that’s where you’re going to see potentially transformative gains in terms of how you actually fulfill your organization’s mandate. I think the internal efficiency piece though, when you’re looking to build some initial enthusiasm and trust in AI, I think that’s an area where you can identify some quick wins. So interestingly, one of the things we hear frequently from clients when we ask them, what was the AI use case that you feel you were able to execute on with relatively little investment that your employees are actually saying we see a lot of value in this. It’s things like meeting transcription, identification of action items, next steps, right? It’s not the bleeding edge frontier transformation use case, but it’s a small thing where you’re applying AI to deliver real gains internally for employees. Once you do that, exactly to Richard’s point, then you can start to spin up some proof of concepts and pilot projects in the constituent-facing piece. Get the wins there as well, right? So again, when you’re thinking about AI, I kind of separated into those use cases driving forward gains and transformation for constituents, internal operational efficiency, and to be clear, those aren’t always mutually exclusive. But I think starting with some pilot projects that prove internally to employees that hey you know AI actually can deliver some great wins that helps build that culture of amenability towards AI going forward.

I know we’ve talked a lot about data so let’s talk more about data for a business leader who doesn’t live with data how do they know whether their agency’s data is good enough to even begin with we’ll start with you.

So there’s two ways that you can go about doing that. The first is go a little more formal way which is doing a data quality audit. This is where you’re taking either all of your data or a subset or random sample of your data and you’re looking for duplicate records, stale records, conflicting records, right? Is literally auditing what percentage of my data in a given domain do I have issues with. Uh the benefit to that is that it’s very rigorous. It gives you concrete numbers in terms of the scope of what the data quality problem may or may not be but it is time-consuming right and we find a lot of organizations don’t have the throughput internally to do a full data quality audit so oftentimes they’ll engage external partners to assist with that part particular piece of the puzzle. The other element you can look at is just go to the front lines, talk to your leaders, talk to folks that are actually serving constituents and get a sense of what’s their level of trust in the data. Uh do they have difficulty executing automatic workflows based on that data, right? So conversations with people that are in the front lines can actually as a leader that can give you some pretty good insights into the state of your data. So, if you’re hearing things from the front lines like, “I don’t trust this report. I don’t trust the data. I have to run the report multiple times before I get the correct answer.” These are things that indicate to me that you have a potentially severe data quality issue. The other piece I mentioned again is look at can we actually use the data in its current state to drive forward process automation. If the answer on that is no or it’s potentially going to expose uh constituent-facing problems if we attempt to do so that’s another red flag that you have a sphere issue with data quality.

One thing I will say on this is that the majority of organizations in my experience do struggle to varying degrees with data quality. It’s relatively rare that I speak to one of our clients where I hear from them our data is in fantastic shape. We have no issues with it. Right? So one thing I would say to anyone listening to this is if you’ve got issues with data quality, you’re in good company, right? A lot of organizations struggle with this and there’s a lot of techniques that you can then look to apply in terms of preventative and corrective data quality to address it. But in my opinion, diagnosing the problem, formal data quality audit, that’s the one way to go about it. The other way is again book some interviews with folks on the front lines and query them. What’s your trust in reports, your trust in the data? Can you execute workflows automatically with the data? That should provide some clues as to the overall health of your data quality approach as well.

Definitely. And I like that quick hack on, you know, asking the team, do they even trust the data? Because, you know, if they don’t, then you definitely have a data problem there. Richard, I’m curious what a lot of times that answer is no.
Yes, exactly. Yeah. Richard, any other ways to kind of understand whether there is a data problem?
Yeah, I mean I would agree with Ben. Nearly always there is the extent of that data problem will vary. I would suggest maybe a couple of uh quick and dirty approaches. So you know what you could do, you know, in our hypothetical you know AI assistant, right? That’s answering questions. You know, likely in a 311 or some other scenario, you’re going to have your top 10, top 50 questions. and what those answers are. You know, point an LLM at your language or at your knowledge base and ask those questions. See what it comes back with. You’ll get a really good sense pretty quickly uh what kind of data quality issues you have. Another thing that you could do is just point the LLM at your uh knowledge base and ask it how many duplicates do I have? How many conflicting answers do I have? And you’ll likely be surprised at what you find.

Yeah, those are definitely great tips there. Let’s move along to getting the team ready. And this is a question that we hear a lot, right? Just in general and overall around AI. Job security is the number one concern that government staff raise about AI. How do you have that conversation honestly without either overpromising or scaring people?
Richard, we’ll start with you. Yeah, I think the best way I can answer this is maybe with my own journey and I don’t think my own journey with AI is unique to me. I think everybody kind of goes this same journey. Back in 2022, you know, I tried out using OpenAI on, you know, programming related problems and, you know, my mileage varied significantly, right? Uh so much so that I basically set it aside to call core job related issue tasks and used it for other things. But then like checking in periodically it got better and better and better and better and I got more comfortable with using it in different ways and finding different ways to use it. And so exposure maybe is the simple summary of that like exposing people to the technology and that goes back to something Ben said earlier, right? Like meeting transcription is a perfect use case. You know where I would have to take copious notes in the past now I can get an email transcription and summary of those notes like that’s like great so the more exposure that you can get the more I think people realize that their jobs change as opposed to you know eliminate.

I think those are great points, Richard. One thing that you were describing that kind of bubbles to mind is that we occasionally talk to folks about their viewpoint around what AI is, it’s a couple years out of date, right? And the reality is this space moves so incredibly quickly that if you’re not exposed to the frontier models and capabilities, it’s very easy for users to have this outmoded viewpoint on what AI is, AI strengths and weaknesses. So the exposure piece I completely agree that’s critically important. Building that familiarity is paramount. This is a common point of trepidation we hear from a lot of our members and I think this is a challenge that is particularly pronounced in public sector. As a leader messaging around this, I think it’s critically important to highlight, you know, AI is a force multiplier, right? It’s not about replacing jobs. It’s about making folks more efficient and allowing them to deliver better, higher quality outcomes for constituents. To get there, one thing I’ll link back to is you have to be targeted and intentional with your use cases. So, just telling staff, use AI, right? Here’s a co-pilot license. Go use it. Right? That’s not an efficient way to drive this forward. As leadership, you need to have clear messaging not just about how we think AI will deliver value to the organization, but you need to do some role-based personalization of those benefits as well. Right? So, you need to be saying to particular roles, here’s how I think you should be using AI in the best way possible. Here’s some things where I believe will actually deliver value to you. It’ll make your life easier. and really focusing in terms of training, socialization, adoption approaches around call it three to four high value use cases that are personalized to that specific role. Right now in many cases what AI is promising to do is alleviate employees from having to do the more call it mundane, boring, highly repeatable parts of their jobs so that they can focus on higher value tasks, right? ideation, answering more complex inquiries from constituents rather than the really, you know, paint by number stuff. So, while AI is going to shift the types of work that people are going to be doing, I think it can ultimately lead to more fulfilling employee experiences, right? Because again, it’s removing the administrivia, right? So completely agree with what Richard said about the exposure piece and then building off of that. Again, be very targeted, intentional, personalized with the use cases that you’re driving forward with AI. Don’t just tell people you have to use AI. Obviously, again, show them, right? Do the lunch and learn, do the training, find advocates or evangelists within departments that can drive forward familiarity and comfort in AI. And again, I think the messaging is that listen, AI is a force multiplier for quality. It’s not about replacing people, right? The other thing I would mention is leadership leading by example, right? This is another thing you want leaders to make sure that if they’re telling people to use AI that they too are actually using AI tool sets as well. I think that’s something that’s very important for driving forward adoption.

Absolutely. And that goes into our next question because you both talked about peer adoption over top down mandates. What actually works to bring skeptical long-tenured staff along and Richard we’ll start with you on this.

Yeah I mean I’ll maybe repeat an answer to the previous question to a certain extent which is like I think exposure is the biggest thing you know Ben Ben definitely hit the nail on the head that the landscape is shifting so quickly that you know six months ago things were just vastly different. I know I’ve experienced that in my own journey. So, getting skeptical. So, identifying who your skeptics are and oftentimes they’re not quiet, which is great, you know, that’s perfectly fine, you know, and then, you know, in one-on- ons, hey, what have you tried this? Have you tried that? You know, have you looked at the, you know, latest model from OpenAI? Right? Have you tried it to, you know, build the PowerPoint that you were working on for me? you know what what are the like exposure in very specific and targeted ways to help people realize what it can do for them and then you know that ought to spark other ideas get them thinking about other ways that that it can be used to improve efficiency one area that we see a lot of folks struggle with is tool fragmentation around AI right and if you look at some of the research there’s a signal there showing that a lot of employees feel burnt out from the number of AI tools that are being put in front of them. So when you think about AI in the context of productivity use cases, employee facing use cases, right? Wherever possible, rationalizing the number of tool sets that are exposed specifically to the employee I think is very important. Now on the back end from a a technology architecture perspective you may be making calls to multiple applications or systems to actually get the answer for the employee right but you want as much as possible I recognize there’s pragmatic constraints sometimes but as much as possible when you think about AI a single pane of glass a single chat interface that’s something that helps alleviate some of those adoption barriers and as I said this is very supported by the research that you’ll see that some employees report they are struggling basically with AI burnout, right? It seems like every other week a new distinct proprietary chat interface or AI agent is being pointed in their direction, right? So that’s something I think is important to think about from a technology and call it IT infrastructure perspective. Rationalize and consolidate your IT tools so that the user isn’t overburdened by every week. There’s a new tool that management is telling me that I have to use, right? Um, the other thing that I’ve seen work really well in practice is setting up a formal advocate or evangelist program. Basically, the idea here is that you’re finding someone on particular teams or in particular departments that is tech-savvy that knows how to use AI that’s fluent, familiar, and comfortable with it and having them serve almost like a tier zero resource for answering questions from their peers about how to effectively use AI tool sets. uh or someone that can promote how they are reducing the administrivia and annoying parts of their role by using AI tools. When folks see someone at a peer or nearer level that is effectively adopting and utilizing AI tool sets, they’re a lot more likely to understand the value proposition, overcome the resistance, start using those tools themselves versus it being a pure from the mountaintop down edict from leadership, but they’re not actually seeing anyone in their department talking about how they’re effectively using AI to drive forward better outcomes. So, those are two things that spring to mind. First, consolidate the tool set so that you mitigate the AI burnout, the AI overload piece. Second piece, setting up a formal advocate or evangelist program that can drive forward more bottom up organic adoption within departments. That’s something I’ve seen clients set up. It’s been successful for mitigating some of that resistance. Uh, as you said, particularly from long-tenured staff to AI adoption.

Okay, let’s talk about residents and bringing residents along the AI journey. We ran some research. Uh, earlier this year, PayIt research found that residents are more worried about government use of AI than the private sector. Why does that gap exist and what does it mean for how agencies roll this out?

Ben, we’ll start with you. I think that’s a really fascinating question. So for me it boils down first and foremost to the optionality of engagement with the organization right if you are a resident right I mean it’s some level sooner or later you’re going to have to interact with varying levels of government right be it u you know at a municipal or a city or a county level be it at a state or provincial level be it at a federal level right uh engagement with government is very difficult unless you’re living completely out in the woods somewhere totally off-grid, it’s very difficult to avoid engagement with the government at some level. If there’s a private sector organization that I deal with that I am weary about how they’re using my data or how they’re using AI, I can vote with my wallet, right? I can simply say I’m just not going to engage with that organization anymore, right? Or I may never start a relationship in a commercial sense with that organization to begin with. So I think the optionality is the one piece of it, right? You have to engage with government on some level. You have a choice of who you engage with when as a consumer I’m looking at various companies, various private sector offerings. The other piece where I think there’s trepidation around this beyond the optionality piece is potentially the consequences of how data how AI is used as well. Right? So there’s concern that AI could be used in a public sector context to drive discriminatory service differentials. Now interestingly enough in the private sector that’s done all the time based on for example account spend. Uh but obviously when we talk about a private sector environment that whole paradigm disappears. So I think there are concerns about AI being used to drive differential potentially discriminatory outcomes in terms of how government organizations and agencies engage with their residents. And again if you extrapolate this to the far end of the spectrum misuse of AI misuse of data again government carries the force along with it right. So in an absolute worst case scenario, there’s potential issues where could there be legal consequences for me as an individual in terms of how a government entity is using AI, using my data. So my viewpoint is that that optionality piece, the consequences piece, the force of law piece, those are why residents tend to have more concerns about government use of data versus the private sector where again I can vote with my wallet. If there’s an organization I don’t like how they’re using AI, I could just not become a customer or I could churn as a customer.

Sure. Richard, how should our state and local governments approach this concern? What should they do about it?
Yeah, I think there are two primary ways that that it should be approached. The first is the non-technical answer which is patience. So, you know, we have to we have to understand that just as our our own internal folks are grappling with how do they use AI to do their jobs, how do we incorporate it into the services that we deliver to residents. Everyone is grappling with this same uh we’ll call it upheaval for lack of a better better term in the sort of total sphere of human existence right this is a big moment of change and everybody is trying to grapple with it all at once and so that has to factor into this you know we talked about an ROI earlier like there are some like ROI isn’t just about dollars and cents like there is an aspect of say other vectors that have to be considered, right? An initiative may be successful from an investment perspective and fail because a lack of trust or a lack of buy-in from residents. So that has to be be accounted for here uh to a certain extent. So patience is probably the first thing and the second is I think things that government do does pretty well which is radical transparency. So you know governments are are relatively familiar with being transparent but this is maybe another level of that. So how are they how can we talk about it openly the things that we’re trying to do the efficiency that we’re trying to bring how are we using AI and agencies should be explicit about how it’s being used and you know okay in this scenario an AI informs a decision but doesn’t make it in this scenario it makes the decision but it’s reviewed like what are the ways that service is delivered and changed because of AI and just be completely transparent about it uh and be willing and open to the feedback that is received.
That’s great. And that kind of leads us into our next question here . You know there is a real difference between AI that helps residents find information and then AI that makes decisions about them. So how should that distinction shape what agencies deploy first and how should they talk about it publicly?

Ben, we’ll start with you. Yeah. So the information sequin is, you know, it doesn’t surprise me that that’s where a lot of folks gravitate to first, right? For example, in web experience workshops that I’ve led with our public sector clients, one of the things I hear all the time is that findability of information on the website is a major major friction point, right? And particularly as traditional websites evolved over time, finding the right services, finding the right information, you know, even if you had a halfdecent search appliance on your site, it was often highlighted both internally when I would talk to organizations as well as when they would talk to their residents or constituents, you know, it takes forever. It’s very painful to find information to find the services I’m looking for. That tops the list of pain points when you think about, for example, traditional web experience. So the information seek use case, I love that because if you can get the right answer to the right resident at the right time with minimal friction, that’s a self-evidently valuable use case for the resident for the constituent and that in my opinion is a good way to drive forward some of those early wins, right? Now the ability to execute successfully on that, going back to a point that Richard made in the start of our discussion is going to boil down to data quality, knowledge curation. So I want to be clear, it’s a high value use case. I think sometimes there’s this perception that it’s a little bit easier to accomplish than it actually is, right? So if you’re giving answers through a conversational interface to someone, where is that data coming from? Right? Are you pulling it from your external web CMS? are you pulling it from internal knowledge stores in order to build the trust around AI residents have to trust that the answer that I’m getting or the services that I’m being directed to that it’s actually the right answer right there’s a degree of full tolerance there right if it’s 99.9% accurate I may be forgiving the one in a thousand times it doesn’t give me the right answer but you really do need a high percentage hit rate of getting the right answer to that resident so reducing information seek through conversational interfaces through AI. I think that’s a huge potential value frontier and public sector when you look at this. The second one also promises to provide transformational value from an efficiency standpoint. Right? Uh I look at the permitting and licensing use case as an example. Right? If I can use AI to speed up cycle times on adjudications of issuing municipal permits, right? That’s a win for the resident because they’re getting an answer turn around more quickly and b it’s a win internally too because now I can increase my throughput of approving permits. Right. Now going back to something Richard said radical transparency. I think you need to be very very clear around when AI is being used to adjudicate a decision. And I suspect one of the interesting scenarios that will evolve out of this as well is when an AI adjudicates in favor of the resident and they do it quickly. That’s great. That’s a win for everyone, right? We did it more efficiently and now the resident is happier because they got a decision turned around a lot more quickly as well. The risk that we have here is when an AI adjudicates in a manner that is not congruent with what the rest is actually looking for, right? And so I think having a human escalation chain is still going to be of paramount importance, right? Uh someone that can either interpret or explain the decision made by the AI or can do a manual review where prompted, right? So I think that decision-making piece with AI again, there’s a lot of value there from an internal efficiencies perspective. If we can turn around answers faster to the resident, that’s awesome. Uh but be prepared for the fact that when AI adjudicates in a manner that the resident doesn’t like, you’re probably going to get a call or a chat or an angry email about that. And having the processes in place proactively to deal with that in terms of being able to explain the AI rationale or a human-based appeals process, I think that’s going to be something organizations are going to have to think about.

Yeah. I mean I I would add you know being explicit about the exception process and then uh you know when uh and how do we reverse the decision that is made and what are the impacts of that both in favor of the of the resident or constituent or in favor of the agency if there is such a value judgment to be made in any case but uh being explicit about how that happens and then I would add on you know you could think about AI’s like large language models as two kind of two vectors. The first one is model complexity or capability and the second is uh context input. So what knowledge is available to it? What context is informing that decision. And so having both of those be assets that are kept alongside the decision, right? That’s really the only way that you can get better and better and better performance out of an AI system is being able to let’s say interrogate where a decision went wrong and alter the inputs to get better answers the next time.

Absolutely. Let’s move on to a new topic and let’s start with governance here. Governance needs to happen before deployment. So for a leader uh starting staring at frameworks, where do they realistically start without it becoming a year-long policy project? And for this one,

Richard, we’ll start with you. Yeah, I mean risk is the answer for me, right? Governments has to be assoc governance has to be associated with some sort of risk assessment, right? So there are certain uh policy decisions or uh adjudications that will occur that just don’t matter that much, right? Not to say that they don’t matter, but they just don’t matter in comparison to something else, right? I don’t know that anyone would talk about uh we’ll say on the extreme end replacing an actual uh judge in a legal case with an LLM, right? So the risk is too high, right? And there’s so much other factors that go into it. So but does that mean that I need a significant governance process for uh internal use cases, right? Like question and answer for my staff or even what’s the risk side to as Ben was talking about earlier, you know, improving the ability for residents to find things on my website. What’s the governance framework that should be used there, right? And I think those are maybe some extreme examples but kind of highlight that we don’t need massive governance for some use cases and in some use cases we absolutely need it and the decision around that is risk and that has to be something that you decide upfront as your projects start to kick off.

My answer is highly converged with yours, Richard, right? Uh I see all the times where organizations overbuild governance for use cases where it’s like yeah the risk profile just isn’t there. So right sizing your governance to the you know what’s the risk to internal operations? What’s the risk to the constituent or providing a bad constituent or resident experience? Absolutely right. Like you have to rightsize this. And I see this in both directions, right? where either folks overbuild the governance or sometimes I’ve seen scenarios where they’re looking to move so quickly in terms of transformation that the governance takes a back seat their risk profile is large and as a result they’ve unfortunately been burned in terms of poor outcomes for residents or internal operational issues automations breakdown things of that nature so totally agree you’ve got to rightsize the governance when I look at the approach to governance I kind of look at it through a couple different lens Right? You’ve got your centralized governance vehicle. Typically, this is a steering committee. This is an advisory board. But it’s the top- down organizational institution that basically has purview or oversight terms of what you’re doing with AI for a particular process, domain or initiative area. The other thing that I tend to think about is, you know, what’s kind of almost that bottom bottom up governance, right? How are folks governing themselves? How are folks providing feedback on governance policies? And that’s an area where having a community of practice can be helpful. Right? So again in a particular particular domain you’ve got your centralized governance structure vehicle. Typically again from a composition lens you know that’s a half dozen to a dozen individuals. Uh but then you may also have a community of practice that can help provide some osmosis for best practices and can provide some almost bottom bottom up feedback around the governance as well. the other thing that plays into this is the technology side of it. Specifically looking at agent observability. This is something that’s tip of the tongue for a lot of our members these days, right? So do I have a platform that can investigate and provide oversight of agentic tools that I may be using in my ecosystem as well. Right? It’s almost that automated governance layer where you’ve got a platform watching the platforms that’s looking for off policy behavior of agents that you may have deployed in your ecosystem. So particularly with Agentic AI observability in the context of governing this ecosystem is something we’re seeing a lot of interest in these days as well.

That’s great. We’re gonna move along to talk about costs and vendors. So we’ll get started with, you know, just AI in general is often priced on usage, which can sometimes surprise agencies who are used to licenses. So, what questions should a leader be asking a vendor before they sign?

Richard, we’ll start with you. Yeah, this is a good this is a good question and and definitely top of mind. I think uh it’s top of mind because everybody is struggling with this basically in real time, right? we’re all kind of trying to figure this out this out, right? And it goes back to that ROI question like sometimes when you begin a project, you just don’t know. You don’t know if it’s going to be cost-effective because you don’t know what the token cost is going to be on a run rate basis, right? You know, we talk about uh this idea that you know, then if if I buy something for a dollar and I sell it to you for a dollar and a penny, uh, and you happen to be okay with buying it for a dollar and a penny, right? I’m going to make that trade as as often as I possibly can, right? The question is, we just don’t know uh, whether or not we’re making the right trade here, if they’re if they’re if the payoff is worth it. So understanding right like what are the pricing tiers, what are the limits that I can set in my uh my accounts and where do I find all of those controls and then also kind of understanding the ability to use different models like not every workload requires a frontier model. oftentimes you can get by with a much much much smaller model that is fractions of the cost of your frontier model. So being very conscious about how you are applying uh your workloads in an agentic manner and that should definitely be part of the training in terms of understanding which model to use as well, right?

100% 100%. Yep. Yeah. Just dovetailing off of what Richard said. I think that’s that’s something you have to look at. And one of the questions I would ask vendors these days is do you provide any flavor within your platform of model optimization? or see particularly a lot of the large enterprise software vendors going in that direction where based on the task at hand they will have a model that basically selects the appropriate LLM for the use case right rather than vanilla using one of the more expensive models so model optimization is something we’re seeing a lot of the big enterprise software vendors invest in position that as as a a value driver within their solutions I think when you zoom out and kind of you know more broadly look at what do I need to think about in terms of the cost piece right it’s the fundamentals right understand the vendor’s licensing approach understand how they do step costs what do you get in each tier of the product that you’re going with how do they handle overages right do I buy you know a single bulk of credits or again is it a real time over approach that they take that’s that’s the first piece of it the other piece of this is just understanding your own internal projected usage volumes I think this is where a lot of organizations are actually getting burned, right, is they budget for a certain amount of AI token usage and then based on their realworld workflows or usage of the platform, they radically exceed that.

Now, my hypothesis on this is that this is probably a self-resolving problem within the next year or two. As organizations get a better sense of what their usage volumes actually are, as they get a better sense of ebbs and flows of token usage throughout the year, for example, seasonality being something that pops to mind, folks are going to be able to better estimate what their spend on AI platforms is going to be. So my viewpoint is we’re kind of in a weird transitionary phase right now where software vendors are trying to figure out what their pricing model is. Their clients are trying to figure out what their usage patterns are. Uh but when you kind of look over the next couple years, I think both parties are going to have a better sense of how the vendors are going to be pricing this and b the consumption patterns that you’re actually going to see within organizations. Interestingly, I draw a little bit of a parallel here between the last kind of major cost or pricing transformation that we had and that was from traditional perpetual licensed applications on prem to software as a service. We saw a lot of the same initial questions, right? Like how many users realistically am I going to have to license on the platform? you saw a lot of interest not just from the IT or the line of business folks but also from CFOs from finance departments in terms of moving from a capex to an opex model and the first few years of that transition yeah it was a little bit rocky u I actually just joined Info-Tech as that transition was going on so I remember it well we got a lot of questions that were thematically similar to some of the questions that we’re now seeing with AI so I guess one message that I would I would take to the audience is, you know, this is something where there’s definitely some short-term turbulence around how vendors are pricing, how you’re consuming. I think with experience, this is a little bit of a self-resolving uh issue where I forecast out two, three years, I think most folks will have a much better sense of a what the cost structure is going to be on the vendor side, b usage consumption patterns internally within organizations, and again, how those might ever flow throughout the course of the year.

As a final question for this webinar, if a leader is watching today uh could do just one thing tomorrow morning to move towards readiness, what should it be?

Richard, we’ll start with you. Yeah, I mean I think probably the most important thing that a leader can do is making sure that they have appropriate training for their staff. You know, uh everything from, you know, how does a how to properly use a context window and manage it, right? which by the way that will help you reduce your cost if people are and they’ll get better they’ll get better results from from AI’s agents you know with that are managing their context window uh back to what I said a little bit earlier selecting the appropriate model for the task I think uh training is is where the the big bang for your buck is going to be the bonus of that is you’ll get exposure along the way especially if you choose good hands-on training for folks they’ll get exposure and you’ll get better adopt adoption and that adoption will be more effective.

Ben, what would you add? Be incredibly clear, targeted and prescriptive with the outcomes that you want to achieve from AI. I’ve seen too many organizations particularly in the public sector where they’re kind of approaching this from a laissez-faire AI for AI sake viewpoint. Now you need to understand what are the actual resident outcomes that we want to improve through using AI. And again that could be things like information seeking simplifying payments. It could be faster adjudication of key decisions or it could be internal efficiency. How exactly am I using AI to make my internal staff more efficient to alleviate the administrivia. But knowing what are the outcomes that you want to achieve? what are the business processes that you then want to point AI tool sets towards that sets you up for success because then you can develop a forward-looking road map of how you’re going to approach that in terms of investments in certainly the technology but also process change data quality governance so link it back to you take a look at dust off the organization strategy what are what’s our mandate what are we looking to achieve and then link AI on a prioritized basis towards targeting key outcomes, key use cases. It’s not just AI for AI sake to really be AI ready. You have to have a clear foundation in place in terms of what you’re looking to achieve first.

Absolutely. Making sure that you know what your objectives are and which way you’re headed. So, while you guys were talking, we did get some questions that came in from the audience. And you know, I’ll ask the question. Whoever wants to jump in here can go ahead. But the first question we got is how should we decide whether to partner with a vendor or maybe develop our own capability in house?

Everybody looks at that question. It’s really a recognition. What’s your IT bench strength at the end of the day, right? So, have you gone through similar transformations or stood up new technologies like AI where you’ve got those lessons learned, you know, where you’ve got the roadmap developed for what you want to do. If you’ve got the internal bench strength within it, you may go your own, right? I’m seeing a lot of our clients though looking to external partners to assist them both with the strategy development and the implementation piece right because I think there’s a recognition you know Richard going back to what you said about the risk profile right it’s material in many cases with AI so being able to lean on an external partner to assist with that both in terms of developing the approach and then executing on it doing the implementation you know that’s something again where a lot of our clients are gravitating towards that approach. They just don’t have the internal bench strength to execute fully on their AI road map internally. Vendors are also something I would mention there, like if you’ve got software vendors within your application portfolio that are at the forefront of AI, lean a bit on them too, right? It’s not just professional services I’m talking about here. So ask your vendors what are they seeing in terms of successful AI adoption. What are the use cases within their products that they’ve seen their clients really excel at. Uh vendors are also a fantastic wealth of information on advising around how to think about AI transformation particularly in the areas that their applications support.

But maybe uh add add on to what Ben said or maybe a different way of thinking about that same his same answer but you know uh this is not too dissimilar from build versus buy right that this traditional that we’ve talked about along for a long time right one of the things that I always think about is is this a core competency of my organization and that I want to be really in tune with the way that it is delivered and if so I should look inhouse Now, what if to Ben’s point, I don’t have those competencies and I have some sort of time pressure feeling, right? So, I don’t have the opportunity to, uh, develop my folks to get these capabilities, then yes, I’m going to look at maybe bringing in somebody to like jumpstart the competency learning process within my team. And that’s kind of how I would think about this question.

Great. Another one that just came through is how does the pace of change alter readiness? Is it better to wait or start now? Because you know things are changing super quickly.

This is the catch 22 in my opinion, right? Because I think you can’t delay investments in AI, right? You’re going to find yourself at a disadvantage if you move too slowly on this. There is particularly in public sector though there’s also a material risk of moving too quickly and the risk is basically what I mentioned around AI for AI’s sake right folks are charging headlong into we’re just going to buy a tool and we’ll stand it up they haven’t really thought through the use cases the business processes they want to support they haven’t thought through the readiness elements we’ve been talking about throughout this discussion in terms of governance change management data quality knowledge curation right so I I learned a little bit particular particularly for public sector I move a little bit to the middle right you want to identify the easy win high-value use cases that you can target call it in the next 3 to four months and then start to identify what are some of the longer range more transformative initiatives that we can tackle in the next call it year and a half right the technology does move very quickly right so those use cases may evolve they may change in terms of how vendors support them and fairly short period of time.

But I would start again with the high-value quick win stuff. As you build up the foundational competencies across the pieces that enable readiness, then I would tackle the bigger transformation initiatives. The risk again particularly in public sector trying to move too fast is that you target a transformation initiative that doesn’t clearly align with your organizational mandate that doesn’t see the adoption and now you’re looking potentially at a failed multi-million dollar implementation of a very complex AI system that you know was AI for AI’s sake and you didn’t build the prerequisites to actually succeeding with that deployment as well.

Yeah, I would look at this kind of in a couple of ways. So I would agree with Ben. Don’t wait but be cognizant of the areas where you anticipate there to be change. Right? So we don’t know which uh frontier model provider is going to win. Nobody knows. Nobody knows who’s going to have the best one. So uh accommodate for the fact or acknowledge the fact that the best model today is not going to be the best model tomorrow. And that’s likely going to change multiple times as we go through this cycle. So look for other areas where you could anticipate change and kind of don’t lock yourself in uh to something that might get the rug pulled out from under your feet so to speak. The other kind of aspect of this is maybe one of, you know, not to beat on a drum, but like for me, I want to make sure that all of my folks, all of my team are exposed to this technology for the sake of their own careers, right? And their own like to be the betterment of themselves, right? So, I have a duty to make sure that we’re trying to adopt so that no matter what happens, I have done the best that I can for them. And oh, by the way, it really does help me provide better service to uh my consumers. So, I think there’s no reason to wait. Just be aware of the things that might change on you and uh try to accommodate that for them as best you can.

Great. And that’s all the time we have for today’s questions, but if we didn’t get to your question, we’ll follow up with you um via email. Richard and Ben, thank you so much for all of your insights today. I hope the audience took a lot from that. And if you did like this webinar, I recommend checking out PayIt GovLab, which is our virtual summit happening September 2nd. We’ll send everyone the link after the webinar so you can register for that. It’s completely free and we have six sessions with AI being one of them as well. Thank you so much everyone for joining and have a great day. Thanks folks. Thank you.