AI and Financial Regulation – Wharton Future of Finance Series

Okay, welcome everyone. I am Itai Goldstein, professor of finance at the Walton School, currently the chair of the finance department. This is the future of finance podcast, uh a series where we explore uh the new advances in the world of finance and how the future of finance is shaping up. This is the third season of the future of finance podcast and in the third season we are diving into the very important topic of AI and finance. How AI is changing the world of finance. And today we are examining the very important angle of the future of finance and AI in finance which is how AI is going to affect financial stability and how we should regulate uh this technology when we are

thinking about its introduction into uh financial markets uh and financial institutions. This conversation pairs very nicely with the launch of a new initiative that we have here in the Warton school. This is the Wharton future of finance initiative. I am one of the academic directors of this new initiative alongside my colleague Pat Harker who is the former dean of uh Wharton and also the former president of the Philadelphia uh Federal Reserve. Uh the Warton Futural Finance Initiative studies the financial system and its support of the real economy and particularly focusing on two forces that shape it. the stability that sustains it

and the innovation that transforms it. For more information about the Won future of finance initiative, I invite you to visit our website using the link in this episode's uh description. And speaking of financial stability and how it relates to innovation, I'm very pleased to welcome our guest for today's episode who is the ideal guest to speak about these uh important uh topics uh the interaction of innovation and stability. Uh this is uh Sarah Breeden who is the deputy governor for financial stability at the bank of England. Sarah leads the Bank of England's work on financial stability, the supervision of financial market infrastructure,

international issues, payment innovation, and uh fintech. Sarah and I spoke together recently in a panel in the European Central Bank Forum uh in Cintra. Uh this was uh this past uh June and we discussed the many moving parts of regulating AI in finance. uh and this will be the topic uh of this episode today and I can't wait to dive into that. Uh so uh welcome to the podcast Sarah. >> Uh it's an absolute pleasure to be here. It's hard to think of a more topical set of issues to be talking about it. >> Great. So let's uh dive uh right in. Um you know Sarah, so financial regulators have adapted to many technological changes over the years. uh and one could

say that AI is just the latest version of that. Uh so maybe you can take us a bit into this uh topic and tell us is AI uh different and if it's different then why? So we've been looking at AI with the financial services firms here in London for five, six, seven years, but there hadn't been anything that was flashing particularly red about uh how it was being used by uh the firms. The sorts of use cases fell into two categories. They're either cyber or fraud where you needed the latest technologies, the best models to be able to make sure you were doing your bit against the bad guys or they were being used for relatively low risk um uh tasks, you know, helping people do

research, those uh sorts of things. So for a long time we've been working with firms understanding how they've been using it and happy that our regulatory regime was doing a decent job of managing uh the risks. uh but suffice it to say I'm not sure that as we look ahead it we have got as much to be comfortable about with the uh arrival of a of a gentic um AI. >> So so you you say that agentic AI is really the thing about AI that worries you the most. Can you say a little more about that? What it is about agentic AI that we should uh worry about? So look, Aentic AI is different, right? We're moving beyond systems that help me make decisions towards systems that are

increasingly taking actions for themselves. It's not writing text, summarizing documents, answering questions, coming up with PowerPoint uh presentations. That's not helping me do my job. Genens AI is given an objective and it works out for itself the sequence of steps needed to achieve it whether it's making financial transactions rebalancing portfolios uh doing cyber uh defenses and so as we're moving towards a world in which AI agents are acting autonomously in the financial system. I think we have to ask ourselves is our regulatory framework set up for it. The the nature of the challenge has changed, right?

We're not regulating institutions where humans make decisions. We're supervising systems that are operating at speed and at scale and levels of complexity that uh it's hard for a human to oversee fully in uh real time. And so for us, we're asking ourselves the question, if the financial system is operating autonomously, at scale, at speed, are we prepared uh for that? Is our is our framework doing the best possible job it can to make sure that what's happening in the system is safe? So one of the things that uh people said uh can protect us potentially is to keep a human in the loop. Uh so that it's not

going to be fully agentic, fully autonomous, but there will always be some human uh in the loop that knows what's going on and can uh inspect and and monitor and potentially uh prevent things from going in a different uh direction. Um but but I think uh recently and and you spoke about it as as well uh it is becoming more and more difficult to keep a human in the loop. So do do you think we will not be able to do that and if we can't then uh how are we going to uh address that? So look, I I think kind of it's really easy to present this as a choice between humans in control and machines in control. And the reality is likely to be a much more complex partner uh ship

between uh the the two. But if what we're talking about is kind of systems that are making thousands of decisions, millions of decisions at machine speed, it is difficult to see how a human can meaningfully review every individual uh action. Now in many ways we face similar challenges right now. Human supervisors don't review every transaction processed by a bank, right? They oversee the systems, the controls and the governance that sits around those transactions. And so I think it's on us to think about what is the analogous approach in the context of agentic AI. We don't need to approve every individual decision that an agentic AI might be making, but we do

need to create a structure in which humans are accountable for actions and that accountability is meaningful because they understand the risks that are being generated and there's an ability to uh intervene. We need to test, we need to monitor and we need to set guard rails around the outcomes of um AI that there is the issue of explainability, right? Kind of financial regulation has typically been built about models being explainable in the complex models we're talking about here. Uh that's just not possible. So we can't we can't be focused on the model. Like I say, we have to be thinking about the inputs and the outputs and thinking

about what guard rails do we need to put around those. >> So, how far along would you say are regulators in understanding these issues and uh introducing those guardrails into the financial system? I I know I mean it's you know one thing about AI is that it's fairly recent at least in uh the the current versions that we are seeing um and and it's also moving very fast uh so can regulators uh really uh keep track and and be ahead of of the curve how far along are you >> what we are doing is working with industry uh we have an AI consortium uh for example in in the UK. uh we and our

fellow regulator uh uh markets regulator at the FCA uh are involved in it and we're talking to financial firms talking to the hyperscalers talking to the model developers talking to academics like yourselves to understand try and get our arms around the uses of uh of AI and what that means for regulators and firms in uh adopting them. I think it's fair to say that firms have been pretty cautious so far in their adoption. Um uh and so I don't feel like we're very far behind just yet. But what I am conscious of and what I think really saw in the spring with the new models that were very capable of identifying cyber vulnerabilities is that these things move quickly and uh you need to be

nimble. I don't think we're behind, but I'm not sure that we've got terribly much time ahead of us. Look, AI is a good thing, right? uh it should drive productivity improvements. It should enable me to have more capacity to do my job well and and so it's on us as as regulators not to say no but to support its responsible adoption. Uh that's obviously the case in terms of cyber and fraud. If the bad guys have got these uh tools, you absolutely want regulated financial institutions to be able to use them. Um but I do think we need to think about where might this where might this bring financial stability risks and how do we make sure that uh models are being

used responsibly. Yeah, I I think that's an important point to keep in mind that at the end of the day, AI is a good thing and uh it can help uh many functions in the economy, many functions in the financial system, but of course we also uh need to be uh vigilant and make sure that it doesn't get out of hand and uh create uh financial fragility. So, so one of the things that I've been interested in in my own uh research is the interaction uh among different uh AI agents. And uh you know we've been running experiments uh kind of showing how uh when you have uh different autonomous AI agents they learn uh to collude uh potentially uh

preventing the market from being competitive uh or even uh starting to create their own uh price path of uh you know pump and dump manipulation where the price goes up then it goes down and they manage to make profits on on the way. So this all sounds uh of course pretty uh science fiction but in the lab when you uh let these uh agents uh start trading against each other they do find ways uh to uh coordinate and create this kind of collusion and manipulation. Uh now when you think about how collusion and manipulation uh were treated by regulators in the past uh I think the dominant thinking was that with humans you know you're going to look for signs

of explicit communication or explicit intent before you're going to pursue any uh anti-olusion or anti-manipulation uh enforcement. Uh but with uh those machines uh that's probably not the way it works. So do do you think we need to worry about that and and maybe think about collusion and manipulation uh differently going forward when we are dealing with those uh autonomous AI agents. >> So it's a great point and I think your research has made us ask some really quite deep questions about how markets uh work. Exactly. As you say, the research that you have have done suggests that the manipulation or collusion that you normally think about

people coordinating to deliver conversations, agreements, intent that is is is just not appropriate or or not not likely to capture everything that might happen in the context of AI. we get similar outcomes without explicit uh coordination. Uh all of the AI systems are there trying to maximize returns. They're learning from market outcomes given enough time. They learn that certain forms of behavior are collectively uh profitable. Nobody told them to collude. They didn't intend to collude. However, the market result may nevertheless look remarkably uh like uh collusion. Now, uh I don't want to over dram

uh dramatic. its early days and all that, but it does to me raise some pretty important questions as to whether the elements of our existing legal and regulatory frameworks that place a focus on intentional behavior are appropriately uh uh calibrated because what matters from a market stability perspective is not your intent. it's what the outcome uh was regardless of whether that was something uh you set out uh to achieve. So I think we need to pay greater attention to behaviors of of this uh type behaviors that arise from interactions between agents not just uh the actions of any single uh participant. Um the AI consortium that I mentioned earlier, they had this lovely

phrase, we need to shift from thinking about whether any individual user of a model is managing it well to thinking about how can a market observe and control the collection of behaviors in uh a a market. So we need to we need to flip our frame and the sorts of experiments it that you've been doing are the sorts of experiments that we're trying to do now. And we're we're thinking not just about kind of how do these behaviors arrive, but what kind of market structures are they more likely uh in? How might one be able to design ways of intervening to uh minimize the impact of uh of such um uh markets and and uh the point that you raised with me earlier I don't think we have to do this

by tomorrow but we need to get our skates on if we're properly to understand uh these risks to make sure that by the time this sort of a genetic trading is common in uh financial markets. We're ready with uh an financial stability response, >> right? And and you mentioned uh something like a kill switch uh that uh we can put on these AI agents such that if something uh goes out of whack, we can just stop it. Um so how do you envision envision that working? Is is that going to be a requirement from each uh particular firm or is that going to be at the level of the regulator to kind of put a stop to all trading? H how can

we implement that? >> So we've always built in mechanisms to help limit the consequences of stress in in financial markets. We've got circuit breakers, trading holds, resolution frameworks for failing institutions, operational resilience arrangements. All of those are examples of trying to contain the impact of a stress once uh uh it has uh arisen. Suppose multiple agents are responding to the same signal in similar ways. supposing we get powerful uh feedback loops. To be clear, we have those in markets now, but they might just happen at greater scale or speed in uh an agentic uh world. Uh, I think what we need to think about is how can we use AI to identify where those

sorts of issues are arising and to think through what might AI be able to do to help us better manage those. I talked earlier about how in a cyber or a fraud context, regulated financial institutions use AI to ensure that those um uh that bad guys aren't able to um uh do bad things. In the same way, maybe in financial markets, I need agent Sarah Breeden on uh on the pitch thinking about how we can use AI to stop financial market feedback loops uh happening in practice. Now, I haven't got an answer to what those uh what that kill switch or circuit breaker might look like right now. That's something that we're hoping to test in uh the

experiments uh that we're we're doing. But the idea of having governance mechanisms uh arrangements to pause things when things uh financial markets are stressed. uh that strikes me as a a very sensible thing for H to explore even if I don't yet quite know exactly how uh it would operate in practice. >> Yeah. So so you mentioned using AI to regulate AI and and I guess that that brings us back to something we talked about in the beginning keeping a human uh in the loop. uh that there is a bit of a worry here, right? Once you start using AI to regulate AI and you go further down the the spiral of just AI

all around uh and minimizing the role of of humans. Uh so do you worry about that? >> I always think that uh you need to have uh how do the uh uh the AI labs call it domain expertise as well as uh model uh capability. We as central banks are information intensive organizations. We spend a huge amount of time trying to understand what's happening in the economy in uh financial markets, what the risk landscape looks like. I'm pretty sure that AI is going to be able to help me do my job uh better. I'm not sure it's going to be able to do it all on uh its own kind of it takes a huge chunk of

judgment, doesn't it? To to process and properly understand uh what's going on in the economy and uh and the financial system. So, I do think it's important that we use AI. we have the referee on the pitch but at the same time at least at least for a while I think uh we've still got jobs to do. Another aspect uh that we all uh think about and and talk about uh with regard to AI and financial stability is uh the financing of AI and the possibility that right now we have a bubble in the financial market because of all the investment that went into AI. Um do do you have a view about that? So look, this is um one of the most uh important issues that we've been looking

at uh recently. The degree of investment that is required to build the AI infrastructures that we need so that it can be used as the next general purpose technology to drive productivity and uh and growth. The pace of investment required is historically unprecedented. So the scale might be below the railways but the pace at which it is being uh pursued is historically unprecedented. And what it means of course is that a lot of institutions are raising equity capital and debt capital in order to be able to fund that uh in investment. What we have been trying to emphasize is people should think about what are the what is the risk that they are um potentially

exposed to as they undertake uh these investments. Historically, the hyperscalers have been cash flow positive, right? But kind of there's some pretty clear uh graphs that show all of those cash flows are being pointed to uh infrastructure investment. There's a chunk of debt being pointed to infrastructure investment as well. And the path to monetization uh to repaying that debt is as yet uh unclear. So look, is it a bubble? I don't know. But what I am sure of is that there's a chunk of investment uh that is being made. The payoff to that may take longer. It may acrue to different uh players. And so as this

investment takes place, have an eye to uh uh downside risk as well as upside risk. >> Okay. So we're coming to the end of the episode here. Uh so let me take you maybe 10 years forward. uh trying to predict what the market is going to look like in 10 years and um what would you think uh is a successful AI regulation at this point? F first do you think the market is going to look completely different in 10 years and second how uh would uh we then judge whether we've done a good job regulating the market and maintaining financial stability uh with all this innovation that is going on. So look kind of I I mentioned earlier the idea of tech surprise and

how much I have been surprised by the rate at which capability in AI has progressed even over the last 12 18 months. So I feel very nervous about projecting 10 years uh ahead. I feel more comfortable in trying to articulate what success looks like. Uh for me success is not measured by kind of how much AI we've regulated but it's measured by whether we've managed to adopt it across society in a way that captures for the benefits as we were talking about just poor productivity and growth while maintaining trust and stability. So if we in 10 years time have seen AI contribute to stronger productivity growth, better financial services, greater economic uh

opportunity and we haven't crushed uh the system on uh the way that would be my definition of uh of success. I think that's really easy to say kind of uh and I think we'll have to be nimble and agile in how we get there. We've been really focused to date on um cyber. Uh we've also been focused on financial markets as AI is adopted uh there. I think we've got an awful lot of work ahead of us and in that respect the community of Wharton with it research with its students uh with people leaving Wharton and coming into the world of central banking and and regulation. I'm sure there's a lot of interesting opportunities uh for us to work on together. >> Yeah, absolutely. Uh thank you very

much. Uh so I guess with this uh somewhat uh optimistic uh but but still realizing that we have a lot to do uh note we can uh close this episode. So thank you very much uh Sarah for uh joining us today to talk about AI financial stability and financial regulation. This was a great uh conversation. Uh thank you everyone for joining us. This is the third season of Warton future of finance podcast uh where we dive in to the very important uh topic uh around the innovation of AI and how it is transforming financial markets. I am Itai Goldstein here at the Won School. Thank you.