Free corn being so yeah, perfect you can you can start audacity okay? I did it and I'm starting since yesterday perfect so we are live Gianluca bio Benvenuto su learning statistics.

Thank you. Thank you very much, Alex. Thank you for inviting me. Very excited to be here.

Yeah, thanks so much for taking the time. And I think that you're one of the first Italians to come on the show so that's really good. Yeah. Or do you beasts releases like it's yeah, that's that's

I'm very happy to hear that I'm very happy to be to be sort of supporting my country.

Yeah, and in doing so to podcast patrons who are truly recommended me your names and I'm gonna find the name I always forget but like Yeah, somebody in the in the Slack channel was like, oh, you should check out what, what Gianluca bio is doing. It looks super interesting and that be awesome to have him on the short sale? Oh, no. Yeah. Yeah, that's order which thanks so or for your recommendation. Awesome. Well, let's, let's start with my favorite first question. What's your origin story? Gianluca? How did you come to the stats and data world in how serious of a path was it?

Right so um, I think it was pretty much by chance. When I finished the equivalent of high school in Italy, I knew that I wanted to continue to study and I probably would have gone in do some kind of economic space degree. But I wasn't quite it wasn't like a passion or a calling of life. You know that you always wanted to do that. And I started kind of in Italy, you have a system where you can choose your high school and I went to the more kind of scientific one so there was maths and physics and science, that kind of stuff. And then just before deciding formally what kind of program I wanted to apply I got a leaflet from the factbook It wasn't even a faculty it was a degree program. In statistics within the Faculty of Economics initially. And it sounded really cool. I mean, I had no real exposure to statistics other than, you know, recording things and thinking in terms of how long will it take me to, you know, queue up the post office or something and so because I didn't really have anything planned in my life, I thought, well, well, that sounds cool. I'll just go and do statistics. And I've been very, very lucky actually, because I've enjoyed very much my undergraduate degree. It was really good because in Florence at the time, it was a very small degree. I remember the first year it was only 30 students overall. So very, very small cohort, which meant it felt like a continuation of high school, you know, you make very good friendships and it's easy to all get together and revise and do that kind of an experience, which was really nice. And I managed to make a living out of it, so I guess I didn't call myself very lucky.

Hey, that's super cool. I love that. Yeah, and like I mean, I love the the elements of random chance always. Oh, yeah, yeah, people's paths. It's really fun. And two, actually, can you can you tell us what you do nowadays to how would you define the work you're doing and also the topics that you're particularly interested in?

Yes, so um, I most of my my research in the past several years, has been related to Bayesian modeling, particularly as applied to decision problems in healthcare. So the classic problem that I tend to work in is when you have maybe a new drug or a new healthcare intervention, and in many systems like in the UK, many European countries, you the state is the provider of health care, and they decide whether they should pay reimburse a certain healthcare intervention. So this is increasingly well now it's very much established in many jurisdictions, certainly in the UK. The Netherlands, Germany, Italy, Spain, many, many other countries. And so this is all based on formal statistical modeling. And a lot of my research has been throughout the years devoted to a mixture of applications. So sometimes we work on a specific case study where we have a new drugs or we collaborate with the UK regulator mostly, which is called nice and they're responsible, exactly for this kind of thing. So they get the reimbursement dossier from companies, they have to evaluate it and then decide whether it's worth or not paying for that particular intervention. And then they recommend that the state the, the Department of Health in the UK, some of the times, the projects that we've been doing are more methodological. So you know, you have the problems because most of the data are in a certain particular way. You're because you see the decision making process always going in a certain direction. And so you develop new methods to to deal with these questions, essentially. And we've been working for a few years for quite some time now, on this combination of things, which is very interesting, I think because I, since my undergraduate, I think I've kind of been fully committed to the Bayesian sort of approach in stats I think I naturally think as a Bayesian when I do even the little analysis or you know, preliminary bits and bobs. I tend to just do a Bayesian model that comes natural to me. And health technology assessment, which is how sometimes we call this area of analysis, which is very much related to statistical modeling is one of those areas where historically you've had a very strong Bayesian component. So you know, I don't think this happens anymore. But for example, when I was studying, there was still some kind of remaining lingering kind of fraction between whether you do things in frequencies or in a Bayesian way health economic evaluation, by its nature because it's a relatively recent discipline. There's always been the idea of some of these models are naturally Bayesian, so let's do them Bayesian. So you don't have to really fight and convince people that doing a Bayesian model is the right way to do it. Which I really like because, you know, I tend to do that anyway. So that's good.

Yeah, I love that. And I love the like a week or two small theories, a small sentence at the very end of their website that I'm not going to spoil but I loved it. It was it was quite funny so people will have to go there to see what I'm talking about. Talks about statisticians in the world in Bayesian statistics, so that that made me laugh.

I can say more if you want. I don't want to spoil it, obviously. But I said something like that, and one of the big health economic conferences where most of the audience were economists. So of course I think you annoy some people by just saying things like that, but I kind of believe it.

Yeah, I'm pretty sure you were like, yeah, like you had to walk to do a walk of shame afterwards in the in the hallway. Oh,

no, no. No, very proudly. Yeah. Thinking on the chin.

Perfect. I like to see some videos. Actually, do you remember exactly how you how you first got introduced to Bayesian methods and today also, like, I'm wondering how frequently you use them.

So I think it was my department in Florence when I was doing my undergrad, but my department I say the Department of Statistics in Florence, where I had my undergrad had a very strong Bayesian component. So there were quite a few courses modules in Bayesian modeling. And I remember at some point I, because the cohort was so small, you had quite some freedom in how you would choose your modules and the kind of the syllabus of your of your degree. And so I took one advanced course, which was Bayesian modeling, where essentially I was the only undergraduate and everybody else was either a PhD student or a master's level. And I really enjoyed it. I think it was the first step in throughout my my career throughout the whole time that I've been working with statistics where I started to understand what I didn't really understand before, you know, because you get exposure to the standard, the normal statistics, and I think you do have questions at the back of your head, like, you know, but this isn't exactly what I what I'm interested in. I would like to know if if this parameter does even make sense, what is what is the probability about that parameter rather than some convoluted statement, but I haven't fully registered that. And then when I actually was exposed to, to the Bayesian machinery, I thought, Well, that makes a lot more sense. That is exactly how I think now I think some people still are trying to see this as a kind of a religious battle and they try to convince everybody else that they should be Bayesian. I do as a joke. But in the end, it just convinced me it's, I think it's how I I reasoned about stuff and and so ever since I have been very convinced that this is how I should start anyway. I mean, I joke and I say that everybody should do start in a Bayesian sense, but at least I should do it that way.

Yeah, I see. And, well, actually, I'm curious. We, I mean, you touched on that already a bit. But how common are patients that's in your in your field, actually,

in health economics evaluation quite a lot. We've been lucky, I think, because like I said, this is fairly new. It's called Health Economics sometimes because economists in the 70s started working on this idea of checking whether something works and whether it is value for money, we should pay for it. Because by more or less at that time, people started realizing that you know, you don't have infinity dollars to spend on health care. There are limited resources and so you have to prioritize and make sure that you spend it wisely, to benefit most people in your population. And when everything happened in the UK started to happen, particularly in the UK, you had people like David Spiegelhalter, Tony O'Hagan Claxton, who are very much super strong Bayesian statisticians, and they were there at the forefront. Establishing the discipline. So, like I said, I think it's a bit easier. You have to, I don't think you have to find anywhere in any area of statistics or applications right now, but some places is a bit more difficult. Like you know, if you're doing clinical trials, for example, then you have to justify strongly why you want to go Bayesian rather than the standard frequencies P value sort of analysis. In HDA, you know, technology assessment, you don't so much. You can have a very good argument for doing your Bayesian model because, you know, that's established, it's a bit more established and also because it's embedded in a decision problem, and therefore, it comes more naturally, perhaps, to use that kind of approach.

Yeah, yeah, it was. I mean, when I prepared the episode into the kind of models you're working on, I was like, Yeah, that's really good. Even though that's economics and I know economics are not very Bayesian, usually. I was, I was like, Yeah, I mean, that makes total sense to me that you're using Bayesian models here B's like, and we're gonna talk a bit more about that, like in the episode exactly what you're doing that that makes total sense to use Bayesian stats here, but he's as open saying that Bucha is just like, it's not in the end. It's not a question of religion. It's just like using the best tool for the premier have at hand. Yeah. And here in those contexts, that does make a lot of sense because like you like you have prior knowledge, integrating it into models is super important and also the data can be noisy and not that not that big. So yeah, that often makes a lot of sense. And especially if you're focused on causal inference, in the causal reasoning in the Bayesian framework and model, it's just like there from the get go. It's not something you have to add. So yes, yes.

I think I'm lucky enough because I managed to convince enough people that I know what I'm doing, even if sometimes you don't, as you don't and so, you know, when I just say yeah, no, we need to do a Bayesian model. They kind of believe me, so that's fine. I get away with it.

Yeah, yeah, I guess once you get once you get to a level of credibility that people have seen you already use that weird stuff. That's called Patient stats, and that they are not used to because well, that's not digital art in university. But they've seen you use it in networks. And that's actually more intuitive in the interpretation of the results. Well,

I think doesn't does 70 an element that I was a bit kidding, you know, but I think will also help is one thing that we've been trying to do in my group is to make sure that there's a clear way of communicating what it is that you're doing. And sometimes, you know, if you don't have that element, Bayesian modeling sounds well, statistics in general is a bit like magic. Some people don't understand what's happening, because it is complicated. But sometimes we we hide even more than we should, and therefore there's a lack of trust, because it's not very clear and transparent. Whereas I think a Bayesian model has the potential to be extremely transparent because you know, I can hide everything in my prior but then you can call me out and just say, you're an idiot, you don't know what you're doing. Show me what you're doing in your private and I'll tell you if I believe it or not. So eventually, everything is out there and should be anyway very transparent, which is a good thing.

Yeah, exactly. I mean, that's, that's well of priors, right. It's like it takes the test that was under the rug and it put seats in front of everybody. Yeah, and I'm a bit surprised, though, to hear you say that. It's still not very popular in the clinical trial runs and stuff like that. Because to me, that would be like, exactly domain where it would be super interesting to use patient sets, because most of the time you have limited data, and also a lot of prior knowledge, especially if you work with extremely specialized doctors.

I think things are changing and they're changing for the better. Like for example, there's a lot of work there's a huge amount of research and applied work on the whole business of adaptive trials where you know, you have maybe limited information to start with and then you want your trial design to change continuously to adapt to the signal that's coming from the data. And then you have the possibility of making a decision on whether the drug should go on or be killed earlier on. So that's, that's very helpful and people are starting to realize that and it's becoming more of state of the art. There are many, many cases many situations particularly down the line, you know, the clinical development is a very long process. And eventually you get to a stage where you have to demonstrate that the drug actually has some effectiveness on a relatively large population. At that stage, there's still some resistance and lots of agencies like the FDA, the European regulators, they tend to insist that there's a recipe and I think some of it perhaps is due to the fact that as statisticians maybe we have, to some extent, back, you know, to the to the great work of Fischer and people like him, maybe to the non statisticians. The message was that we provide a set of recipes, you have this problem, you do this test, you have these other problems, do an ANOVA. And maybe people think exactly in those terms, which I think is not very helpful because no problem is, you know, unique and you have to think about this specific issue that you have and and then you just set it up accordingly. So there are cases where it is still harder to go fully Bayesian on on particular designs and studies. But having said that, for example, the Pfizer COVID vaccine was was marketed on the back of a Bayesian analysis. So you know, it's happening it's changing.

Yeah, yeah, I was like all the COVID think was very interesting, especially in the UK, a lot of the modeling was was patient based, and I know a lot of people from from Stan helped for the modeling into gell Mann also was was consulted a lot. So I was like, I was super happy to hear that. And also now we're in neiger. PMC team, we, we worked on some modeling more, more for the US and so like, yeah, I'm really happy, eco tone and that was much more of a practical issue. It was like oh, well, actually, that stuff is super useful here. Because we don't have a lot of data. It's like, kind of the first time we have such a huge pandemic. At that level. It's not a relatable experience all the time, you know, so it's like priors are useful here. So, yes, in generation basically thinking about the data generating model here was super powerful. Yeah. Okay, super. And actually, I mean, I love those topics. So already, I have two episodes on that. So Episode 45, which was with Frank Harrell. Your stats and clinical trial design, so if any listeners when I refer back to that, that's that's super useful and Frankie's just like an amazing guest like I could have made a three hour episode with him. And then a bit more general discussion I had with actually David Spiegelhalter sweeps up 50 If people want to refer to that and yeah, about like the notion of risk and uncertainty and probability, especially around coffee modeling and also communication to the to the general public, which is something also I find super interesting. Because yeah, you have like those. Like I think that the you have the kind of too extreme way of communicating would be like, you know, being too, like communicating too much on the certainty side. And then being like seeing statistics as a black box and toolbox as you were saying, like, Oh, do you have that do that test if you have that do the test and then like with time if people see that as a black box that will forget their models and so they will forget they actually uncertain and they can fail and they fail a lot and then you have the other extreme, which is like once you've highlighted that to the public. Well, you know, models are not perfect. They are a human product. So like you always have to take everything with a grain of salt and keep the uncertainty in mind. Then you have a lot of reactions, which are like Oh yeah, well then statistics is no use because you can fudge the numbers super easy. You know, it's like if you want to, if you want your model to tell me a percentage, you know how to do that. And so I'm not going to trust your model. So I'm not going to trust any, anything that you're telling me which is like fear based and I'll just trust the people around me and whether they have problems with the vaccine or not.

I think it's a historical accident that we as statisticians are responsible for, because again, I think that we, for whatever reason, we made the message past that statistics was about proving things, which I don't think it is, I think that what we are as statisticians is people who take information and try if we're lucky and good enough to reduce uncertainty about stuff, because the world is so complicated that we just don't have a clue. We don't know almost anything about how the world actually works. And what we're doing is to try and say, Well, given these data, I think actually, you know, before I didn't know anything, and now I know a little bit, that's what statistics does, and but again, it's complicated because people don't like to hear you that. You know, the best effort that you produce is just still uncertain. People would like to know that. Oh, well. Yeah, I've done my stats and it's five but it's never five is it?

The number is 3.2. Yeah, yeah, yeah, exactly. And so actually, I'm curious if you have like, because you're not working in a statistic like you know, statistical department in a way you're like, much more applied. And so I'm wondering if you have, like, colleagues come to you with their, you know, stance problems. And so in which probe for which problems and in which circumstance, circumstances would you advise someone to learn and use Bayesian stats because like that may be also something that listeners right now have on their mind they have a problem at work or, and they're like, like, should I invest my precious time into learning more but that they should stuff or not?

So I do, I am in a Department of Statistics in fact, I have become last year the head of my department at University College London, and we are the first ever established Department of Statistics in the history of academia. So we're very proud of that. Obviously, you know, we've we've had a massive decrease in the quality of our heads, because he got to me at the end. So, you know, I'm very much I consider myself a statistician, I work in a statistical environment. The area of applications mostly is to do with biostatistics and this idea of decision making in healthcare. But in reality, a lot of the developments that we do are are more general and actually I enjoy working on areas that are nowhere near so to the healthcare arena. So, in fact, I was once at a conference and I got introduced to some people and the response was a very excited person who was just like, Oh, you're the Eurovision guy. Because they had read the paper that I written on Bayesian modeling of the Eurovision Song Contest, which was just a joke. Essentially, we were trying to do it because it was fun, but I guess it got picked up more than the serious stuff that I was doing. But also, I think this highlights kind of the answer to your question, it seems to me, you know, statistics is complicated. There's no easy stats, you can pretend it. You can do some easy statistics. You know, you just go to a software, click on a couple of menus and then you get the five number that you would like but you're not really doing statistics. If you're doing it seriously. It is a complicated business. It is something that requires technical expertise. And the Bayesian component of it is just like a marginally perhaps in the beginning anyway, a bit more complicated than the standard way of doing it, but it's not a complete change in in what you are doing. So if you're interested in and again, we've been doing work on stuff for fun, like you know, prediction of football results, or the Eurovision Song Contest that just to see whether there was bias, because again, we picked up in the media that some politician would just say, well, the UK never wins the Eurovision contest because Europe hates us so that you know we should Brexit because of that. And I was just like, No, that's not true. So we got the data and we actually did the model and but again, it can be applied to anything and so taking the effort getting to know the methods, if you come from statistics, I think it's your use, even if you're not a Bayesian statisticians, I think it's your duty to know that development. It's part of your discipline, so you should do it. If you're not a statistician, then you're still exposed to some stats. You need to understand how some things work. And you know, at least at the higher level, you don't need to necessarily be able to use your standard programming and create a particular module to run within Stan and then C++ and stuff. You don't need to do that. The statisticians that can do it for you, but you need to understand what happens at the higher level. So, again, I may be biased in this but it seems to me that everybody should have a working understanding of how you know, works and Bayesian stats is it's just marginally bit more complicated to start with, only to make it a bit more clear to me anyway, in the long run.

Yeah, yeah, I love that. And further record, I think, no, France already hated the UK before Brexit.

No, that's okay. It's yeah, it's a love hate. Relationship. Thankfully, it's much more in in football stadiums and rugby stadiums these days. That's Yes. Yes. Yeah, actually. Well, you you started talking about that a bit. So I'm wondering what do you think the biggest hurdles intubation work? So currently are?

I think, to do when I when I for example, when I when I work with my students and and I teach them and I'm very lucky because the stuff that I teach is very much related to the work that I do. So it's nice because I can kind of be very enthusiastic of the stuff that I show my students because there's the stuff that I do for a living anyway. The doing Bayesian statistics can be very complicated because you know, you you may need to learn new tools, you may need to learn something that is more of I think David Spiegelhalter has a fantastic book, which is the art of statistics, which I think is a brilliant title because you know, we think of us as scientists, which we are but we're also artists in the way that you know some things you need to have the kind of know how and particularly if you are from a Bayesian persuasion, you know, the prior sometimes you use some structure and you know, if it's a probability used to compete a priority, yeah, okay to start with, but then how you actually model that is a lot bigger is a lot more plain English or whatever language that you're trying to map onto mathematics onto distributions. So that may be more complicated than maybe a barrier, but in reality, the the understanding of how things work without you know, leaving aside the technicalities, how complicated HMC could be or MCMC or Gibbs sampling or whatever it is. I think that it's fairly intuitive what it is that we're trying to do. And then again, I don't I don't I disagree with people that say, oh, Bayesian statistics is a lot more complicated and frequently statistics because if you do things well, even frequentist statistics is very complicated. If you're doing it well. And you know, you don't have a normal distribution and and you have to write down a massive likelihood to maximize, you need to know what you're doing. It's not something that anybody can just whip up, you know, on pen and paper. So, the level of complexity depends on how well and how realistically you're trying to model your your problem your world and he just so happens I think that being Bayesian has some advantages because he lets you push the boundaries of of how realistic your model would be, which in turn makes it more complicated perhaps, but because it's a better model.

Yeah, exactly in a in the frequencies framework. If you don't have to test that exactly like for what you want, then you need to do extremely complicated mathematics to to derive the text, the test, whereas if you have the legal approach of the of the patient framework, yeah,

yeah, yeah. Sometimes you know, there's a there's a number, but there's a bias comparison. You know, if you have a fairly standard t test problem, where you have two bunch of people, and you calculate the mean, and you just do the t test to see whether they're statistically different, then maybe to do the equivalent version of that model in a Bayesian framework might seem a lot more complicated and a lot of work for nothing. But the advantages come up when the model is so complicated, and you have so many parameters and so much complexity, integration of different data, which like I was saying, I think it makes for a more realistic model. And at that stage, any model you make is complicated. And then the marginal incrementing complexity that the Bayesian modeling can can offer offsets the advantage is offset by the advantage that you have. So you know, overall, you don't lose if you compare real model complex models with complex models.

Yeah, yeah, I could continue on net. But I want to get a bit more technical because I'm really curious about what like the kinds of models you you do in your work so, so let's, let's switch gears and talk about that. And yeah, if you can take an example from your work to help us understand basically how you use Bayesian stats for decision making in healthcare.

That's, that's very interesting. I think that's one of the most interesting thing about working in HDA. So the typical model that we that we have is, is fairly complex, because, first of all, we are different than the trial clinical trial people in some ways, because we're not interested in seeing whether my drug is better than placebo. That's not the decision that we're trying to make. The decision that we're trying to make is, is my drug better than another? drug that is already on the market and the people are normally using? And actually there are five more different drugs that are already on the market. So which one is the best? That means that typically, you don't necessarily have evidence like head to head comparisons like trials for all the combination of different drugs. So most likely, what you will have to do is to combine different data sources, you might have a head to head trial of drug A versus drug B, or maybe versus placebo. Maybe you have a bunch of different placebo controlled studies with different active treatments and you want to try and figure out the indirect comparison. So you have a trial of V versus placebo and a trial of B versus placebo. So can you use that sort of data and information to figure out whether a is better than active drug B, rather than both compared with against placebo? So that's one thing. The other thing is that sometimes you have a combination of individual level data, because perhaps that's the trial that you've been working on or the study that you've been working on. So you do have all the information you have you know your outcome and then a bunch of covariates age and sex and other things about the patient. But often, for the other drugs, you don't get access to the individual level data because they belong to a different company, and you don't work for that company with that company anyway. So you need to combine a bunch of individual level data with aggregated summaries that may come from literature or maybe you get information through expert opinions. So again, that kind of speaks directly to the Bayesian way you're doing things because you need to combine different sources of data, potentially some data that are very contain very little information, data that are kind of bias perhaps and so you need to rebalance things and then the invasion may help in terms of having priors that can be centered away from the data because you don't expect the data to tell you the truth about a particular thing. And the other interesting part is that often, you need to construct what we call like a decision model. So there's a there's a purely stats component, which is about estimating a bunch of parameters. But what you do with these parameters, you don't do tests or even simple estimates with that. You just want to propagate the uncertainty on these parameters and combine them because eventually whether a drug is worth buying or not, is a function which is often a nonlinear highly nonlinear function of all of these parameters. So for example, you know the cost is one element, but the benefits may depend on odds ratios multiplied by some other parameters multiplied by something else, and then everything gets put together to calculate some kind of utility function. So it becomes a fairly complicated model. We've been working on models that had I don't know 100 parameters or something like that. So being Bayesian helps because you can be modular, you can do like different different little modules which might talk to each other if you have evidence to connect them, and then you estimate everything at once. But you don't stop there. You bring it to the next level, which is calculating these economic summaries and then making a decision on top of them.

Yeah, so that's an I find that it's really natural innovation. We're in the Bayesian framework to do that, right because once you have your posterior samples, then you just basically count the number of samples that are in the scenario you're talking about and the other ones that are not and then you get you're

absolutely right. Exactly, exactly. So you know, you do the decision model is based on getting the expected utility and then you have a winner, you have the treatment that seems to be better given current evidence because it has a higher expected utility. But current evidence is always a bit shaky. So you want to account for the underlying uncertainty, which is what we were saying before, so some people don't like that, but that's a fact of life. You're not certain that the new treatment will be best. And so you want to account for that in your in your decision model as well.

Oh, yeah. Okay. And so how do you like what's a recent example that you can take to help people understand how that kind of model for instance, would work? And yeah, like also the main difficulties that that usually appear when working on these kinds of projects?

I think the main well there are two levels of difficulty that then the main barriers The first one, I think, is the fragmentation in, in the bowel, especially the people who often essentially, this business is about two components. There's the estimation of how much better clinically something is in comparison to something else. Like, you know, if you take this simple example of a drug, how much better is drug A than drug B, essentially. And then there's the other component, which is about the costs, because you're not just interested in you know, if we had infinity money for every single disease, we'd always pick the best drug, but we don't and so we have to figure out the balance between how good the drug works and how much it costs to to administer and deliver to the overall population of patients. And crucially, whether you know, you might invest a bit less in this disease area because there are better interventions in other disease areas. And so you save a lot more lives by concentrating source resources somewhere else. And I think that often there is a divorce between who does one side of the story and who does the other side of the story. It's not always that there's the same team of people who have the same background with the same level of skills ability that would do the whole economic evaluation. Often people do very, very well, the clinical side of things. And then they just pass some kind of summary data to the people who will do the economic evaluation, which is a bit silly, and you know, it's not it's not the best way of doing it. I think. Perhaps the other barriers and this is something that we've been doing a lot of work we've been trying to kind of push people around and just, you know, tell them you have to do things this way. And often, the industrial standard is based on suboptimal tools like a lot of these models are still unfortunately based on kind of Excel spreadsheet, which is ridiculous. But you know, it's a it's a big industry. And it's not impossible that a lot of people in Nigeria would naturally go towards those kind of models. So there's a thing which is called our HDA. We have a website, or hda.org, which is a consortium of various academics and people working in this area, trying to convince people that you know, we should move away from non fit for purpose tools, like spreadsheets are not good for this because the models that you need to do are so complicated that you need to be doing it seriously. So you know, our or open source tools are one way to go. You could do it with any proper start software, but we focus on our because it's just easier and it has some advantages being open source and freely available. So you know, it opens up for jurisdictions where they have few resources and all the rest of it. So again, I think these things kind of compare, in some ways the development of proper statistical methodology that feeds into that. But I think you asked sorry, I took a massive tangent and I rambled on for for an hour but other things that I think you asked if I could give an example of things that we've done recently

that was already a part of part of the answer to the the other question, which was like the main difficulties that you were witnessing, so you should put the link to our HDA in the in the show notes so that people can can refer to that because I think it's really useful. Like the less Excel sheet we have in this domain, the better and yeah, okay, perfect and so, these these difficulties are interesting to me also, because, so it seems like eliciting priors basically don't seem to be a big problem here. Nikhat were How was your How has your experience been in the in that matter? I think

people are coming to accepting the fact that you know, you're asking them for for information and it's not frowned upon. It's not like Uber, you're cheating them. I'm asking you to tell you something about the model. Why don't you look at the data. There's an understanding of that. You need to be able to communicate, a lot of the people who work in this field don't necessarily come from statistics. So you need to, you need to figure out how to speak a common language. That's true, I think in any application area. Related to statistics. I think even if I worked in a different area, I would need to be able to do this. And I think again, this is something that comes with the art of statistics. Perhaps you learn how to do it. You learn how to communicate with people, you learn how to how to say things so that they would understand what it is that you mean and what it is that you're asking. It does help a lot to have a lot of visualization. You know, I remember the first few times that I was doing this, I was coming to it with a with a purely statisticians mind. So I would ask, what kind of distribution Do you think this thing would have? And clinicians would just look at me weird and just wanted to punch me in the face and they were right. You know, now you come with a different perspective. You just show them a picture and you say, does that sound like reasonable to you? Does that make sense? You know, is this kind of range what you'd expect this thing to happen within? And then again, it's your job to map that to change that information into a distribution. And to some extent, they don't have to see what you're actually doing. I don't need to tell them necessarily that the prior is normal. 6538 on the logit scale, they don't care. I need to show them a picture that says, Well, this is what you told me Does that make sense to you? So you know, once you learn to do that, then the conversation happens in a much easier way. And then people want to be involved because again, they don't feel like you're you're cheating information of them. You're actually telling something that they believe in and that they would be happy to justify in front of the panel and just say no, I said that because that's the truth because as a clinician with 50 years of experience that's what you expect people to have blah, blah, blah.

Yeah, I completely see what you mean. Yeah, like having people participate in the process is super important. And also you were like, they don't care what the name of the distribution is, and what the parameters are. And so to me, it's more and more useful to go to domain experts in like basically showing them prior predictive checks posterior predictive checks and asking them how, how they see those plots in AI how credible they think they are. Yes, it's much more useful and much more important and then like in PI MC, for instance, now we have that p m dot find constrained prior function where you can basically ask pi MC to use some mathematical optimization in the background. But basically, it's just like asking pi MC, okay, give me a normal distribution. Like which 95% of the mass is going to be between one and three. And that's all in that's all you have to specify and then pi MC will tell you okay, then you can use normal with that mean and that standard deviation as your prior, and you can do that on the go. You can then generate prior predictive plots. Thanks to that prior and then show that prior to the the expert in and validate it with him, and also before well, then the one inputting the values in the Find constraint prior function will be the expert telling you Well, most of the time, it's between one and three. Like sometimes it's been visible, but not a lot. I think that's

that's crucial. Again, we've said it before, the expert, the domain expert, the people you need to work with, they don't care about the technicality, but it's correct that they don't care about the technicalities, it is our job. You know, it's mapping, it's translating what they tell you in, in plain language into mathematics, but that's what we're trained for. That's what we should be able to do.

Yeah, exactly. And well, I actually made a YouTube video on like on a tutorial to show how to use that, that new function pi MC find constraint prior so I put that in the in the show notes for people and I find it's a way more intuitive way of finding priors and especially eliciting them, which is usually something that can be hard. Awesome. And so another topic that's super important, especially in health is causal inference. Right, exactly as you're saying. How do we know that the drug is more efficient than another one? And that actually, because it's the drain, not some confounding effect? So how does that work for your hand based on the structures of the models that you already talked about? How do you add a layer of causal inference into this?

Absolutely, that is a crucial thing. And actually, it's very interesting because the field of health economics again, it's a big mixture. You have lots of statistician working on it, but you have also economist you have modelers, you have clinicians. So I think it's been a bit slower to get to the point where everybody realizes that, you know, you need a bit more sometimes the model that you need to do is a bit more complicated because you have all of these neurons and all these complexities. So I think, by and large, the models that we do they they can be a bit bigger. And a bit more complicated than the standard kind of statistical model that you can think of when you need to. You have observational data, and you have to make sure that the different treatment groups are balanced. And you can do propensity score, you can do matching, there's all sorts of different things, of course, in reality, a lot of what we do can be fed directly through that area. And then on top of that, we have to embed something more, a bit bigger because maybe we have to model the costs. Or maybe we have to combine the parameters that were managed to divide us by doing some matching or something to do the actual economic evaluation. So a lot of the of the practicalities of the applied causal inference is based on methods that are existing and again, most of it is through the biostatistics literature. I think, in terms of applications of, again, all of these rebalancing methods. There are very interesting examples of which are perhaps a bit weird, and, and they only make sense if you start understanding where we're coming from. So imagine that you have a study like you work with some company data, maybe you know, you're a big pharma company you're doing you're more or less small study, and you give me your individual level data, and your trial is, you know, your new drug versus placebo. And it just happens that in your trial design you've selected a population that is fairly young. Yeah. So people to start with are not necessarily that sick. And so it may be that your treatment effect against placebo is not huge, because you know, not many people are very, very sick. So the treatment effect gets kind of diluted in some ways. Like I said, this isn't what we're interested in, because in the real world, placebo doesn't exist. So maybe there's another study that somebody else has done of another drug versus placebo. And because they've done it in a very old and sick population, then the treatment effect of drug versus placebo is huge. It looks like that drug is fantastic. He's really, really good. Now I can be naive and just take, you know, we have a common comparator, you have my drug versus placebo and your drug versus placebo. So maybe I can use that and kind of take the placebo effect away and have an estimate of my drug versus your drug. Problem is of course that I'm starting with two populations that are very imbalanced because mine is very young. And so the treatment effect is very weak. Yours is very old. So people are sick, and anything that you give them actually has a big effect. So you find yourself in a situation this is something that is called sometimes indirect comparisons or matched adjusted population comparisons. And there's a new strand of literature and we've done a lot of work with some of my PhD students and colleagues. And the idea is essentially to embed this element of causal modeling, to rebalance the two datasets that you have in the first instance, the complexity might be that, you know, if you had individual level data for all these two studies, you could do some kind of propensity scoring or something like that. But in reality, in our case, you have individual level data for my drug versus placebo, and only summary statistics for your drug versus placebo. So it's a bit more complicated, and there are some new methods that people have been trying to develop based on a lot of Bayesian machinery and kind of Bayesian G computation perhaps, to try and effectively rebalance things out and make sure that you can do a my drug versus your drug comparison, accounting for the fact that your drag looks better only because it's tested on population that is sicker than mine say.

Let me see. And so how would you do that? Here for instance, you knew you need two different models for each of the hypotheses that you had.

So essentially, what you do in a nutshell is you try to rebalance the treatment effect, making sure that you're you trying to test it on comparable populations in terms of the profile of covariates. So the different methods in which you can do it. Some are based on multi level modeling. Some others, like I said, in Bayesian G computation. So essentially, you take the information from the aggregated level data, and you try and reconstruct the effect that you see in your own individual level data on a population that has a profile of covariates that is similar to the other. So essentially, you kind of try to rebalance your treatment effect. If you were to do a study in a population that is actually older and sicker rather than the young and okay population that you've actually done the study on.

You see, yeah, so I understand the way multilevel models would be super interesting here. Yes. Because even if you didn't observe, like some parts, some cells of the population or had very sparse data, well, if you have fitted a hierarchical model on top of that, then you can generate posterior predictive samples from that sounds, even though you haven't observed a lot of the automation. Exactly. Uncertainty, uncertainty flows in the whole graph, and also like information flows in the whole graph. Yeah,

exactly. Exactly. So that's, I think, it's a very, it's a big thing in our field at the moment, lots of people are trying to work on it. So there's been quite some literature coming out. And again, I think it's very interesting because it comes from a very noble area of methodology, which is very much related to the business of causality and you know, rebalancing populations, making sure that you're comparing like with like, so it's very challenging, but it's very interesting. And again, the application is, is also very topical and it gets even more I don't know what the right word is fulfilling, perhaps is be pretentious. But if you see what I mean it's you know, because it's embedded in the bigger decision problem, then rebalancing things has an even bigger effect, perhaps. So it's a very interesting bits of modeling that you could do.

Yeah, no, I completely get that. I mean, hierarchical models for data. It's just like, still magical for me, even though I use them all the time. It's just like, I just love that and recently, actually, I recorded a video with one of our clients in PMC labs. I worked with them on the on a model on like, population model. Not for health, but more for basically, political orientation of people like trying to understand what people think based on survey data. And so the NGO is called SALC. They are from Estonia and they run they were in polls every every month. So they have those polls, but then they are interested in in demographic cells basically like so that could be so you have a big Russian speaking part of the population in Estonia, so depending on what you're working on as a as a politician you could be wondering, Okay, what does the Russian speaking men from 25 to 34. Think about what we should do regarding the war in Ukraine. And so of course, this is quite a specialized cell. Yeah. Which in your survey, you don't have a lot of people from but yeah, we used a hierarchical model and also poor stratification because they have really good demographic data in Estonia. And using that, then you can actually ask that question to the model. And to me, that's really wonderful because you get poorer estimations for these people for what they think, and they are pretty actionable. It's not like you don't get an uncertainty of like, okay, so they think the Estonian government should take more draconian refugees by 50% plus or minus 40%. You know, it's like it's really like the uncertainties are still very much. And three is just I know, I know how it works mathematically but still seeing it and how, like, useful, it makes your model for then the analysis in how people will consume the model to me makes it almost magic.

Yes, yes, it's true.

Yeah, I just saw that and so like if people are interested in that, well, there are a couple of, of episodes related to hierarchical models are ready on the in on the podcast so people can check that out. And actually now you can access if you want through our playlist from the podcast. So, folks, if you want, you can just like subscribe to a playlist with a particular topic. And it's a particular RSS feed. And if you add that to your pod catcher, you will receive all the podcast episodes, which are dedicated to that field. So for instance, that could be very easily biostats in health modeling playlist. And these episodes is going to be into it and so you would only receive these episodes on these RSS feed if you like, follow that when and also it's on YouTube. So I got three people. Yeah, so puts that in the in the show notes. also put a link to the video I made with tomasik internal URI still where we talk about that hierarchical Bayesian model of survey data with both certification. Bs. It's just like, yeah, it's a huge topic, as you were saying, but it's so useful and I encourage people to look into that because it sounds complicated, but to me, it's not that complicated. Because once you understand that, it all relies on the data generative model, then you you're pretty much good your your goal is to run the model and once you have that, you can generate poster samples for Sir predictive samples for any part of the population. And it's just that if you don't have a lot of data for that part of the population, then your prior is going to be more important and also the information you get from the other groups that are more or less close to those groups are going to be very important to

and I think, hierarchical modeling, whatever you want to call it. Multi Level modeling is one of those things where to try and put a non Bayesian spin on it actually makes it very, very hard because you need to think in terms of a super population of effects that exists in some ways, but not in other ways. Whereas to actually embrace the full Bayesian nature of it actually makes it very straightforward because effectively is like you're saying, you know, you have things that you observe then later in groups that you can define them through some measure of similarity exchangeability and then everything seems to me kind of flows more naturally. So it's not that complicated. Once you once you look at it from from the right angle.

Yeah, exactly. He's like these kinds of analyses, kind of like counterfactual analyses are like first class citizen in the Bayesian in the Bayesian framework. It's just like it's something I've been, I've been puzzled by actually, like, all that excitement about counterfactual, you know, plaatsen analysis. To it's really natural, it's just like, it's doing posterior predictive sampling in a way is just like you change the data that the model has seen. But actually, if you don't come from the Bayesian framework, it's like Ken over revolutionary to be able to do that is quite hard. But if you do that in the Bayesian framework, it's just like, Oh, you just swept the data and then see what your model is telling you. It's just posted. We've been calling that forward sampling and posterior predictive modeling for years. But no punch. It's called counterfactual. Okay.

In another life, almost in another century, when I was doing my PhD, actually, I was working on something related to causal inference, and part of it was about causal modeling. But I was working a lot with lots of the models that Phil David, who's then become my boss at UCI when I moved here, was working on so in a kind of causal inference without giving counterfactual so much prominence. And I think his work maybe hasn't got as much traction from the applied side of things. But there's a lot of value to it, because essentially, what he was saying is, all of his approach was there are problems where you need to rely on counterfactuals the things that he was calling the causes of effects when you're imagining something that you just don't have and you will never have the data to figure it out. And so it has to be this kind of potential metaphysical world. But in many other problems, actually, you can just think forward and do some kind of hypothetical modeling rather than counterfactual modeling. So I think counterfactuals are, you know, sometimes they're great and they worked really well. The very beginning nature, and I think it's no surprise because, you know, Don Rubin was a massive Bayesian statistician. And I think when he was thinking of these things, even in the missing data arena, which he did a huge amount of work on. He was thinking as a Bayesian and he was doing like somebody who didn't have the Bayesian machinery, but you know, everything that he has done in missing data, is just a Bayesian model. And now we're stuck with this weird kind of something in between where again, people think in terms of Oh, what if I could do simulations of my model and replace the missingness through some distribution, but then they do like Rubin started doing in the 70s, because he didn't have MCMC and all the computers, so you do the kind of 10 Multiple imputations. But that is just an approximation to a full Bayesian way of thinking about things.

Oh, yeah, I see what you mean. Yeah, that's, that's, that's interesting. efficient,

but again, it's no surprise because Don Rubin is was a massive Bayesian, so he would think that way naturally, yeah, it was just the lack of of tools that he didn't have at his disposal back then when he was working with these things.

Great. Oh, yeah, actually. I'm curious which package to use to run your your models.

I do a combination of things. So I am weirdly proud that winbugs And I share a birthday so the first ever release of Winbox was was out on the day on my birthday in 1997. So he winbugs is a bit younger than I am, but other than that, we shared a birthday. So I've been using it for a very long time, and I still do sometimes bugs or Jigsaw sort of the sampling sampler essentially software I've done and I can use, I think, to a very good level, Stan. So obviously, you know that that kind of takes a bit of a barrier to get into it's more than to understand how how it works and why it works in a certain way. I have done lots of work with INLA integrated nested Laplace approximation. I not dogmatic in terms of the software that I use, I think as a statistician even before as a Bayesian statistician, you need to be aware of the plethora of methods that are out there or tools that are there. So, I've I've tangentially, worked with Nimble, which again, is very similar to it's kind of a mixture between bugs and standards. The the language is very similar to bugs, but it's more of a probabilistic software rather than a sampler, as Balto jugs are. So, it depends a bit on the problem. I try and leverage the advantages of the different tools for the problem that I have. Yeah, okay.

No, I, I mean, I always ask that question, especially for people nickimja Because I find that interesting to see what are the the most useful package to them and also, always highlight the fact that they don't have to write their own custom MCC MCC MC MC simpler, like that's why Stan and pi MC and all the goodies are there for so yeah, let us let the nurse do that and then write the models. I think you might some

extent he might be it might be frustrating because you know, many people may come to this world and they take the time to learn one of the tools only to realize that when they finished understanding something and being able to do the models themselves, then the tool is kind of obsolete at that point. You know, you've spent a lot of time learning bugs and then he comes down and everybody jumps on that wagon because it you know, it has lots of advantages, and for good reasons. But maybe, again, maybe we need a bit of a better communication and to some extent, as long as you know what tools are doing as long as you know that they are out there. Maybe then the marginal impact of learning a new tool becomes a bit smaller, because you're more familiar and you're more solid with you know the basics. But again, I think if you have the luxury of being in in the position where I am where I can sort of try and steal even now make some time to see new things. I think it's our duty, we should take that kind of opportunity to to stay up to date, more than

Yeah. And actually, like I'm curious what the future of Bayesian stats look like to you, especially in your field. And more specifically, if there are things you'd like to see and things you would not like to see.

That's an interesting question. So I think like I said, and like I see on my website, I feel very much a statistician and I feel like statistics as a discipline should have a primacy I again, that might be my bias obvious because everybody likes what they're doing and they think that everybody else should do it. So I can see very good reasons for the huge popularity and development of what people call data science. And I don't have a problem with that. I think, however, that we should keep the connection we should keep maybe both alive we should recognize when a discipline feeds off another and again, I think as statisticians we haven't done a fantastic job of this, you know, historically, you have fields like econometrics, Epidemiology, you can I could be very annoying to people and just say that everything is statistics and you just call yourself an econometrician and people would want to punch me in the face and they wouldn't be right. But realistically, we've kind of made mistakes along the line in history by just focusing on our own very narrow, perhaps, methodology and an area of of application research. Whereas maybe we would have benefited more if we had had more connections and interchanging with the econometricians the epidemiologists and now the data scientists, so I think and I hope that's where we would go as a discipline. I I wouldn't want to see statistics kind of embedded somewhere else. I think there's a scope we need to be statistics. We need to make sure that we keep doing what we're doing. And even if it means having very, very close connections with all of our cousins, you know, we have to keep working with the computer scientists because we need better and more efficient algorithms to run our simulations. We need to work with the data scientists because obviously they need to bring problems needs, ways of improving the you know the way in which we visualize our data or pre process and post process our results. We need to keep talking to epidemiologists, because they have interesting problems and, and tools that they might need to expand. But I think I hope that we continue to be kind of central. I think one of the things that I really, really like about being a statistician is that essentially, you know, I can talk to many of my colleagues I'm lucky because UCL is a is a huge university, we have virtually all departments, you can talk to political scientists, doctors and historians, and I think there's very few chances where I couldn't actually push something about statistics in the conversation. And I think that's why so we should we should be very proud of that. We should try and make it stay for as long as we can.

Nice. Yeah. Yeah. Completely agree. I think that's that's very important in having that that diversity of approaches here is extremely important, especially when modeling to me that that that was super helpful. So time is flying by so before asking you the less quick two questions. I'm curious if like more globally. What do you think is your field biggest question right now you know, the one that you'd like the answer to before you die?

Do you mean statistics or in economic evaluation?

No, I'd seen economic evaluation.

I think at the moment a lot of people are doing a lot of work on two main areas. I think one is what I was saying before this business of comparing limited information from populations that are different, but that you need to kind of pull across and have an indirect comparison and the other one is the field of modeling which is related to kind of survival analysis and to event and this is very prevalent because for example, almost invariably every cancer study every cancer drug is first put on the market and then decided upon the reimbursement on the back of some kind of survival modeling. But we have a quick in the HCA niche technology in the health technology assessment, because most often the data that we have for the survival analysis are very mature. So you know, theoretically you would like to see a full survival curve where people are, everybody's alive in the beginning and then everybody's dead at the end of the observation, where you don't have censoring, essentially. But in reality, you get a lot of studies, particularly with the new drugs with immuno oncology in therapy that tend to work very well where the study is not long enough to get you to see a survival curve that goes down below point five. So you know, you don't even get to see median survival. And yet, we need to use those models those data to have a full extrapolation of the survival curve all the way down to zero because that's what you need for your economic model. So, again, I think there's a huge amount of work people have tried to figure out what is the best way to do an extrapolation that is sensible based on very limited information just at the beginning of the time horizon. People have suggested splines or flexible models but again, I believe, by and large, this is a problem that you can't solve just with one data set that you have because he has very limited information without including external knowledge. So without being fully Bayesian and openly Bayesian about it. So I think a lot in HDA is to do with this kind of thing, because there's a I don't know some people say about 40% of everything that happens in this arena is about cancer drugs, survival analysis type of things one way or another. So that's certainly a big area of research.

Okay. Well, awesome. I would have a lot more questions, but let's, let's call the show and, and let me ask you the last two questions, I ask every guest at the end of the show before letting you go of course. So first one, if you had unlimited time and resources, which problem would you try to solve?

I don't know. I mean, I don't think I would like to go and obsess too much about one single problem. Because I think the beauty of again, this is going to be very nerdy, but I am nerdy, so that's fine. What I really really like about my job on mind about my discipline is that you know, you you live your life and you listen to the news. You you go to watch my son's football game or something. And this always invariably something that you can think of in starts terms. So I guess that's the nice part. You know, that's not concentrating on one single thing if I had a limited time and unlimited money to do research I would just probably span it to even more than that unlimited money and time can can actually try and do. So I don't think that I would have a single a single thing that I'd like to do. But if you have unlimited money, you can send them along and we'll figure something out.

I see. I think if I had unlimited money I just invested for to make Neapolitan pizza available for anybody at any time. That would be a good

point. Actually. I haven't thought of that. Yes. You know, I we actually do we have a tradition in our family. We do pizza ourselves every week, at least once a week. So we just do the door and then I am the official pizzaiolo of our family. Oh,

yeah. Ooh, okay. Yeah. So now to come to London. Like normally I don't come a lot to London, but now I have reason. You have a reason now. Yes, for sure.

have a reason now yes for sure Okay Well

gentlemen great simile I learned a lot in I'm sure the listeners too. I mean they it was very tense episode. So the show notes were are going to be are going to be as dense as the discussion and that's really good sign. So yeah, as usual I put resources and a link to your websites in the show notes for everybody who wants to dig deeper. Thank you again Gianluca for taking the time and being on the show. Thank you. Thank you very much, Alex. It's been an

absolute pleasure and I've enjoyed it very much. I had lots of fun, so Well, thank you for having me. Perfect, Canada. Well, next

time we'll do that live while eating your pizza. Absolutely. That's the deal. Okay, so you can stop with SCT Okay, and now once you're in no desti you go to file, export, export as WEV. Yep. And then you save

it wherever you want. Make sure that it's 24 bits signed PCM. Yep, you keep the default metadata. And then you don't need to put anything in the

target is just Okay. Yep. Okay, good

and then do it. It's a big five. So, you just you can send that to me via Google Drive or Dropbox or whatever you prefer. We transfer and once I have that, I can start the editing.

There's one thing I wanted to ask you. So we do have a departmental podcast as well. Would you mind if we syndicated these episodes and put it on our sort of archive as well? Well, yeah, no, for sure. Yeah, for

sure. And obviously we can, you know, we

can share and just say that this is syndication from your own podcast and put all the links so they have kind of double the traffic, if it makes sense. Sure. Sure. Yeah. Do

that. That's actually more more efficient usually. So like, Yeah, anything that gets the word out more efficiently. I'm, I'm for so yeah, don't don't hesitate. I'll send you the so what I'll do is I'll send you the mp3 fine once it's edited and ready. So that you can also put that on your on your end. Okay, awesome. I

will

Transcribed by https://otter.ai