Episode

Let me show you how to be a good Bayesian.

Alex Andorra: My guests today are three of the biggest names in Bayesian statistics.

Alex Andorra: Andrew Gelman, Aki Vehtari, and Richard McElreath.

Alex Andorra: Today, we're talking about Bayesian workflow.

Alex Andorra: The book the three of them wrote together with eight other co-authors.

Alex Andorra: It is not an intro to Bayesian inference.

Alex Andorra: It's for people who already know the basics and want a theoretical, grounded way to make

Alex Andorra: dozens of judgment calls that come up in a real analysis.

Alex Andorra: How to build a model incrementally when partial pooling helps versus quietly dominating results.

Alex Andorra: How to do prior sensitivity analysis the reverse-based way.

Alex Andorra: And how to communicate a posterior to people who just want one number.

Alex Andorra: We'll also try something new on the show.

Alex Andorra: The grand prize winner of our contest, Dorota Wojcik, not only got a signed copy of the book by Aki, Richard and Andrew,

Alex Andorra: but she joins us live to present her own hard problem, modeling bat mortality at wind farms from noisy carcass counts.

Alex Andorra: And she puts it directly to my three guests.

Alex Andorra: Along the way, we also get into causal inference with leaky instruments, decision-making under uncertainty in regulated industries, and what's next for Andrew, Aki, and Richard.

Alex Andorra: This is Learning Bayesian Statistics, episode 164, recorded June 16, 2026.

Let's adjust those expectations.

What's a Bayesian?

It's someone who cares about evidence.

Alex Andorra: Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference,

Alex Andorra: the methods, the projects, and the people who make it possible.

Alex Andorra: I'm your host, Alex Andorra.

Alex Andorra: You can follow me on Twitter at Alex underscore Andorra, like the country.

Alex Andorra: For any info about the show, learnbayesstats.com is Laplace to be.

Alex Andorra: Show notes, becoming a corporate sponsor, unlocking Bayesian merch,

Alex Andorra: supporting the show on Patreon. Everything is in there. That's LearnBayesStats.com.

Alex Andorra: If you're interested in one-on-one mentorship, online courses, or statistical consulting,

Alex Andorra: feel free to reach out and book a call at topmate.io slash alex underscore and Dora.

Alex Andorra: See you around, folks, and best Bayesian wishes to you all.

Alex Andorra: Hello, my dear Bayesians. I have two fun announcements for you today. First one,

Alex Andorra: we have a brand new Instagram account for the podcast.

Alex Andorra: So if you want some behind the scenes content,

Alex Andorra: like how was it like to be at StanCon 2026

Alex Andorra: and play football over there

Alex Andorra: and what was the dinner like and things like that,

Alex Andorra: well, you can find that content there.

Alex Andorra: Also, I did some very short interviews

Alex Andorra: of some of the people you know and love from the podcast,

Alex Andorra: like Osvaldo Martin, Tomás Capretto, Oriol Abril, Bob Carpenter, Aki Vehtari, lots of people.

Alex Andorra: And I recorded these short videos and they are on the Instagram account.

Alex Andorra: All this content will only be there so that you have a more, let's say,

Alex Andorra: exclusive behind the scene feel over there.

Alex Andorra: So I will see you at LearnBayesStats.

Alex Andorra: That's the Instagram account.

Alex Andorra: You can also look for Pierre Simon Laplace on there.

Alex Andorra: So yeah, feel free to hang out over there too.

Alex Andorra: I would be happy to talk with you there.

Alex Andorra: Second announcement, I will run a Bayesian workshop

Alex Andorra: that will also double as a Learn Bay Stats episode

Alex Andorra: at the Carnegie Mellon Sports Analytics Conference

Alex Andorra: in Pittsburgh, Pennsylvania on October 23rd.

Alex Andorra: And also the conference continues on the Saturday, October 24th.

Alex Andorra: I will of course be there.

Alex Andorra: So if you want to come meet me and also attend my workshop, that would be awesome to see you there.

Alex Andorra: I will probably teach you something about bass, I'm C, and take a soccer model,

Alex Andorra: maybe that you've heard about from me, to teach you all of that.

Alex Andorra: So yeah, the idea will be to teach you how to causally infer the effect of a player

Alex Andorra: and the effect of a team and how to disentangle them with one model.

Alex Andorra: I will give you more detail later, but you can already sign up on the page of the conference that you will find in the show notes.

Alex Andorra: So hopefully see you there on Friday, October 23rd.

Alex Andorra: And also a big thank you to Ron Bjorko for inviting me and the podcast over there.

Alex Andorra: You can find out more about Ron and his background and his research actually in sports analytics in episode 140 of the podcast.

Alex Andorra: That was a very fun one. I definitely recommend it.

Alex Andorra: And now, without further ado, let's hear from Andrew Gelman, Aki Vehtari, and Richard McElreath for a very special of this podcast.

Alex Andorra: Andrew Gelman, Aki Vehtari, and Richard McElreath, welcome to Learning Bayesian Statistics.

Alex Andorra: Thanks.

Alex Andorra: Thanks.

Alex Andorra: Glad to be here.

Alex Andorra: Yeah.

Alex Andorra: Well, Aki, Andrew, you know the place now.

Alex Andorra: Maybe a different software, but you know the show.

Alex Andorra: Richard, first time here, so it's a real pleasure to have you on.

Alex Andorra: And we'll start a bit with you, you know, because you're the new kid on the block for this episode,

Alex Andorra: even though I'm sure everybody knows who is listening right now.

Alex Andorra: But I'm actually curious because I haven't heard too much about your origin story.

Alex Andorra: So I think most people know what you're doing nowadays.

Alex Andorra: And what I think is that maybe not a lot of people know how you ended up doing that.

Alex Andorra: So can you give us a bit of an origin story of Richard?

Richard McElreath: Okay, I'll give you a short origin story.

Richard McElreath: I should say first, yeah, thanks for having us and me on the show.

Richard McElreath: Long time listener.

Richard McElreath: first-time visitor. Yeah, what's my origin story? I guess now most people think of me as a

Richard McElreath: Bayesian statistician. It's sort of dominated my career now. But I'm technically an anthropologist,

Richard McElreath: an evolutionary anthropologist, which is what my current job is as the director at the Institute

Richard McElreath: in Leipzig. And I stumbled into that too. So my career is just a series of stumbling into

Richard McElreath: things that I need to do.

Richard McElreath: And

Richard McElreath: when I started college,

Richard McElreath: I didn't start off in anthropology.

Richard McElreath: I was doing,

Richard McElreath: actually, I did a year of

Richard McElreath: music theory and composition

Richard McElreath: my first year in college.

Richard McElreath: I thought maybe I wanted to be a music teacher.

Richard McElreath: And then I decided

Richard McElreath: I was just too average at it.

Richard McElreath: Right? So,

Richard McElreath: I mean, I enjoyed it a lot, but it's

Richard McElreath: extremely competitive.

Richard McElreath: I was just too average. At the same time,

Richard McElreath: I was taking ancient Greek because I had had Latin in high school and I really enjoyed translation.

Richard McElreath: And I enjoyed taking ancient Greek, but I decided there was no future in being a classicist either.

Richard McElreath: Sorry to classicist listening. And I randomly took an anthropology course just as a distribution

Richard McElreath: requirement and thought it was... And it's extremely interdisciplinary field. So it was

Richard McElreath: It was all the things I liked in one place.

Richard McElreath: I say also in high school, I was on the math team.

Richard McElreath: And I was pretty good at it.

Richard McElreath: But I didn't particularly enjoy math, so I didn't go into math in college.

Richard McElreath: But that has turned out to be useful because as I went on in anthropology, it turns out I could go into the, say, statistics journals and understand what was going on and translate things and read papers and so on.

Richard McElreath: Um, and, uh, my, my introduction to, to stats really, where I got more serious about it was in graduate school at UCLA. And I took the, um, the, uh, school of public health there teaches a quite intensive year long, uh, course for their, their math. I think masters in public health students is what it was designed for, but it's quite good.

Richard McElreath: and has a bunch of real issues about design and survey and dropout and Heckman corrections and

Richard McElreath: all those practical things and a bit of causal inference stuff because, of course, there are

Richard McElreath: some famous causal inference folks on the UCLA campus. And it was a great, great course. I

Richard McElreath: learned a ton from it. And my PhD supervisor at UCLA is also an anthropologist and he's a

Richard McElreath: theoretician. And that's sort of what I went to graduate school to do is to do evolutionary

Richard McElreath: modeling. But he's Rob Boyd's name. He's a well-known evolutionary anthropologist.

Richard McElreath: He doesn't do statistics really. So his advice to me was when I asked him how to,

Richard McElreath: I had collected some data and I wanted to fit it to a theoretical model. His advice was, oh,

Richard McElreath: no, go get a book on maximum likelihood. And I think that was good advice in the sense that

Richard McElreath: like, actually, they get you pretty far. I mean, this is maybe sacrilege for a Bayesian podcast,

Richard McElreath: But it kind of gets you pretty far and it gets you on the Bayesian course, I think. And so, yeah, I went from there and I ended up reading a bunch of books and including Andrew's books, of course, at that time. And and Jane's, I found Jane's, which, of course, Jane's is a mixed bag, but it's very entertaining.

Richard McElreath: is sort of the most interesting, and I like eclectic, heterodox things.

Richard McElreath: So that's how I sort of got into it.

Richard McElreath: And then I got a, well, you know, my career, I moved around.

Richard McElreath: I had a few professional jobs.

Richard McElreath: I eventually got a job at UC Davis, University of California, Davis,

Richard McElreath: in the anthropology department.

Richard McElreath: And I was a first-year, untenured faculty member,

Richard McElreath: and the chair of the department heard that I was good at math.

Richard McElreath: At that point, I had written my first book.

Richard McElreath: I hadn't published it yet, but I had a manuscript. My first book is an evolutionary game theory book. And he's like, you're good at math. We need someone to teach statistics. So you're going to teach statistics. And I was like, yes, sir. Because, you know, I want a tenure. But I really had never taught stats before and I wasn't sure how to do it.

Richard McElreath: So, you know, I started trying to develop a curriculum. I got a lot out of the old regression, multi-level regression, Gelman and Hill sort of approach. That's sort of where I started from.

Richard McElreath: I think I know that book.

Richard McElreath: Yeah. And it's a classic, right? And no, it's got this pragmatic style. I mean, all joking aside, it stood out, especially at that time, and Peggy was this very pragmatic, applied style, very interested in the data and the scientific background and all the examples. And that really stood out, I think, as Statsbook goes, that they're going to teach multi-level modeling.

Richard McElreath: and um there's many many more books now that were of course inspired by that book i think

Richard McElreath: including mine and then i started adapting the curriculum to my students and one of the things

Richard McElreath: that that was important to me was that when they work with the software that they are really

Richard McElreath: writing every line of the assumption in the bayesian model so i wrote my r package for that

Richard McElreath: reason to be maximally annoying so that they would have to write every line which i'm not

Richard McElreath: against convenient software, the formula interface like in BRMS is great. But for teaching, I wanted

Richard McElreath: an interface that's much more explicit, but isn't all the way into like raw Stan code. Now, of

Richard McElreath: course, Stan didn't exist at the time, right? Or it was a twinkle in Andrew's eye or something.

Richard McElreath: And I didn't want to have them write in bugs exactly either, because I wanted to do simple,

Richard McElreath: start with simple, like maximum posteriority stuff that would run really fast. So I wrote

Richard McElreath: my R package with that purpose in mind, just as a teaching tool. And it kind of cascaded from there.

Richard McElreath: And then I should say that, you know, like everybody's career, I think it's just lots

Richard McElreath: of lucky coincidences. The stats department at UCLA has some people in it who were editors of

Richard McElreath: that series, the red cover series. And they kind of heard gossip on campus about my course and it

Richard McElreath: gotten popular and, and the book kind of fell out of that and got into that series that way.

Richard McElreath: And it's a great series because you get a lot of, you know, it's in the same series as BDA,

Richard McElreath: Bayesian Data Analysis. So you get a lot of free press by being listed with it.

Richard McElreath: And that helps a lot as well. Anyway, I think that's kind of the story. And, and the stats

Richard McElreath: thing is, I mean, I think it's really taken over my career almost, I mean, I'm probably 20% an

Richard McElreath: anthropologist and 80% a statistician now, which is very rewarding because there's just a very

Richard McElreath: large interdisciplinary audience to benefit from my efforts. Anthropology is a very small field.

Richard McElreath: I love it to death and I'm not going to give up on it, but it's very, very small. So the impact,

Richard McElreath: the effective impact from work on stats has been very satisfying to see how much people get out of

Richard McElreath: recorded lectures and things that right anyway i'm being a bit on wind apologies but um yeah i

Richard McElreath: just stumbled into this and we'll see what i stumble into next maybe i'll start a podcast

Alex Andorra: and yeah yeah yeah if you if you want to take over you know i could i could use some holidays

Alex Andorra: yeah so we can try that maybe sometime um yeah i mean your your career is really super interesting

Alex Andorra: and I remember talking about it with you last year at StanCon.

Alex Andorra: This year, I need to get it again on the show.

Alex Andorra: Maybe at StanCon, we'll see.

Alex Andorra: Not announcing anything.

Alex Andorra: Talk a bit more about anthropology and how you do that concretely.

Alex Andorra: I think it's super interesting.

Alex Andorra: It's going to be super concrete for listeners.

Alex Andorra: Also because anthropology, I agree, is extremely interesting and so fascinating.

Alex Andorra: so definitely something we need to do but today we're gonna focus a bit more about I think you

Alex Andorra: three have a book coming up that's that's what I heard on you know on campus also like just

Alex Andorra: whispers and so I want to talk about that but first yeah how how did the the idea the idea of

Alex Andorra: the book come up um who like did one of you had mainly the idea of contacting the other one or

Alex Andorra: the other ones or was it something like spontaneous what happened was that we

Andrew Gelman: we had the you know bob carpenter wrote the stan manual the users in the stan users guide

Andrew Gelman: and but it's kind of written it's great but it's written a kind of computer computer manual style

Andrew Gelman: like it it has little snippets of code but it doesn't really have examples and stuff run from

Andrew Gelman: beginning to end and so we were talking about having a book a stand book um and then i made

Andrew Gelman: some draft this was almost 10 years ago uh then it somebody put it on github and then everyone's

Andrew Gelman: supposed to help but then like nobody does anything or people do arbitrary things it was

Andrew Gelman: like nothing was happening so we decided just to have a bayesian workflow book and not like not tie

Andrew Gelman: it to the Stan project just because Stan is great, but any project gets paperworky and like, then you

Andrew Gelman: have to go through certain rules and then you can't actually get anything done. So like, I don't want

Andrew Gelman: to have to have to do a pull request to like add a sentence to a book. So yeah. And then,

Andrew Gelman: so we had this idea of writing the book. And then a few years ago, we wrote an article

Andrew Gelman: The purpose of the article was really just to get us to start the book.

Andrew Gelman: So we wrote the article and then stuck it in an archive.

Andrew Gelman: And then the original idea was roughly each chapter, each section of the article would be a chapter of the book.

Andrew Gelman: And then we reorganized it.

Andrew Gelman: And we have 11 authors.

Andrew Gelman: And the other eight authors have done a lot, too.

Andrew Gelman: So it's like the three of us were kind of kept the project moving and basically took responsibility for all of it.

Andrew Gelman: So all three of us read every word of it.

Richard McElreath: Yeah.

Andrew Gelman: Yeah.

Andrew Gelman: And they wrote every word multiple times.

Richard McElreath: What?

Andrew Gelman: Rewrote every word.

Andrew Gelman: Well, they rewrote every word.

Andrew Gelman: There are parts of the, you know, some of the case studies were written by other people and original versions.

Andrew Gelman: And there were bits taken from the article that had been written by co-authors there. So we didn't literally write every word, but we take responsibility for it. And then Aki put together, with Jonah Gabry, put together a website which has all the code.

Andrew Gelman: And there was an earlier version of the book where the code didn't all run, kind of like my other books.

Andrew Gelman: But this time we made sure it all ran.

Andrew Gelman: And so that's pretty exciting, too.

Alex Andorra: Yeah, yeah, definitely.

Alex Andorra: And so these links are going to be in the show notes for this episode.

Alex Andorra: Both the website from Aki and Jonah and the original paper.

Alex Andorra: So folks, feel free to look into that.

Alex Andorra: And also, of course, the link to buy the book if you want to read that.

Alex Andorra: I'm pretty sure if you're listening to that, you want to read the book.

Alex Andorra: And actually, can you give us, guys, the elevator pitch of the book?

Alex Andorra: What are you trying to do with that book?

Alex Andorra: And why did you think it needed to be written?

Alex Andorra: and does it have a particular audience,

Alex Andorra: like the who of the book, if you want?

Alex Andorra: Who wants to take that one?

Andrew Gelman: Aki, you want to take that one?

Aki Vehtari: Okay.

Aki Vehtari: I think that we wanted to put there

Aki Vehtari: everything we know that's not already in BDA3

Aki Vehtari: and statistical rethinking.

Aki Vehtari: Plus, of course, then these case studies

Aki Vehtari: that instead of just saying, like giving recommenders what to do, actually so that in practice,

Aki Vehtari: that you can follow for different data sets, problems, the steps from the beginning to the end.

Aki Vehtari: Of course, different case studies are also focusing on different aspects.

Aki Vehtari: So not all case studies go through all the different workflow steps we describe in parts one and two, but in a way that each case study shows actual steps, what happens, what we can see from different steps and diagnostics and so on.

Andrew Gelman: And I see, to me, there's a big picture that, which is mentioned in the book, of a spectrum. And on one extreme, there's examples, and on the other extreme, there's theory, and this is somewhere in the middle.

Andrew Gelman: So the idea is that you have theory, like mathematical theory, like in Bayesian data analysis, here's Bayesian inference, and you can figure out the answer. And then there are methods, like specific models to use and so forth.

Andrew Gelman: then on the other side you have examples and case studies and to us workflow is something that

Andrew Gelman: i think it's more codified than a case study but less formal than a method like maybe everything

Andrew Gelman: would like to be ultimately a theory and before it gets to be a theory it would like to be a method

Andrew Gelman: But these are things that it wasn't quite like, it's not quite an algorithm or a method, but it's things that you would want to do. So maybe the most important idea we emphasize in the book is simulation-based experimentation, fake data simulation.

Andrew Gelman: So we use simulation to do posterior predictive checking, and that's an old idea that's in Bayesian data analysis and prior predictive checking.

Andrew Gelman: But just kind of more generally, the idea that you understand, you can understand a process by doing some simulation.

Andrew Gelman: If you fit a model to data, the first thing you can do is fit data from the model and check that your procedure or see if your procedure can recover things.

Andrew Gelman: You learn so much from that.

Andrew Gelman: And that's something that's, it's not really in our other books in that way.

Andrew Gelman: Like we have examples where we do that, but the idea that this is sort of a very, this is a technique you should do, be doing all the time.

Andrew Gelman: Then there are more specific things, iterative, like iterative algorithms.

Andrew Gelman: The starting point is very important.

Andrew Gelman: That tends to be hidden either because people are kind of embarrassed about it or they think it's kind of cheating.

Andrew Gelman: So we have a little.

Andrew Gelman: section where we just talk about the importance of starting points we've made a little example

Andrew Gelman: that's kind of a principle um shoot there was another one of these things i was going to

Andrew Gelman: mention um now i'm forgetting anyway different things come up like certain oh we have a long

Andrew Gelman: section about priors bda we have what we call the bayesian cringe that it's like true historical

Andrew Gelman: thing that we are embarrassed to use prior distributions. My book with Jennifer also,

Andrew Gelman: it's the same thing. We have a lot of reassuring the reader that, no, you don't really need a prior

Andrew Gelman: with a hierarchical model. You could just estimate the hyper-pronters from the data.

Andrew Gelman: Over the years, we started to realize that actually even with a hierarchical model,

Andrew Gelman: you can get a lot from prior information. That's really part of the workflow also.

Andrew Gelman: and so we put a lot of stuff in there that's not in Bayesian data analysis about like ideas like

Andrew Gelman: how to think about priors for effect sizes and a lot of stuff we've been thinking about um the term

Andrew Gelman: workflow itself I really like I don't know where exactly we picked it up I think it's in the air

Andrew Gelman: in engineering um and it just it's it seemed right I think there's a little confusion because

Andrew Gelman: It's not a checklist, like a workflow, like the workflow if you need to repair your air conditioning unit, that kind of thing.

Andrew Gelman: But it's a set of tools.

Yeah.

Andrew Gelman: Richard?

Richard McElreath: I think my perspective on the book is that it meets directly the kinds of questions I get from colleagues and students much better than the other, say, my other book, which I also like to think it meets that.

Richard McElreath: But this book is much more for people who already know the basics. This isn't a book to teach you Bayesian inference, but it's to help you become more professional about it and have the ability to make theoretically informed justifications for the particular branching points that are present in realistically complicated analyses.

Richard McElreath: Lots of books, including mine, have really simple curated examples for teaching.

Richard McElreath: They're like the data sets come clean in the example, and then you go right to one part of the workflow that's relevant to that example.

Richard McElreath: And that serves its purpose in an introductory course or someone's first course in Bayesian modeling.

Richard McElreath: But for actual practice, for people who end up doing this for a living, like yourself, you know, the real workflow is much more complicated. And there's a bunch of decisions. And those decisions, when they're not presented in a book, it seems like they're just arbitrary. But of course, there are principled, theoretically underpinned ways to make those choices. And we wanted to present some of that.

Richard McElreath: And I think the book's very unique in that regard as being much more of an engineering, if I can use that word, maybe Aki appreciates that, but much more of an engineering approach and respecting the idea that there are lots of little decisions along the way.

Richard McElreath: and you should feel free to explore them with, as Andrew said, simulation-based inference,

Richard McElreath: but also that there are principal diagnostics and ways to inspect, say, the strength of priors

Richard McElreath: and those things as well. And so this is a book that lays that stuff out in a clear way.

Richard McElreath: It also has already ended up being useful to me professionally because lots of the times

Richard McElreath: in collaborations or just colleagues will write me and say, I have this manuscript and I need to

Richard McElreath: cite weekly informative priors, like the justification for that. And so this book is

Richard McElreath: basically ideal. It's the new state of the art for having a mature reason section on say weekly

Richard McElreath: informed priors and what they really mean and why we prefer them as we do. And there's other things

Richard McElreath: too, in various cases. And I forget, there was an example that came up earlier this year where

Richard McElreath: a student has done some trick that I had taught her and then needed to cite it. And I was like,

Richard McElreath: oh no. And it turns out it's in the book. And I think Andrew wrote it in there actually,

Richard McElreath: and not me. I forget what it was, but it was some nice trick for solving a problem.

Richard McElreath: And so it's full of stuff like that too. It's got use. The first half of it goes through steps of

Richard McElreath: the workflow and lays out theoretically informed diagnostics and principles of choice and uses

Richard McElreath: thin examples to motivate those. And then the second half is the case studies where they get

Richard McElreath: applied in the context of a full example, often a quite complex example that weaves together

Richard McElreath: theoretically derived generative modeling with estimation. So I think it's also nice in the

Richard McElreath: sense that the examples are much closer to the sort of realistic complexity, at least some of

Richard McElreath: Some of them are simple, but a lot of them are quite complicated, especially in going beyond off-the-shelf models to something that's really generatively bespoke to the scientific background, which I won't speak for my colleagues, but I think they agree is that that's the thing that attracted me to Bayes in the first place was the ability to take in all of that scientific background I have as a professional scientist and express it in a joint probability distribution.

Richard McElreath: and some of the examples are like the golf putting one this is really nice in that regard

Richard McElreath: aki wants to say something yeah yeah what you said i reminded also that like

Aki Vehtari: scientific papers have the final model everything just works like so so only the final model so

Aki Vehtari: that all the diagnostics were great and many of our case studies then illustrate problems something

Aki Vehtari: doesn't work and how we then go forward from that how we diagnose the problems but also fix the

Alex Andorra: problems and how this kind of iterative process is okay yeah yeah i mean i i love that so much in

Alex Andorra: in in your own research ackee and also what what you folks are doing at at alto and i think that's

Alex Andorra: also why i have so many people on the show came here to uh to take over their research because

Alex Andorra: it's really not only um about the final output but trying to help people along the whole workflow

Alex Andorra: and the whole sausage making which is actually the hardest part like which is always something i

Alex Andorra: say um people who come and ask me about how they can get started into that is like i'm always

Alex Andorra: telling them to try and find something they really care about a problem they really care about because

Alex Andorra: um they're gonna have all the models they're gonna work on are gonna fail except for the last one and

Alex Andorra: even the last ones you're the last one you're gonna be like yeah it it works but they already

Alex Andorra: know how to improve it you know so it's like always the final output something you're happy

Andrew Gelman: One thing we like to say is that even if you knew ahead of time what model you wanted to end up with, you'd still want to be fitting simpler models to understand what you got out of fitting the more complicated model.

Andrew Gelman: And you'd also want to fit a more complicated model just to see that you didn't get more out of that.

Andrew Gelman: And a key part. So Bayesian data analysis, like back when it came out in 1995, was kind of revolutionary. And I called it Bayesian data analysis rather than Bayesian inference, because our big thing was that there are three steps. You build the model, you fit the model, and then you check the model. And then you can expand it.

Andrew Gelman: Before that, people really didn't talk about building the model and they didn't talk about checking the model. They pretty much only talked about fitting. But the Bayesian workflow or statistical workflow goes beyond what we had in Bayesian data analysis because that's all about you build the model, you fit it and you check it. Maybe you expand it.

Andrew Gelman: But a lot of workflow is going from model to model. And a lot of the operations that we do in statistical workflow can be framed as fitting additional models.

Andrew Gelman: Even gathering new data could be thought of as fitting a new model to data in the sense that if I have a bit of data and I fit a model, then I get more data and fit a bigger model.

Andrew Gelman: You could think of the original version as a very simple model fit to all the data, which just didn't use the new stuff.

Andrew Gelman: So thinking about our workflow, I mean, it has a lot of implications.

Andrew Gelman: So one implication is like, how do you, like if you have an iterative algorithm, how long do you run it?

Andrew Gelman: So some people say, oh, well, just to be safe, run everything overnight.

Andrew Gelman: Well, that doesn't really work with workflow because that will reduce the number of models you can realistically fit.

Andrew Gelman: And we learn so much by fitting model after model after model.

Andrew Gelman: And then being able to simulate from a generative model means that you can.

Andrew Gelman: Well, let's say this.

Andrew Gelman: If you think about, like, I like to say that all statistics is frequentist, including Bayesian, because we're averaging over the prior, which is a frequency.

Andrew Gelman: Ben Goodrich doesn't like when I say this, but I think this is basically right.

Andrew Gelman: But if you think about in old school textbooks, what is frequentist statistics?

Andrew Gelman: You use a method and then there's a mathematical theorem saying that it has certain statistical properties.

Andrew Gelman: But then in the last 50 years, there's been a lot of papers where people have, they can't do that, so they do simulation studies. So lots of papers by like Stanford types, which will have tables at the end with a bunch of numbers that look like 0.93. They're like between 0.93 and 0.96, which are supposed to demonstrate the coverage of their 95% intervals under various conditions.

Andrew Gelman: But what's really cool is that you don't need the, I mean, the theory helps because it gives you some mathematical intuition, but you don't need the theory and you don't actually need that published paper.

Andrew Gelman: Like if you're going to use a method, a model, traditionally you would say, well, let's use a method or a model that's been published somewhere.

Andrew Gelman: And then let's hopefully somebody wrote a paper demonstrating that has good frequency properties.

Andrew Gelman: But now what do you do?

Andrew Gelman: I'm going to fit a model and it's a new model, never got created before.

Andrew Gelman: I can simulate from the model and I can check that at least under ideal conditions, how well can it reconstruct, how well can the parameters can be estimated.

Andrew Gelman: And so I can build this tool, giving me more confidence in my own work. So we try to demonstrate that to people that what's necessary, what's important is not just to get the good answer, but to get an answer that you know is good or you have some degree of confidence that's good.

Andrew Gelman: And the idea that people can, we can empower people that as a user, you can check your own model. And I think that's not just a fit to the data, but check its statistical properties. That's super important. And it's not really something that we had been saying before. We had been living it in some way, but writing it down makes a difference.

Alex Andorra: yeah yeah completely agree and and i am very impatient to get my hands on on this book and

Alex Andorra: see what it looks like and the content of it i can tell you and i'm sure all the listeners here are

Alex Andorra: and uh actually thanks to your editor um everybody listening get a discount code that will be in the

Alex Andorra: in the show notes folks so uh definitely use that if you wanna you wanna read the

Alex Andorra: that book uh it's gonna get you directly on the site of the editor and also actually before

Alex Andorra: the show um i ran a small context uh for people to you know like basically give me other

Alex Andorra: hardest Bayesian workflow problems.

Alex Andorra: I guess it's like, you know, it's my French laziness.

Alex Andorra: I didn't want to think too much for this episode.

Alex Andorra: So I was like, I'm going to outsource the thinking.

Alex Andorra: No, so kidding aside, I got a number of great contributions

Alex Andorra: from a lot of people listening to the show and curious about the book.

Alex Andorra: and uh one of the one of the main ones uh was about bats so modeling bats and and trying to

Alex Andorra: make sure that um like that your model works and we're gonna get the um the person who actually

Alex Andorra: won the contest on the show in a few minutes she's gonna join us her name is uh dorota she's from

Alex Andorra: Poland. And so when she joins, I'll have her present her problem to you folks and hear what

Alex Andorra: you would tell her. But before that, I wanted to ask you also, because you write a lot, you teach

Alex Andorra: a lot. So I'm actually curious in this book, is there like what's, it's going to be a hard question,

Alex Andorra: But what's the main thing you would say, the main insight that you added to the book that was something that was new for you in your teaching, like a new insight that you got and you thought was very important to put in this book?

Alex Andorra: Maybe Richard, if you want to start with this one.

Richard McElreath: That's a good question.

Richard McElreath: I think it's really as simple as just being much more explicit about the structure of the workflow.

Richard McElreath: And so I just have an updated set of slides for my course that I taught in January this year, starting in January this year.

Richard McElreath: And it does have a lot more workflow diagrams in it.

Richard McElreath: And the feedback I got from students is that they really enjoyed that.

Richard McElreath: And it seems like a simple thing, but it's not.

Richard McElreath: but to actually draw out the workflow like a network and make it distinct from just a pipeline.

Richard McElreath: I think most of your listeners are probably familiar with the idea of a pipeline.

Richard McElreath: There's similarity between a workflow and a pipeline, but a workflow is much more dynamic to start with

Richard McElreath: because there's engineering parts involved in it where the pipeline is something you deploy.

Richard McElreath: Maybe you have to fit the model every day like in your line of work, right, Alex?

Richard McElreath: and something's in production and so on.

Richard McElreath: But that isn't what I'm talking about.

Richard McElreath: And so the workflow is different than the pipeline

Richard McElreath: in that sense, that there's more dynamics

Richard McElreath: and we explicitly label decision points

Richard McElreath: and the diagnostics that we might recruit at those points.

Richard McElreath: But also that it's not just steps,

Richard McElreath: but it's also justifications.

Richard McElreath: It's reasons for how,

Richard McElreath: why the inputs were combined in this way,

Richard McElreath: like why this model,

Richard McElreath: given the combination of the question

Richard McElreath: an generative model, for example. And then all the way towards the end of the workflow where

Richard McElreath: we're doing the summaries and the summaries again are justified in light of the questions

Richard McElreath: that we want to calculate. We're going to do these set of marginal effects because they're the right

Richard McElreath: answer to the original motivating question and not something else. And I made a series of diagrams

Richard McElreath: for that and students really enjoyed that a lot. And I think they learned way better from that

Richard McElreath: as before. And it's interesting that just that change in the visual representation

Richard McElreath: and the rhetoric about it had, I think, a strong effect on learning in that regard.

Richard McElreath: Personally, in my own research, writing the book was very stimulating because a lot of the book

Richard McElreath: actually is unsolved problems in workflow. Like many of the sections, like here's what we know

Richard McElreath: now and here's what we'd advise and this is why. But this is an open problem. We need more work

Richard McElreath: here. There's a bunch of places where we can think about good components of the workflow,

Richard McElreath: but it's just inconvenient to do it right now. And a lot of those have to do with things that

Richard McElreath: which we might call reverse bays, which is an old idea going back decades. But the idea that

Richard McElreath: you might want to vary components of the model. So reverse Bayesian, I think this is a term from

Richard McElreath: I.J. Goode originally, is the idea that you solve for the prior rather than the posterior. You say

Richard McElreath: you could start with a model and a posterior and a data set and say what range of priors justify

Richard McElreath: this. So we think of this as prior sensitivity analysis. And Aki's priorsense package is a huge

Richard McElreath: advance in the convenience and transparency of doing things like this. But there's lots of

Richard McElreath: innovation to do along those lines. So what I call the reverse Bayes approach as essential components.

Andrew Gelman: If you want to formalize that, I mean, I hadn't heard about that before now, but if you want to

Andrew Gelman: formalize that may be the way to think about it. And there's two ways of thinking about the prior.

Andrew Gelman: So one is the prior is the range of problems for which the method is intended to be applied or to

Andrew Gelman: which the method gives reasonable answers. But another way is to say the prior represents the

Andrew Gelman: population. So I wonder whether what you're calling reverse Bayes could be formalized as

Andrew Gelman: just Bayes where you're trying to do inference for a population distribution.

Andrew Gelman: I mean, that's like hierarchical modeling. So you're saying like usual hierarchical modeling,

Andrew Gelman: we have the data and from the data and our data model, we can make inference about the population

Andrew Gelman: distribution. Now you're saying something different. You're saying something more like

Andrew Gelman: this is our posterior. So what should the prior be? But I guess, I guess the statement,

Andrew Gelman: this is our posterior, probably there's a way of framing that in terms of data is my guess.

Andrew Gelman: So it could be sort of as say, cause I, I think I, I J good had a kind of annoying writing style.

Andrew Gelman: I think, I mean, he has a lot of interesting ideas, but I'm saying like to call it like this alternative approach, like maybe there is a way of framing it as a full Bayesian approach.

Richard McElreath: As an open problem to be refined, I would say that I remember his example was something as simple as, and you've done this, I think, Andrew, and I hope you'll accept the charge.

Richard McElreath: And you take, say, a non-Bayesian analysis of maximum likelihood and you say, what priors make this sensible?

Richard McElreath: I've seen you do this a bunch of times and show that the priors which would make that conclusion sensible are just not scientifically reasonable.

Richard McElreath: yeah and that's that's kind of the reverse-based approach is what range of priors are compatible

Richard McElreath: with this conclusion and are they also compatible with the scientific information we had

Richard McElreath: and i think making that an efficient and routine part of workflow means that we need to do some

Richard McElreath: more theory and formalization as you suggest and figure out what the boundaries are but also some

Richard McElreath: tools we need tools for people to pick up and and make it convenient so they don't have to refit

Richard McElreath: models over and over again. Some important sampling, right? Aki is always a useful thing to

Richard McElreath: have on hand. But I think there's a bunch of open problems along these lines. In addition,

Richard McElreath: there's a bunch of stuff in the book that I quite like, which is about even when you know the model

Richard McElreath: you want to use for inference, you need to build it incrementally, not just for the understanding

Richard McElreath: regions, as Andrew said earlier, but also just for software engineering, responsible engineering

Richard McElreath: reasons you you do feature engineering incrementally and having good advice about that

Richard McElreath: is i think still there are open problems there for a complicated hierarchical model

Richard McElreath: um uh it's it's there's a bunch of paths you can go to build it and um and there may be trade-offs

Richard McElreath: involved in that and and how it works and i have this tendency to start with what i call the empty

Richard McElreath: model with just the random effect structure and get that working before I build in any predictive

Richard McElreath: variables at all. But I have no justification for that, except that's the way I do it.

Richard McElreath: And I'd like to have some theory or some set of examples that we could develop that better

Richard McElreath: and think about how this works. Andrew has some interesting ideas about model hypergraphs,

Richard McElreath: which relate to this, I think, and what we learn from models which overlap in their features.

Richard McElreath: And so there's lots of open problems to do that came up to writing the book. And so we want to invite readers to help out with these problems or nominate new ones for us as well.

Andrew Gelman: Well, it was also fun to write because we could just kind of pour ourself into it. So just for example, on page six of the Bayesian workflow book, right at the beginning, section 1.1 is called Why Bayes? And then we have a subsection starting on page five called On the Borders of Bayes.

Andrew Gelman: and then on page six we talk about Bayesian interpretation of non-Bayesian methods

Andrew Gelman: and we I mean we start we say a sometimes annoying habit of Bayesians is to take non-Bayesian methods

Andrew Gelman: and give them Bayesian interpretation for example maximum likelihood is just Bayesian inference with

Andrew Gelman: a flat prior fixed effects are just random effects with a group level variant set to infinity

Andrew Gelman: and a few other examples.

Andrew Gelman: And then we talk about how that attitude can be valuable,

Andrew Gelman: but isn't always quite right.

Andrew Gelman: Like it's very refreshing.

Andrew Gelman: I feel like when you write journal articles,

Andrew Gelman: you're under such pressure to never admit

Andrew Gelman: that you could be wrong

Andrew Gelman: because the reviewers will jump on you.

Andrew Gelman: And so, yeah, we talked about a lot of that stuff.

Andrew Gelman: The other thing I wanted to say is

Andrew Gelman: if you have a paperback version of the book

Andrew Gelman: and a pair of scissors,

Andrew Gelman: If you cut it at around page 200, the first 200 pages are Bayesian workflow.

Andrew Gelman: And it's this crisp little book.

Andrew Gelman: And then the next 300 are the case studies, which is like a bunch of case studies.

Andrew Gelman: There's no point in making it two books because the case studies help you understand the workflow, blah, blah, blah.

Andrew Gelman: But if you think of it as a 200-page book, then I think we're really covering a lot and with a lot of open questions.

Andrew Gelman: At one point, we're going to have an appendix. We have some appendixes. One of the appendixes is how to, if you're not a Bayesian, what you can get out of this book, which we think there's a lot, like simulation's important, a lot of things. There's another appendix on how to get the most out of Bayesian data analysis, our earlier book, and what sections you can skip now because they're out of date.

Andrew Gelman: but we were thinking about adding a chapter an appendix a list of open problems but then we

Andrew Gelman: decided that like the book itself was like a list of open problems so it didn't seem to make sense

Alex Andorra: to do that yeah yeah no for sure so i need so two things i need a pair of scissors now i need to

Alex Andorra: to point one.

Alex Andorra: And second, yeah,

Alex Andorra: these appendices,

Alex Andorra: yeah, I really like that idea.

Alex Andorra: Especially if you're not a Bayesian one.

Alex Andorra: Although if you wanted to provoke,

Alex Andorra: you could just make it one sentence.

Alex Andorra: If you're not a Bayesian,

Alex Andorra: become one.

Alex Andorra: And then you're done.

Alex Andorra: But yeah, so we'll get back to that.

Alex Andorra: Keep that in mind.

Alex Andorra: I have still so many questions for you

Alex Andorra: about the book.

Alex Andorra: But first, we've got Dorota here.

Alex Andorra: So let's welcome her in.

Alex Andorra: Hi, Dorota.

Alex Andorra: You're joining live from Poland.

Alex Andorra: So first, welcome and congratulations.

Alex Andorra: You are the grand prize winner of the LBS contest.

Dorota Wojcik: Thanks a lot.

Alex Andorra: Yeah, no, it's really great to have you here.

Alex Andorra: So you'll get a signed copy of the book by the three authors here that you're seeing.

Alex Andorra: But today you're here because you're working on some really interesting stuff.

Alex Andorra: So, yeah, maybe first tell us a bit about yourself, you know, what you do,

Alex Andorra: how did you end up being interested in Bayesian stats,

Alex Andorra: and then we'll talk about your specific problem.

Dorota Wojcik: Okay, so I have a medical background and I was working as an assistant professor at medical university for about eight years, but I finished that and I started to work as data analyst, but I'm a junior.

Dorota Wojcik: and well

Dorota Wojcik: last year

Dorota Wojcik: I knew that I will just

Dorota Wojcik: resign from my previous job

Dorota Wojcik: and wanted to start

Dorota Wojcik: Bayesian statistic

Dorota Wojcik: course

Dorota Wojcik: and I took

Dorota Wojcik: statistical rethinking

Dorota Wojcik: course and

Dorota Wojcik: in November

Dorota Wojcik: probably I will take also another course

Dorota Wojcik: in Uppsala in Sweden

Dorota Wojcik: and

Dorota Wojcik: And I would like to thank guests for choosing my topic.

Dorota Wojcik: And I also can't wait to read the book because I'm waiting for the case studies.

Dorota Wojcik: And I would also like to thank Alexandre for the support because I have no experience with podcasts.

Dorota Wojcik: So I will continue with the topic.

Dorota Wojcik: So I'm building a Bayesian model of bat mortality from post-construction wind farms monitoring studies.

Dorota Wojcik: And this project grew out of my short placement at Forest Research Institute's Vertebrate Ecology Lab.

Dorota Wojcik: And I'm preparing the data set from these studies.

Dorota Wojcik: It's zero inflated.

Dorota Wojcik: And the challenge is that observed carcass counts reflect not only mortality, but also

Dorota Wojcik: the detection process, including carcass persistence, searcher efficiency, and also search design.

Dorota Wojcik: So when I observe zero carcass counts, I don't know whether it reflects low mortality,

Dorota Wojcik: low detectability, or both.

Dorota Wojcik: and many studies are missing parts of the detection process.

Dorota Wojcik: So I was thinking to model the detection hierarchically

Dorota Wojcik: to allow studies with complete detection data

Dorota Wojcik: to inform studies with missing detection data.

Dorota Wojcik: And I am still in an early stage of model development.

Dorota Wojcik: That's why I'm trying to understand

Dorota Wojcik: how much information can be shared across studies

Dorota Wojcik: without letting the model assumptions dominate the inference.

Dorota Wojcik: So my question is,

Dorota Wojcik: at what point does hierarchical pooling of detection parameters

Dorota Wojcik: stop being principled borrowing of information

Dorota Wojcik: and start becoming an assumption

Dorota Wojcik: that dominates the inference about mortality.

Alex Andorra: Fantastic.

Alex Andorra: Yeah, thanks, Dorota.

Alex Andorra: I think that was very clear.

Alex Andorra: So I'll just restate a bit

Alex Andorra: and then you guys can take it away.

Alex Andorra: But I really love that question

Alex Andorra: because I think it's super concrete,

Alex Andorra: but it's also very interesting.

Alex Andorra: So basically the question is like,

Alex Andorra: how many bats wind turbines kill

Alex Andorra: where you only observe carcasses that persist

Alex Andorra: and get found and fall in searchable ground.

Alex Andorra: So I really love that pyramid setup basically here

Alex Andorra: with non-missing randomness,

Alex Andorra: which I'm sure Richard, you're interested in.

Alex Andorra: And also the core question that you're asking

Alex Andorra: is a very concrete one that many people, I'm sure,

Alex Andorra: get into their own workflows.

Alex Andorra: It's at what point does hierarchical pooling

Alex Andorra: stop being principled borrowing of strength

Alex Andorra: and become an untestable assumption

Alex Andorra: that quietly dominates your inference?

Alex Andorra: And how would you even detect that?

Alex Andorra: So on that note,

Alex Andorra: who wants to take your first step at it?

Andrew Gelman: Let me throw some general statements out,

Andrew Gelman: and then I think maybe Richard and Aki

Andrew Gelman: will have more specifics.

Andrew Gelman: So first, I think that assumptions are important,

Andrew Gelman: and it's very rare to have a problem

Andrew Gelman: where assumptions don't matter at all.

Andrew Gelman: Like if you're in a situation

Andrew Gelman: where the true parameter

Andrew Gelman: could really be 10 to the 50th,

Andrew Gelman: like if that could be,

Andrew Gelman: then like your method

Andrew Gelman: would have to change to allow for that.

Andrew Gelman: And so in some way,

Andrew Gelman: I think a failure of some of my earlier work

Andrew Gelman: has been this idea that,

Andrew Gelman: oh, a hierarchical model

Andrew Gelman: is this innocuous thing

Andrew Gelman: and it will always work.

Andrew Gelman: But I think now that it is very valuable

Andrew Gelman: to have contextual information.

Andrew Gelman: That said, even a week prior can be useful.

Andrew Gelman: So to say I have a certain parameter

Andrew Gelman: that can be somewhere in some range,

Andrew Gelman: it's not going to be less than 0.01

Andrew Gelman: or if it is, it doesn't matter

Andrew Gelman: and it's not going to be more than 100 or 10.

Andrew Gelman: And that kind of week prior can be enough.

Andrew Gelman: And it is an assumption, though.

Andrew Gelman: I think that it would be that the idea of the purely data-based approach that will work for any value of the underlying parameter, there's only a very few problems for which that happens.

Andrew Gelman: And I think we have to move ourselves away from that particular expectation.

Andrew Gelman: The other thing is with hierarchical models in particular, there's something the epidemiologist Sander Greenland told me many years ago, like 20 years ago, I think. He said, well, he likes hierarchical models, but he'll just actually set the group level variance parameter to a fixed value based on his priors.

Andrew Gelman: And I naively said, but what about hierarchical modeling? We can get inference from the data. And he said, sure, but often you don't have a lot of groups. You don't get a very precise estimate of the group level variance from the data. You kind of think you know more than you do because you have this posterior, but often you have quite a bit of prior information too.

Andrew Gelman: So I think he would say the best thing to do would be to use an informative prior and do a hierarchical model. But he also said that in practice, often you have enough prior information about the group level variance that setting it to a fixed value won't be so bad.

Andrew Gelman: And related to that is that we often will have a model we fit to data.

Andrew Gelman: Then we fit the same model to another data set, the same model to another.

Andrew Gelman: This happens in political science all the time.

Andrew Gelman: You might want to fit a big time series model to all your data, but realistically, you might

Andrew Gelman: fit a separate model to each poll or each data set that you have.

Andrew Gelman: If your statistical inference for the hyperparameters is not well regularized, so you don't have

Andrew Gelman: a strong prior on it, then your inference for your hyperparameters will jump around

Andrew Gelman: from dataset to dataset.

Andrew Gelman: And this is something, I don't know if we really explained it so much in the book, but

Andrew Gelman: imagine you have data and you fit a hierarchical model and you have a fair amount of uncertainty

Andrew Gelman: about your group level variance parameter.

Andrew Gelman: And you're like, oh, that's okay.

Andrew Gelman: I'm Bayesian.

Andrew Gelman: I have a bit of uncertainty.

Andrew Gelman: I integrate out over it.

Andrew Gelman: I get inference for my parameters of interest.

Andrew Gelman: That's fine.

Andrew Gelman: But the thing that's funny we don't always think about is that uncertainty in one inference kind of maps to variation in repeated samples, new inferences from new data.

Andrew Gelman: So that means if I have this broad posterior for the group level variance parameter centered at a certain point, you might feel as if Bayesian is okay to average over it.

Andrew Gelman: But if you had a new data set from the same process, it would be moved over a lot.

Andrew Gelman: Like if your posterior standard deviation for this group of variance parameter is 10,

Andrew Gelman: then that also means that your point estimate, roughly speaking, could be 10 higher or 10

Andrew Gelman: lower in a new example.

Andrew Gelman: And so that will create inferences that jump a lot from one data set to another, which

Andrew Gelman: isn't really what you want.

Andrew Gelman: And so in that case, a strong prior would help on that. So I guess my very general messages are first, basically, yes, the prior is going to matter. I think you don't think that you're going to be in a situation where it's innocuous and the data tell you the answer.

Andrew Gelman: the data will tell you the answer but within a certain

Andrew Gelman: constrained space you know within a certain playing field

Alex Andorra: yeah fantastic thanks uh thanks Andrew maybe uh Aki you wanna you wanna chime in and then

Aki Vehtari: Richard okay um yeah this kind of data collection problem of that we don't know the effort

Aki Vehtari: and yeah they're also not the the carcass problem too but like this effort problem is common in

Aki Vehtari: ecology epidemiology clinical data where you kind of collect from the registry data and so on

Aki Vehtari: like the one possibility is that

Aki Vehtari: there is just no

Aki Vehtari: any information that can be used to then estimate

Aki Vehtari: the effort or this

Aki Vehtari: like disappearing carcasses and

Aki Vehtari: that's the kind of thing where you have least amount of information

Aki Vehtari: but you can still look how sensitive your results are to different assumptions that what if

Aki Vehtari: the effort varies in certain range so this like the detectability then the better possibility is

Aki Vehtari: that if for each site is it possible to somehow estimate the this detection detectability like

Aki Vehtari: effort plus the disappearing carcasses that then you then it goes to kind of similar models that

Aki Vehtari: Andrew has used these multi-regression post stratification that you can try

Aki Vehtari: like the seeing the similarity of different sites assuming that that similarity then

Aki Vehtari: connects to also similar effort and similar carcass detection rate

Alex Andorra: okay thank you is it my turn uh yes yes and then uh i i think there is also a

Alex Andorra: a hidden a hidden subtopic in in dartha's questions question that i want to ask aki

Alex Andorra: afterwards um because i think it's his uh his uh ballpark but please richard uh go away i think

Richard McElreath: you're gonna have a great feedback on that um all right i'll try to i'll try to be concise and

Richard McElreath: brief because i know there's more stuff to discuss here let me try a different tech is is just very

Richard McElreath: practical stuff with with modeling problems of the type you said you suggest dorota um i also i

Richard McElreath: collaborate with ecologists a lot i think you know on on these sorts of things is as a workflow

Richard McElreath: approach i always begin with a big generative simulation of the modeling problem that includes

Richard McElreath: the measurement process, which is, as you know, in ecology, it's the huge part of the modeling,

Richard McElreath: the simplest part of ecological modeling is the occupancy model. And it's already a complicated

Richard McElreath: model with all these problems in it. And so I always begin, before I even think about the

Richard McElreath: statistical model, just trying to simulate a data that has the structure. And then I talk through

Richard McElreath: with the experts, the assumptions of the simulation, and we work through the graphs

Richard McElreath: and try to figure it out, just as the forward workflow is the momentum, and get that in place.

Richard McElreath: And then take parts of the overall generative model and try to develop statistical models to estimate the quantities of interest

Richard McElreath: and approach this partial pooling problem that you've brought to us that way.

Richard McElreath: And I think I agree with Aki in the sense that there's scant information on some of the important aspects of the measurement problem, which is very common. So don't feel like you're alone in this. I think this is most of the applied ecologists I know have this kind of similar situation.

Richard McElreath: You want to do some kind of, you're going to end up with some kind of sensitivity analysis where you make different informative assumptions about in the statistical model and look at how much pooling that does and then compare that to the ground truth from your simulations.

Richard McElreath: So lots of profiling and calibration using synthetic data or fake data is sometimes called, but synthetic sounds nicer, doesn't it?

Richard McElreath: It sounds more scientific.

Richard McElreath: That's the workflow I use in these problems to help me forward.

Richard McElreath: And then the target ceases to be a single posterior, but a set of posteriors that give a range of estimates under different scenarios.

Richard McElreath: And because the core problem you've brought to us, I think, is a very common one is partial pooling is this incredible technology.

Richard McElreath: But sometimes you don't want the outliers to be shrunk in because their information, especially you're sort of on the bleeding edge of an applied problem like this.

Richard McElreath: It may be that the variance across units is something you don't want to shrink towards the mean because you want to use that for a follow up or something.

Richard McElreath: And so as part of the set of posteriors you conclude with would be ones that do very little partial pooling at all.

Richard McElreath: I think, Andrew, you call this the secret weapon or something like this, where you always compare the shrinkage estimates to the fixed effects estimates. And I think that's just a standard part of the workflow, or it should be, because even if you're going to use the partially pooled shrinkage estimates, you learn so much about the data and the phenomenon by seeing where the shrinkage has happened.

Richard McElreath: And if it's happening in places where you have other reasons to suspect that the detectability is of a particular type, then that'll start your mind going in ways that maybe you don't know where it'll go now.

Richard McElreath: But I think it's that thinking of the workflow and the models you explore as a way of asking questions about this complex phenomenon that is just not possible to study in detailed ways with measurements.

Richard McElreath: Because there are places, if I understood your question right, there are some sites where you can get estimates of detectability and recoverability or whatever word we want to use.

Richard McElreath: But there are other sites where there's nothing, right?

Richard McElreath: And so your zero inflation is haunted by this measurement problem.

Richard McElreath: and a set of models, a suite of models, maybe that's a better word, will help you explore how

Richard McElreath: sensitive inference is to that problem. If it turns out to be really sensitive to it,

Richard McElreath: then that's a headline result of your work. And it's not a bad answer because to the extent that

Richard McElreath: people want to take these estimates that you're delivering and develop policy around it or

Richard McElreath: something of that kind, if there's an applied issue, then saying that this is kind of a very

Richard McElreath: difficult measurement problem for this reason, maybe that suddenly conjures effort to do

Richard McElreath: measurement better in the future. So you're an active participant as an applied statistician

Richard McElreath: in how data gets collected in the future. But you need the suite of models to illustrate the

Richard McElreath: problem to the psychologically healthy people who don't do statistics for a living.

Richard McElreath: but anyway so that's the way I would approach it as a workflow and try to discover the issue

Richard McElreath: and and sense it out sorry I've got a cat who's here the famous one everyone's waiting for my cat

Richard McElreath: but anyway but it sounds like a great problem and yeah we could follow up on it more later you're

Richard McElreath: not so far from from me here at life sick i guess but uh you're my neighbor uh geographically

Richard McElreath: speaking but um yeah that's that's the workflow point uh place that i would go with it is to think

Richard McElreath: of it as trying to look at how sensitive these practical uh estimates are going to be to things

Richard McElreath: you can't directly measure is is the way i'd look at it okay so you understand the problem

Dorota Wojcik: correctly. And thank you very much. It's very valuable. And it's a lot information for me.

Alex Andorra: Anything you want to add, Dorota?

Dorota Wojcik: No, not really. No, I just, well, I'm honored that I could talk to you about this problem.

Dorota Wojcik: No, I don't have anything else.

Alex Andorra: Okay. Yeah. Feel free to follow up with me anyways, afterwards, you know, we're in contact.

Alex Andorra: Okay.

Alex Andorra: But I think before you leave, Aki, you wanted to add something?

Aki Vehtari: Yeah, like this bad measurements or missing measurements. So I'm right now working on

Aki Vehtari: cancer survival analysis. And one of the measurements is mitotic count. So

Aki Vehtari: So they take a biopsy and then they look through a microscope

Aki Vehtari: and count how many cell mitoses are going on, so the cell divisions.

Aki Vehtari: And for some reason they have used the kind of the visual field in microscope

Aki Vehtari: and then how many mitoses you see in that field.

Aki Vehtari: microscopes did get better

Aki Vehtari: so the visual field did get bigger and bigger

Aki Vehtari: and they still kept counting

Aki Vehtari: how many cell divisions you see

Aki Vehtari: in the field

Aki Vehtari: only very lately

Aki Vehtari: someone in some conference said that

Aki Vehtari: actually we should standardize this

Aki Vehtari: and now it's 5mm

Aki Vehtari: 5mm2

Aki Vehtari: But the problem is that we are now analyzing the data where like 80% of the observations are still with this field of view from different years.

Aki Vehtari: And because it's from different years, different sites, different countries, we have no idea what the measurement is, in how big field, how many mitoses.

Aki Vehtari: So it's very similar to your problem that you know it's zero, but you don't know why.

No.

Okay.

Dorota Wojcik: So thank you.

Alex Andorra: Fantastic.

Alex Andorra: Yeah.

Alex Andorra: Yeah.

Alex Andorra: Thank you so much, Torta.

Alex Andorra: I really want to thank you for taking the time and also having the courage to do that.

Alex Andorra: It was not an easy exercise.

Alex Andorra: It's like literally your first podcast.

Dorota Wojcik: you know I will start breathing after I will leave the room

Alex Andorra: no yeah that was um that was great of you to do it like literally your first podcast on a topic

Alex Andorra: you're learning um and presenting it live in a very short time frame to great minds like

Alex Andorra: Andrew, Aki and Richard this is uh it takes a lot of courage so yeah thank you thank you so much

Alex Andorra: for doing that

Alex Andorra: and we'll keep in touch.

Alex Andorra: Okay.

Dorota Wojcik: Bye-bye.

Alex Andorra: Bye.

Alex Andorra: Awesome.

Alex Andorra: Folks,

Alex Andorra: so yeah,

Alex Andorra: thank you so much

Alex Andorra: for doing that.

Alex Andorra: I think that was

Alex Andorra: a great segment.

Alex Andorra: Listeners,

Alex Andorra: let me know

Alex Andorra: how you like that.

Alex Andorra: If you like that,

Alex Andorra: I might do that

Alex Andorra: a bit more.

Alex Andorra: It takes a bit more

Alex Andorra: of preparation

Alex Andorra: for me

Alex Andorra: and anticipation,

Alex Andorra: mainly organization,

Alex Andorra: which is not

Alex Andorra: my strong suit.

Alex Andorra: But if that's

Alex Andorra: worth to you,

Alex Andorra: I'll do it.

Alex Andorra: I think it's a fun

Alex Andorra: It's a fun segment.

Alex Andorra: Aki, actually, I wanted to ask you about something that I think is related,

Alex Andorra: like kind of a subtopic of what Dorota asked about.

Alex Andorra: And it's this idea of trying to detect assumption dominance.

Alex Andorra: I'm really interested in that because I think it's also very concrete in a lot of models.

Alex Andorra: At least I work on, I'm sure a lot of people,

Alex Andorra: because there is always that issue somewhere you have, you know,

Alex Andorra: some clusters of some part of the domain of the model where you don't have enough data to inform

Alex Andorra: the model and so do you what do you recommend to do to detect assumption dominance here like

Alex Andorra: the tools i'm thinking about is sensitivity and calibration mainly but yeah i'm really

Aki Vehtari: curious to hear what you what you have to say on that um yeah at least the sensitivity analysis

Aki Vehtari: is that how much your inference on quantity of interest would change

Aki Vehtari: with different assumptions.

Aki Vehtari: And then the already prior sensitivity analysis is one of that.

Aki Vehtari: Richard mentioned, said that my package,

Aki Vehtari: but it's Noa Kallioinen who wrote the package.

Aki Vehtari: I know it's now continuing also on this data set sensitivity.

Aki Vehtari: Like it started to work in ecological problems where then the models are combining data from different sources.

Aki Vehtari: And then the interest is that how much these different data sources are actually affecting the inference.

Aki Vehtari: because so far they are just

Aki Vehtari: combine them everything but

Aki Vehtari: how to

Aki Vehtari: then to computationally

Aki Vehtari: efficient sensitivity

Aki Vehtari: analysis also with data

Aki Vehtari: but then it's the same

Aki Vehtari: whatever assumption

Aki Vehtari: is made

Aki Vehtari: or

Aki Vehtari: or any kind of this kind of

Aki Vehtari: suspect that there's

Aki Vehtari: something missing can be changed

Aki Vehtari: to be assumption

Aki Vehtari: and then we can just try.

Aki Vehtari: Richard would just simulate every time,

Aki Vehtari: change this assumption and simulate.

Aki Vehtari: But then if we don't want to simulate too many times,

Aki Vehtari: I guess these kind of the important sampling-based approaches

Aki Vehtari: would be a possibility.

Aki Vehtari: To look at the sensitivity and like local sensitivity

Aki Vehtari: kind of the gradient of what if we change something.

Yeah.

Yeah, I have to say also.

Alex Andorra: Yeah, go ahead, Richard.

Richard McElreath: I just said that sounds great.

Richard McElreath: That's a very routine part of workflow.

Richard McElreath: And having a more principled and efficient way to do it, that's really where the money is.

Andrew Gelman: Well, if you imagine just what it would look like if it's all working smoothly, imagine some display of your data, like a scatterplot or maybe a grid of plots.

Andrew Gelman: and then but let's say it's just a scatter plot of your data and your model might have

Andrew Gelman: some number of parameters in it you could have this scatter plot a bunch of graphs that look

Andrew Gelman: very similar it's the same scatter plot over and over again one for each parameter

Andrew Gelman: the same scatter plot but for each one you color the points based on how influential they are for

Andrew Gelman: inference for this particular parameter and then you'll find while some parameters are

Andrew Gelman: very sensitive. Certain data points are really telling you about this parameter,

Andrew Gelman: certain data telling you about that parameter. Perhaps a transformation can allow you to

Andrew Gelman: kind of more clearly isolate what matters. And then prior, that's just data too.

Andrew Gelman: So that's, but you'll have, it's something we don't fully understand. And we do discuss this

Andrew Gelman: in the book for sure.

Andrew Gelman: And, but, so, let's see.

Andrew Gelman: Like, if you have,

Andrew Gelman: well, sir, first simplicity.

Andrew Gelman: Suppose you have a model

Andrew Gelman: with two parameters

Andrew Gelman: and you just have two,

Andrew Gelman: like your parameters,

Andrew Gelman: theta one, theta two,

Andrew Gelman: and theta one's estimated

Andrew Gelman: from a bunch of data.

Andrew Gelman: Theta two is estimated

Andrew Gelman: from a bunch of other data.

Andrew Gelman: And you have a prior,

Andrew Gelman: a joint prior

Andrew Gelman: and theta one and theta two.

Andrew Gelman: Well, what's the influence

Andrew Gelman: of the prior?

Andrew Gelman: Well, in this case,

Andrew Gelman: it's a separate, if your priors are independent for the two parameters and the data are independent,

Andrew Gelman: then it's a separable model, meaning that the equivalent sample size of the prior would be

Andrew Gelman: different for theta one than for theta two. In fact, you could imagine you could have a situation

Andrew Gelman: where you have a very strong prior for one parameter and a very weak prior for another

Andrew Gelman: parameter and that that would map onto influence but if data one and data two are if the two

Andrew Gelman: parameters are entangled in the likelihood then getting inference about one parameter will give

Andrew Gelman: you automatically give inference about the other so for example if you have a regression and the

Andrew Gelman: echo with one predictor and the predictor is not centered at zero that if you get the there's a

Andrew Gelman: mapping between the slope of the regression line and the intercept. If the center is not at zero,

Andrew Gelman: then the intercept will be at the extreme. So if your data are positive, when your slope goes up,

Andrew Gelman: your intercept will go down. So now imagine you have a strong prior on the slope, but no

Andrew Gelman: information on the intercept. Well, that will still give you, that actually becomes informative

Andrew Gelman: about the intercept through this correlation or vice versa. So it's not always intuitive.

Andrew Gelman: And I think that's one reason why we have to think about ways of displaying and understanding it.

Alex Andorra: One of the recurring themes I got in the applications to the contest was how to communicate the results.

Alex Andorra: There were a lot of submissions about that.

Alex Andorra: And in particular, there was one from a listener who's called Philippe Calvet.

Alex Andorra: And basically his question was for people trying to bring Bayesian workflow into regulated industry, so something like credit risk, where the incumbent is a deterministic score everyone trusts, what actually works to get committees and regulators to accept a posterior distribution instead of a single number, even though it can be a comforting number?

Alex Andorra: is that a problem you you had in your your own workflows how do you recommend dealing with that

Alex Andorra: um is this to me should i answer this i i think go ahead yeah i thought you would chime in i have

Richard McElreath: opinions here i know my colleagues do too but uh i think i think for for pure inference projects

Richard McElreath: which is different than the problem you're giving me but it's a place to start so if like a pure

Richard McElreath: inference project, I often think that posterior distributions are not the right summary,

Richard McElreath: or at least the posterior predictive is a better summary. It's something, not the posterior

Richard McElreath: marginal distribution of some parameter. It's like, I think of parameters as these

Richard McElreath: gears in the machine, and they cooperate to produce predictions and scenarios. And so,

Richard McElreath: from a decision maker's perspective on the end, someone who's not a statistician,

Richard McElreath: interpreting a parameter is very difficult because the scale may be all wrong. And,

Richard McElreath: you know, if you've learned that the posterior distribution has a mean of 1.2, what does that

Richard McElreath: mean on the outcome scale? You've got some link function and it all gets transformed. So having

Richard McElreath: summaries on the outcome scale, some posterior predictive, and it could be a posterior predictive

Richard McElreath: contrast for causal effects or something, it would be my default that I recommend to people.

Richard McElreath: And this is what I present to my course increasingly often.

Richard McElreath: But for what you're saying, it sounds like, you know, you're making a presentation to people who are used to making decisions based upon just numbers, just like this is the estimate and there's no uncertainty.

Richard McElreath: It's just a deterministic score.

Richard McElreath: In that case, I'd go all the way into decision analysis probably and talk about costs and benefits of the different decisions that they could make based upon the existing information.

Richard McElreath: I could say like, and that, cause that's, that's a world that in my experience, that's a world that those folks live in. They, they think in terms of the costs and benefits of taking action and often postponing action is not possible. So some decisions going to be made, whatever information you give them. So why not meet them where they live and talk about consequences? Yeah. And that's what I would often do.

Richard McElreath: I come from, I didn't mention this to me anyway, I come from a military family. I'm the only member of my family to go to college. And I'm surrounded in the McElreath clan is a bunch of professional military people. So when I talk to them, that's the way that they feel about everything, this stuff. They're like, well, we have to do stuff like all the time. And everything is like lives saved, lives lost. And that's where these people live.

Richard McElreath: And whereas, you know, scientists like me, like I study human evolution, like, you know, no one's going to live or die based upon the estimate of how old the Neanderthal is. But so, yeah, your friend did or your colleague who lives in the real world, I think, yeah, the costs and benefits, some utility function and you integrate over the posterior.

Richard McElreath: So Bayes, I want to say, Bayes still has huge advantages there. Bayesian decision theory is the business. And our colleague Christian Robert, this is his expertise, isn't it, Andrew? And he's got that great book on Bayes that has a bunch of decision theory stuff in it. And that's what I would recommend.

Andrew Gelman: We have, yeah, chapter nine of BDA three, we have some real world decision theory examples that I really like those. It was hard to find because decision theory textbooks typically have fake examples. So we put in some effort for that.

Andrew Gelman: I will say there is an opposite mistake that people make, which is people who are doing scientific inference who are trying to be more hard-nosed than they really are. And they're like, well, you want to do all this Bayesian stuff, but we want to make decisions. We're decision makers. You have to be realistic. They need to know what drug to approve. We need to have p-values because we have to make...

Andrew Gelman: Because we're always saying you have to express your uncertainty.

Andrew Gelman: So don't say that because your summary statistic is statistically significant that it's real or that's not real, you know, like that.

Andrew Gelman: We're always talking about uncertainty.

Andrew Gelman: You get these people who are not actually decision makers.

Andrew Gelman: They're academics who will kind of scold us and say they're trying to help people who have to make real decisions and we're not helping by emphasizing uncertainty.

Andrew Gelman: I...

Andrew Gelman: As you could tell by how I just spoke, I don't agree with that attitude. I think that decision makers don't make decisions entirely based on a statistical summary.

Andrew Gelman: That's true.

Andrew Gelman: But it's not true that if you do an experiment and you get a result that's two standard errors away from zero, that a good journal will necessarily publish it.

Andrew Gelman: You have to have a lot of theory and explanation.

Andrew Gelman: It's not true that a drug will be approved just because blah, blah, blah.

Andrew Gelman: So I do feel that our job as statisticians, unless we're specifically asked to give a decision recommendation, I think our job typically is not to make a decision for somebody. Our job is to give people a sense of the uncertainty and also a sense of what more could be learned that could change things.

Andrew Gelman: so one thing i like to say is a probability is not just a number it's part of an it exists in

Andrew Gelman: a network of conditional statements um and the example i will always love to give is either

Andrew Gelman: you're flipping a coin and it's head or tails or the world's greatest boxer is fighting the

Andrew Gelman: world's greatest wrestler to the death and who wins well i don't know it's kind of 50 50

Andrew Gelman: how are those problems different one is that like there's really nothing i can do with the coin

Andrew Gelman: before flipping it that will give me information about that it's 50 50 no matter what that's a

Andrew Gelman: very hard 50 50 there's nothing much i can do about it the boxer and wrestler i could learn a

Andrew Gelman: lot by having practice fights and things like that i could gather data so often one of the

Andrew Gelman: roles of the statistician is to say it's 50 50 or whatever but another role is to say what data

Andrew Gelman: could be gathered that could change things. And that can upset people. I mean, people don't

Andrew Gelman: like it. People were annoyed at me for election forecasts when we said the election was,

Andrew Gelman: that the presidential election was roughly equally likely to go either way. They felt that we're

Andrew Gelman: supplying no information. And we had to explain that saying that it's roughly equally likely to

Andrew Gelman: go either way is information because not every election is so close, but it's not intuitive,

Andrew Gelman: right people often will look at the outcome and not look at the process which i guess is kind of

Andrew Gelman: big point of bayesian workflow that the process is as part of it too yeah yeah 100 percent and

Alex Andorra: in in my experience also it depends a lot on who the decision makers are in front of you um

Alex Andorra: like i've definitely worked with complicated decision makers who didn't like nuances but i've

Alex Andorra: also worked with decision makers who really were able to think in scenarios and probabilistic

Alex Andorra: thinking mostly and these make my job the easiest because usually the way I report so to answer also

Alex Andorra: to Philip I usually never report just one number even if they ask me for that I'm like oh okay I

Alex Andorra: forgot I always report at least three scenarios and I and I frame them as scenarios and I take

Alex Andorra: them in the different distribution of the posterior distribution different point and usually have like

Alex Andorra: a you know median scenario optimistic one and pessimistic one and that's usually how i report

Alex Andorra: these kind of analyses and then that usually generates a lot of discussion about okay so how

Alex Andorra: can we make the median scenario actually become the the optimistic scenario actually the median one

Alex Andorra: and that relates to what you were talking about, Andrew,

Alex Andorra: about what would it take,

Alex Andorra: what would we need to get to that scenario?

Alex Andorra: So, yeah, in my experience,

Alex Andorra: what will maximize your ability to convey the fact

Alex Andorra: that you don't want to give fake certainty

Alex Andorra: by just reporting one number is to report scenarios

Alex Andorra: and not talk about distributions at all.

Alex Andorra: Nobody cares about, oh, that's my posterior distribution,

Alex Andorra: I use standard

Alex Andorra: to get that

Alex Andorra: nobody cares

Alex Andorra: about that

Alex Andorra: unless they

Alex Andorra: ask you about

Alex Andorra: it then it's

Alex Andorra: great you've

Alex Andorra: got a great

Alex Andorra: person in front

Alex Andorra: of you but

Alex Andorra: usually it's

Alex Andorra: just give me

Alex Andorra: the results

Alex Andorra: that works

Alex Andorra: quite well

Alex Andorra: Aki do you

Alex Andorra: have anything

Alex Andorra: to chime in

Alex Andorra: on that or

Alex Andorra: shall I

Alex Andorra: get to another

Alex Andorra: topic

Aki Vehtari: at least you

Aki Vehtari: should remind

Aki Vehtari: me about the

Aki Vehtari: question

Alex Andorra: yeah that

Alex Andorra: was mainly

Alex Andorra: about how

Alex Andorra: do you

Alex Andorra: communicate

Alex Andorra: uncertainty to decision makers

Alex Andorra: that was mostly

Alex Andorra: that was the gist of it and I think the cat

Alex Andorra: really likes that conversation because

Alex Andorra: it's still here

Alex Andorra: yeah go ahead

Aki Vehtari: one case was this

Aki Vehtari: first time I was making

Aki Vehtari: this country survival analysis

Aki Vehtari: and then

Aki Vehtari: like their

Aki Vehtari: important part

Aki Vehtari: was also that it had to be

Aki Vehtari: like the model

Aki Vehtari: the prediction result had to be

Aki Vehtari: something that can be printed

Aki Vehtari: like I was told

Aki Vehtari: like it was like about

Aki Vehtari: 10 years ago that for cancer

Aki Vehtari: doctors it has to be printed

Aki Vehtari: on a paper so it was

Aki Vehtari: this kind of the

Aki Vehtari: maps where you can

Aki Vehtari: choose whether there's

Aki Vehtari: been rupture

Aki Vehtari: what is the location of the tumor

Aki Vehtari: and then you read from

Aki Vehtari: x and y axes the

Aki Vehtari: tumor size and mid-duty count

Aki Vehtari: and then you see the color

Aki Vehtari: telling the probability

Aki Vehtari: and this

Aki Vehtari: I was told that

Aki Vehtari: it was

Aki Vehtari: success

Aki Vehtari: but in a way that like also

Aki Vehtari: like simplifying there

Aki Vehtari: that it's easy to read that there were

Aki Vehtari: only like five levels of

Aki Vehtari: these probabilities told

Aki Vehtari: and I'm quite sure

Aki Vehtari: that these

Aki Vehtari: cancer doctors were also including other information

Aki Vehtari: they had and like they used it for

Aki Vehtari: decision making or providing recommendation for a patient that

Aki Vehtari: whether they should then start

Aki Vehtari: eating medication which has also then

Aki Vehtari: nasty side effects so

Aki Vehtari: at least the important part is that the

Aki Vehtari: result needs to be somehow easy to use.

Richard McElreath: Alex

Richard McElreath: But I know you have more questions, Alex, but this is,

Richard McElreath: this is such a huge area that's sort of underdeveloped

Richard McElreath: sometimes. Yeah.

Alex Andorra: No, no. I mean, if you if you really

Alex Andorra: want to say something about that,

Richard McElreath: I'm curious what your next question is all these

Richard McElreath: questions are.

Alex Andorra: Okay, yeah. Yeah. So you guys still have time for one more

Alex Andorra: question before I ask Richard the last two questions or does

Alex Andorra: somebody need to drop very soon?

Richard McElreath: Richard Streitzel, No, I have food being delivered, but I

Richard McElreath: think it'll be delivered after we're done.

Alex Andorra: Paul Cutler, okay, cool. Cool. Yeah. And I mean, and the cat

Alex Andorra: can do that for probably

Richard McElreath: Richard Streitzel, yeah, the cat can check the door and yeah, he

Richard McElreath: speaks German. It's perfectly

Alex Andorra: Paul Cutler, yeah, I mean, I can see that. Yeah. He's

Alex Andorra: between being greeting people from the window. So yeah. So

Alex Andorra: Yeah, I have actually a fun mix of causal inference and modeling question that I got from another listener, Ian Costley.

Alex Andorra: And yeah, I really like that one.

Alex Andorra: It's a bit of a, you know, it's like it's a thinker, but I think you guys will have interesting things to say.

Alex Andorra: So here it is.

Alex Andorra: Imagine you send out a promotional mailer, and what you really want to know is whether redeeming the coupon causes people to spend more.

Alex Andorra: So one of the problems is that only a tiny slice of people redeem, so you cannot just compare redeemers to everyone else, also because they are different kinds of people.

Alex Andorra: so usually the way the the trick that's used for this kind of analysis is just to

Alex Andorra: lean on who received the mailer as a stand-in since that part is random but

Alex Andorra: this trick only works if receiving the mailer affects spending only through redemption which

Alex Andorra: is probably false here because just seeing the offer in the mailbox might nudge someone to buy

Alex Andorra: So that's kind of a contrived example,

Alex Andorra: but I think it speaks to a lot of different analyses,

Alex Andorra: especially causal analysis,

Alex Andorra: when you want to understand the effect of an intervention, basically.

Alex Andorra: And so when you know,

Alex Andorra: the question is when you know your clean liter shortcut like that is leaking,

Alex Andorra: can you build the leak into the model

Alex Andorra: and let it tell you how much that leak could be distorting your answer

Alex Andorra: instead of just pretending the leak is not there.

Andrew Gelman: Yeah.

Andrew Gelman: Yeah, people have done that.

Andrew Gelman: They call it the principal stratification model.

Andrew Gelman: So there is some, I think like Avi Feller

Andrew Gelman: and some other people have done some

Andrew Gelman: Bayesian principles stratification models in STAN.

Andrew Gelman: And the idea is that you would characterize people

Andrew Gelman: that there's a latent variable.

Andrew Gelman: Well, it's observed for the people who got the miler and not available, not observed for the others.

Andrew Gelman: And it's whether you would have.

Andrew Gelman: Well, I can't remember how it works, but like the people who.

Andrew Gelman: The latent variable is if you receive the stage one treatment, would you do the stage two treatment?

Andrew Gelman: And for the people who did receive the stage one treatment, you know that.

Andrew Gelman: But the people who didn't, you don't know that.

Andrew Gelman: But you can form a model.

Andrew Gelman: And you can do inference on that. You'd want to use pre-treatment predictors. You'd want to use characteristics of the people in the study, their age and where they live and whatever spending patterns they have in the past.

Andrew Gelman: and then you can do that.

Andrew Gelman: You should be able to do that model.

Andrew Gelman: And that will,

Andrew Gelman: then you can then, from that,

Andrew Gelman: you can estimate the effect of the,

Andrew Gelman: you can compare the people

Andrew Gelman: who would or would not

Andrew Gelman: do the stage two treatment

Andrew Gelman: and you can estimate what would happen

Andrew Gelman: if someone who didn't do it

Andrew Gelman: were to do it.

Andrew Gelman: But basically you have to,

Andrew Gelman: you'd model the whole process.

Andrew Gelman: I haven't done this myself, but I know people have done it. It seems like a very natural Bayesian problem. And I think that standard solutions such as instrumental variables would correspond to special cases of this model, assuming various things are zero or with flat priors.

Andrew Gelman: Usually standard procedures usually correspond to some mix of some flat priors and some parameters and priors with spikes at zero on others.

Alex Andorra: Okay.

Alex Andorra: Yeah.

Alex Andorra: And so, Richard, I'm sure you want to say something on that.

Richard McElreath: Well, no, I mean, that sounds reasonable to me.

Richard McElreath: I recognize this as the like per protocol or intent to treat kind of problem.

Richard McElreath: And there's, I mean, that's a big literature.

Richard McElreath: and there's what Andrew says sounds like it's very sensible there you you try to do the best you can

Richard McElreath: but you you got to be careful about what you're adding because now you have it's this downstream

Richard McElreath: from treatment problem but people have written a lot about this but this is still yeah the per

Richard McElreath: protocol intent to treat sort of issue if you want to figure out mechanism how the treatment's really

Richard McElreath: working it's it's hard even if the treatment's randomized but there are ways to do it as Andrew

Richard McElreath: has discussed and that's that's the way I approach it yeah awesome yeah I love these problems like I

Alex Andorra: I think these kind of hard causal inference problems are among the most interesting ones

Alex Andorra: because also that's where you get to do the most customized models.

Alex Andorra: And these are the fun ones where you have to,

Alex Andorra: these are kind of Lego bricks that you want to add together.

Alex Andorra: So I love working on these kind of models.

Alex Andorra: Actually, so I'm going to start winding this down here.

Alex Andorra: because it's not getting late for Aki and Richard.

Alex Andorra: I still have so many questions for you guys,

Alex Andorra: but maybe, so one is,

Alex Andorra: is there something you wanted to mention here

Alex Andorra: that we didn't get to today?

Richard McElreath: Well, that's a very open question.

Richard McElreath: Yeah, maybe something that's not in the book yet.

Richard McElreath: We didn't talk about AI and workflow, and I think that's a very big and interesting topic.

Richard McElreath: Maybe you'll have to have us back and we can talk about that.

Richard McElreath: But I think that's a big area, and it's very dynamic, and it's just here.

Richard McElreath: Anyway, but we don't have enough time before my food is delivered to discuss this.

Richard McElreath: I know, I know.

Alex Andorra: I am very frustrated.

Alex Andorra: and I definitely want to get your thoughts on that

Alex Andorra: and also what you're seeing, how you're using that.

Alex Andorra: I know I've been also, of course, doing a lot of that myself,

Alex Andorra: working on some agent skills with some people like Stefan Radev,

Alex Andorra: if you know, but they have been on the show,

Alex Andorra: and a lot of great people doing that.

Alex Andorra: So, yeah, I'm very interested in that also and seeing how it goes.

Andrew Gelman: I want to get Jessica, Aki, and Bob on together to watch them scream at each other.

Andrew Gelman: Sounds like fun.

Andrew Gelman: Yeah.

Alex Andorra: Let's organize that.

Alex Andorra: I'll bring popcorn.

Alex Andorra: That would be great.

Alex Andorra: Yeah.

Alex Andorra: And Richard, I need to have you anyways for a solo episode on the show.

Alex Andorra: We'll dive on these questions.

Alex Andorra: And Aki, maybe do you want to talk a bit about that?

Alex Andorra: because I know you guys do a lot of that at Aalto also alone.

Aki Vehtari: Do what? Like AI?

Alex Andorra: Yeah, so I know you're working on that a lot with your group.

Alex Andorra: Maybe a more pointed question would be,

Alex Andorra: yeah, what's on your mind for the coming month?

Alex Andorra: What are the things you're looking forward to

Alex Andorra: that you're working on that you're curious about?

Aki Vehtari: So overall, this goes back to

Aki Vehtari: like we've been working on the book like six years and it has a lot of also like in the case studies

Aki Vehtari: these software packages and diagnostics that I've been developing and this work continues and

Aki Vehtari: like it was just during the spring

Aki Vehtari: in the last moments adding

Aki Vehtari: like the corrected loop it

Aki Vehtari: plot approaches there

Aki Vehtari: and there what's not in the book

Aki Vehtari: but will be then in case studies

Aki Vehtari: is now the loop package

Aki Vehtari: having these diagnostics

Aki Vehtari: diagnostics when we can trust the unsynthetic quantification in

Aki Vehtari: loop comparison and these kind of things all the time coming more and it's kind of a bit

Aki Vehtari: unknowing that the book is actually printed. I can't edit the printed versions even if we can

Aki Vehtari: like the electronic version can be updated and case studies can update it and we can add

Aki Vehtari: additional case studies

Aki Vehtari: and of course like this is also connected

Aki Vehtari: to

Aki Vehtari: the question about the AI

Aki Vehtari: that we know

Aki Vehtari: agents work best when they

Aki Vehtari: have tools

Aki Vehtari: so that the workflow is not just based

Aki Vehtari: on large language

Aki Vehtari: model but

Aki Vehtari: actually calling tools and

Aki Vehtari: tools that we trust

Aki Vehtari: and then I think that

Aki Vehtari: that

Aki Vehtari: is going to keep me busy

Aki Vehtari: for many years.

Aki Vehtari: Like AI is not replacing me.

Aki Vehtari: I'm just then building tools for AI.

Yeah.

Alex Andorra: Yeah, I have to say

Alex Andorra: what you guys are doing at Aalto

Alex Andorra: is super exciting.

Alex Andorra: I will refer people to

Alex Andorra: some episodes I've done recently

Alex Andorra: with Luigia Sherby

Alex Andorra: Jonas Arruda

Alex Andorra: Stefan Radev

Alex Andorra: a lot of people who've been working on that

Alex Andorra: Osvaldo Martin probably needs to be back

Alex Andorra: on the show I know he's been doing a lot of

Alex Andorra: amazing stuff on the

Alex Andorra: ArviZ side

Alex Andorra: I definitely recommend people using ArviZ to check out

Alex Andorra: the new version 1.0

Alex Andorra: there is so many new

Alex Andorra: things in there especially calibration

Alex Andorra: plots I love them

Alex Andorra: use them all the time

Alex Andorra: and the p-sense

Alex Andorra: also the prior sensitivity

Alex Andorra: features

Alex Andorra: extremely useful to

Alex Andorra: do the kind of things we've been

Alex Andorra: talking about a bit earlier in the show

Alex Andorra: but in a more

Alex Andorra: automated and

Alex Andorra: automated way with guardrails

Alex Andorra: so yeah all this

Alex Andorra: research is extremely exciting

Alex Andorra: and very helpful

Aki Vehtari: it's been great for putting things on Python's side

Alex Andorra: yeah

Alex Andorra: yeah so

Alex Andorra: definitely we

Alex Andorra: we need probably to

Alex Andorra: do another episode

Alex Andorra: about that

Alex Andorra: Andrew anything you want to add on that

Andrew Gelman: I'm wondering

Andrew Gelman: do you think we should have

Andrew Gelman: a web page

Andrew Gelman: like a version

Andrew Gelman: of our web page in Python

Andrew Gelman: because a lot of people are really

Andrew Gelman: it's not always working on it

Andrew Gelman: oh okay

Andrew Gelman: we went

Andrew Gelman: through because it's even

Andrew Gelman: I mean, I think it's kind of funny where you draw the line, because in some sense, a Bayesian workflow isn't limited to Stan either. But I think I feel I might be wrong. I think people recognize, think even people who don't use Stan can think of it as a kind of universal language.

Andrew Gelman: So if you're using some hyper-efficient parallel processing probabilistic programming language, there's still logic to having the code in stand, and you should be able to do that.

Andrew Gelman: But yeah, I think one of the—

Aki Vehtari: Osvaldo is working on the Python version.

Andrew Gelman: Yeah.

Aki Vehtari: No, that's exciting.

Aki Vehtari: He's using also his work time for that, so he's getting even paid for that.

Alex Andorra: Great.

Alex Andorra: Fantastic.

Alex Andorra: Yeah.

Alex Andorra: Damn.

Alex Andorra: Yeah, thank you, Osvaldo.

Alex Andorra: Always doing great stuff.

Alex Andorra: Very, very helpful for everybody.

Alex Andorra: Fantastic.

Alex Andorra: Well, guys, I'm going to let you go because it's been a long time.

Alex Andorra: I know it's late for you, Akin, Richard.

Alex Andorra: So let's call this a show.

Alex Andorra: We could do a four-hour episode.

Alex Andorra: So I need to draw the line at some point.

Alex Andorra: But, Richard, before you go, last two questions.

Alex Andorra: Ask everybody at the end of the episode.

Alex Andorra: Akin and Andrew already answered that.

Alex Andorra: But first question, if you had unlimited time and resources, which problem would you try to solve?

Alex Andorra: Unlimited time and resources, which problem would I try to solve?

Alex Andorra: It can be anything. It doesn't have to be statistics related.

Richard McElreath: Yeah, well, I'll make it statistics related just for the sake of the listeners.

Richard McElreath: But I think the problem I would try to solve is to move forward on what I think of as the grand project in statistics is to make it increasingly formalized in the sense of mathematical proofs.

Richard McElreath: There's this sense in which, of course, there's this old tradition Aki knows very well, computer science, of thinking about algorithms as proofs.

Richard McElreath: But it hasn't been applied very much to the way we develop statistical workflow.

Richard McElreath: And I think there's a lot of homologies there in thinking about methods or algorithms for deriving minimal assumptions that justify procedures.

Richard McElreath: And so that we can really list, you know, make things axiomatic in a sense.

Richard McElreath: And I think that's been extremely powerful in mathematics and applied mathematics, and it would be equally powerful in statistics.

Richard McElreath: But it's really a project that only got going in around 1980, I think, in a serious way, even though the foundations have been there for a long time.

Richard McElreath: So I think that's a really important thing.

Richard McElreath: And now that said, I don't think I'm going to make any important contributions to that because I have a job and I do a bunch of administration and I can't.

Richard McElreath: But you asked, not hypothetical.

Richard McElreath: But I think that's a big, like, multi-generation project for statistics is making things more axiomatic and finding ways to derive licensing assumptions, minimal sets of licensing assumptions.

Richard McElreath: Okay, next question.

Alex Andorra: Yeah, love that.

Alex Andorra: Yeah, so second question, if you could have dinner with any great scientific mind, dead, alive, or fictional, who would it be?

Richard McElreath: Oh, that's a tough one.

Richard McElreath: I mean, no tour.

Richard McElreath: I think that would be great.

Alex Andorra: Yeah.

Alex Andorra: Okay.

Alex Andorra: Brilliant.

Alex Andorra: Yeah.

Alex Andorra: You're the first one to answer that.

Alex Andorra: So that's great.

Alex Andorra: Yeah.

Alex Andorra: Fantastic.

Alex Andorra: Well, gentlemen, thank you so much.

Alex Andorra: It's very early for me in the day, but I have to say this is one of the best ways to start Wednesdays.

Alex Andorra: It was amazing to have you here, all three.

Alex Andorra: You're always welcome back.

Alex Andorra: Again, thank you so much for taking the time and being on this show.

Alex Andorra: Yeah.

Alex Andorra: Thank you, Alex.

Andrew Gelman: Yeah, see you.

Alex Andorra: This has been another episode of Learning Bayesian Statistics.

Alex Andorra: Be sure to rate, review, and follow the show on your favorite podcatcher

Alex Andorra: and visit learnbayesstats.com for more resources about today's topics

Alex Andorra: as well as access to more episodes to help you reach true Bayesian state of mind.

Alex Andorra: That's learnbayesstats.com.

Alex Andorra: Our theme music is Good Bayesian by Baba Brinkman.

Alex Andorra: fit MC Lars and Mega Ran.

Alex Andorra: Check out his awesome work at bababrinkman.com.

Alex Andorra: I'm your host, Alex Andorra.

Alex Andorra: You can follow me on Twitter at Alex underscore Andorra,

Alex Andorra: like the country.

Alex Andorra: You can support the show and unlock exclusive benefits

Alex Andorra: by visiting patreon.com slash learnbayesstats.

Alex Andorra: Thank you so much for listening and for your support.

Alex Andorra: You're truly a good Bayesian

and change your predictions after taking information.

And if you're thinking I'll be less than amazing

Let's adjust those expectations

Let me show you how to be a good Bayesian

Change calculations after taking fresh data

And those predictions that your brain is making

Let's get them on a solid foundation