Proudly sponsored by PyMC Labs, the Bayesian Consultancy. Book a call, or get in touch!
Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work!
Visit our Patreon page to unlock exclusive Bayesian swag 😉
Takeaways:
- Designing experiments is about optimal data gathering.
- The optimal design maximizes the amount of information.
- The best experiment reduces uncertainty the most.
- Computational challenges limit the feasibility of BED in practice.
- Amortized Bayesian inference can speed up computations.
- A good underlying model is crucial for effective BED.
- Adaptive experiments are more complex than static ones.
- The future of BED is promising with advancements in AI.
Chapters:
00:00 Introduction to Bayesian Experimental Design
07:51 Understanding Bayesian Experimental Design
19:58 Computational Challenges in Bayesian Experimental Design
28:47 Innovations in Bayesian Experimental Design
40:43 Practical Applications of Bayesian Experimental Design
52:12 Future of Bayesian Experimental Design
01:01:17 Real-World Applications and Impact
Thank you to my Patrons for making this episode possible!
Yusuke Saito, Avi Bryant, Ero Carrera, Giuliano Cruz, Tim Gasser, James Wade, Tradd Salvo, William Benton, James Ahloy, Robin Taylor,, Chad Scherrer, Zwelithini Tunyiswa, Bertrand Wilden, James Thompson, Stephen Oates, Gian Luca Di Tanna, Jack Wells, Matthew Maldonado, Ian Costley, Ally Salim, Larry Gill, Ian Moran, Paul Oreto, Colin Caprani, Colin Carroll, Nathaniel Burbank, Michael Osthege, Rémi Louf, Clive Edelsten, Henri Wallen, Hugo Botha, Vinh Nguyen, Marcin Elantkowski, Adam C. Smith, Will Kurt, Andrew Moskowitz, Hector Munoz, Marco Gorelli, Simon Kessell, Bradley Rode, Patrick Kelley, Rick Anderson, Casper de Bruin, Philippe Labonde, Michael Hankin, Cameron Smith, Tomáš Frýda, Ryan Wesslen, Andreas Netti, Riley King, Yoshiyuki Hamajima, Sven De Maeyer, Michael DeCrescenzo, Fergal M, Mason Yahr, Naoya Kanai, Steven Rowland, Aubrey Clayton, Jeannine Sue, Omri Har Shemesh, Scott Anthony Robson, Robert Yolken, Or Duek, Pavel Dusek, Paul Cox, Andreas Kröpelin, Raphaël R, Nicolas Rode, Gabriel Stechschulte, Arkady, Kurt TeKolste, Gergely Juhasz, Marcus Nölke, Maggi Mackintosh, Grant Pezzolesi, Avram Aelony, Joshua Meehl, Javier Sabio, Kristian Higgins, Alex Jones, Gregorio Aguilar, Matt Rosinski, Bart Trudeau, Luis Fonseca, Dante Gates, Matt Niccolls, Maksim Kuznecov, Michael Thomas, Luke Gorrie, Cory Kiser, Julio, Edvin Saveljev, Frederick Ayala, Jeffrey Powell, Gal Kampel, Adan Romero, Will Geary, Blake Walters, Jonathan Morgan, Francesco Madrisotti, Ivy Huang and Gary Clarke.
Links from the show:
- Come see the show live at PyData NYC: https://pydata.org/nyc2024/
- Desi’s website: https://desirivanova.com/
- Desi on GitHub: https://github.com/desi-ivanova
- Desi on Google Scholar: https://scholar.google.com/citations?user=AmX6sMIAAAAJ&hl=en
- Desi on Linkedin: https://www.linkedin.com/in/dr-ivanova/
- Desi on Twitter: https://x.com/desirivanova
- LBS #34, Multilevel Regression, Post-stratification & Missing Data, with Lauren Kennedy: https://learnbayesstats.com/episode/34-multilevel-regression-post-stratification-missing-data-lauren-kennedy/
- LBS #35, The Past, Present & Future of BRMS, with Paul Bürkner: https://learnbayesstats.com/episode/35-past-present-future-brms-paul-burkner/
- LBS #45, Biostats & Clinical Trial Design, with Frank Harrell:https://learnbayesstats.com/episode/45-biostats-clinical-trial-design-frank-harrell/
- LBS #107, Amortized Bayesian Inference with Deep Neural Networks, with Marvin Schmitt: https://learnbayesstats.com/episode/107-amortized-bayesian-inference-deep-neural-networks-marvin-schmitt/
- Bayesian Experimental Design (BED) with BayesFlow and PyTorch: https://github.com/stefanradev93/BayesFlow/blob/dev/examples/michaelis_menten_BED_tutorial.ipynb
- Paper – Modern Bayesian Experimental Design: https://arxiv.org/abs/2302.14545
- Paper – Optimal experimental design; Formulations and computations: https://arxiv.org/pdf/2407.16212
- Information theory, inference and learning algorithms, by the great late Sir David MacKay: https://www.inference.org.uk/itprnn/book.pdf
- Patterns, Predictions and Actions, Moritz Hard and Ben Recht https://mlstory.org/index.html
Transcript
This is an automatic transcript and may therefore contain errors. Please get in touch if you’re willing to correct them.
Transcript
Today I am delighted to host Desi Ivanova, a distinguished research fellow in machine
learning at the University of Oxford.
2
Desi's fascinating journey in statistics has spanned from quantitatifiance to the
frontiers of Bayesian experimental design, or BED, BED.
3
In our conversation, Desi dives into the deep
4
world of BED where she has made significant contributions.
5
She begins by elucidating the core principles of experimental design, discussing both the
theoretical underpinnings and the complex computational challenges that arise in its
6
application.
7
Desi shares insights into the innovative solutions she's developed to make BED more
practical and applicable in real-world scenarios, particularly
8
highlighting its impact in sectors like healthcare and technology.
9
Throughout the discussion, Desi also touches on the exciting future of BED, especially in
light of recent advancements in AI and machine learning.
10
She reflects on the critical role of real-time decision-making in today's data-driven
landscape and how patient methods can enhance the speed and accuracy of such decisions.
11
This is Learning Vision Statistics, episode 117.
12
recorded September 26, 2024.
13
Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods,
the projects and the people who make it possible.
14
I'm your host.
15
Alex and Dora.
16
You can follow me on Twitter at Alex underscore and Dora like the country for any info
about the show.
17
Learnbasedats.com is Laplace to be.
18
Show notes, becoming a corporate sponsor, unlocking Beijing Merch, supporting the show on
Patreon.
19
Everything is in there.
20
That's Learnbasedats.com.
21
If you're interested in one-on-one mentorship, online courses or statistical consulting,
feel free to reach out and book a call at topmate.io slash Alex underscore and Dora.
22
See you around, folks, and best patient wishes to you all.
23
And if today's discussion sparked ideas for your business, well, our team at PMC Labs can
help bring them to life.
24
Check us out at PMC-labs.com.
25
Hello my dear vegans, today I want to welcome our two new patrons in the full posterior
tier.
26
Thank you so much Ivy Hwing and Garrett Clark, your support literally makes this show
possible.
27
I am
28
looking forward to interacting with you guys in the LBS Slack channel.
29
Now, before we start the episode, I have a short story for you guys.
30
A few years ago, I started learning machine stats by watching all the tutorials I could
find that a teacher I really liked was teaching.
31
That teacher was no other than Chris Fonsbeck, PMC's creator and BDFL.
32
And five years down the road, senior
33
unpredictable road, am beyond excited to share that I will now be teaching a tutorial
alongside Chris.
34
That will happen at Pi Data New York from November 6 to 8, 2024.
35
And I would be delighted to see you there.
36
We will be teaching you everything you need to know to master Gaussian processes with IMC.
37
And of course, I will record a few live LBS episodes while I'm there.
38
But
39
I'll tell you more about that in the next episode.
40
In the meantime, you can get your ticket at pine.at.org slash NYC 2024.
41
I can't wait to see you there.
42
Okay, on to the show now.
43
Desi Ivanova, welcome to Learning Bayesian Statistics.
44
Thank you for having me, Alex.
45
Pleased to be here.
46
Yeah, yeah.
47
Thanks a lot for taking the time, for being on the show.
48
a to Marvin Schmidt for putting us in contact.
49
He was kind enough to do that on the base flow stack where we interact from time to time.
50
Today, though, we're not going to talk a lot about advertised Bayesian inference.
51
We're going to talk mostly about experimental design, Beijing experimental design.
52
So BED or BED, I like the acronym.
53
But before that, as usual, we'll start with your origin story, Daisy.
54
Can you tell us what you're doing nowadays and also how you ended up working on what
you're working today?
55
Yeah, of course.
56
So broadly speaking, I work in probabilistic machine learning research, where I've worked
on a few different things, actually.
57
So the audience here would be mostly familiar with Bayesian inference.
58
So I've worked on approximate inference methods, namely, know, variational inference.
59
You mentioned Marvin, right?
60
So we've actually collaborated with him on some amortized inference work.
61
I've also done some work in causality.
62
But my main research focus so far has been in an area called Bayesian experimental design,
as you correctly pointed out, BED for short, a nice acronym.
63
So BED, Bayesian experimental design was the topic of my PhD.
64
And yeah, will be the topic of this podcast episode.
65
Yeah, really, really keen on discussing.
66
and very, very close to my heart.
67
You know, how I ended up here.
68
That's actually a bit quite random.
69
So before, before getting into research, right, so before my PhD, I actually worked in
finance for quite a few years as a, as a quantitative researcher.
70
At some point,
71
I really started missing sort of the rigor in a sense of, you know, conducting research,
you know, being quite principled about, you know, how we measure uncertainty, how we
72
quantify robustness of our models and of the systems that we're building.
73
And right at the height of COVID, I decided to start my PhD back in 2020.
74
And
75
Indeed, the area, right, based on experimental design, that was originally not the topic
of my PhD.
76
I was supposed to work on certain aspects of variational autoencoders.
77
If you're familiar with these types of models, they're not as popular anymore, right?
78
So if I had ended up working on variational autoencoders, I guess a lot of my research
would have been, I mean, not wasted, but not as relevant.
79
not as relevant today as it was, you know, four or five years ago.
80
And how I ended up working with Bayesian experimental design specifically, basically,
approached my supervisor a few months before starting my PhD and I said, Hey, can I can I
81
read about something interesting to prepare for a PhD?
82
And he was like, yeah, just with these papers on Bayesian experimental design.
83
And that's how it happened.
84
Really?
85
Yeah.
86
Okay, cool.
87
Yeah, I love these.
88
I love asking this question because often, you know, with hindsight bias, when you're a
beginner, you like it's easy to trick yourself into thinking that people like you who are
89
experts on a on a particular topic and know that topic really well, because they did a PhD
on on it, like they
90
They have been doing that since they were, I don't know, 18 or even 15 or it was like all,
all planned and part of being big plan.
91
But most of the time when you ask people, it was not at all.
92
And it's the result of experimenting with things and also the result of different people
they have met and, and, and encounters and mentors.
93
And so I think this is also very valuable to, to
94
tell that to beginners because otherwise they can be very daunting.
95
100 % Yeah, I would 100 % agree with that.
96
And actually experimenting is good.
97
You know, again, we'll be talking about experimental design, I think.
98
Yeah, many times, you know, just by virtue of trying something new, you discover, you
know, I actually quite liked that.
99
And it actually works better, you know, for whatever purpose it might be, it might be your
commute to work, right?
100
There was this
101
very interesting research.
102
You know, when there is like a tube closure, right, if the metro is getting closed, you
know, some people, like 5 % of people actually discover an alternative route that actually
103
is much better for the daily commute.
104
But they wouldn't have done that had the closure not happened.
105
So almost like being forced to experiment may lead to actually better outcomes, right?
106
So it's quite interesting.
107
Yeah, yeah, no.
108
mean,
109
completely agree with that and that's also something I tell to a lot of people who reach
out to me, know, wondering how they could start working on Bayesian stats and often I'm
110
like, you know, trying to find something you are curious about, interested in and then
start from there because it's gonna be hard stuff and there are gonna be a lot of
111
obstacles.
112
So if you're not, you know, really curious about
113
what you are studying, it's going to be fairly hard to maintain the level of work that you
have to maintain to, to in the end enjoy what you're doing.
114
So experimenting is very important.
115
I completely agree.
116
and actually do you remember yourself?
117
so I'm curious first how Bajan is your work.
118
And also if you remember when you were, were introduced to, to Bajan stance.
119
When was I introduced to Bayesian stats?
120
That must have been probably in my undergrad days.
121
I remember I took some courses on kind of Bayesian data analysis, but then I didn't do any
of that during my time in industry.
122
Yeah.
123
And again, as I said, I ended up working on
124
Bayesian experimental design a little bit, a little bit randomly.
125
The work itself is, you know, it does use Bayesian principles quite a lot.
126
You know, we do Bayesian inference, we do, we start with a Bayesian model, right?
127
So the modeling aspect is also quite important.
128
You know, it's very important to have a good Bayesian model for all of these things to
actually make sense and work well in practice.
129
So I would say overall, the work is quite, quite Bayesian, right?
130
Yeah.
131
Yeah.
132
Yeah.
133
Yeah, for sure.
134
so actually, I think that's a good segue to introduce now, Bayesian experimental design.
135
So it's the first time we not talk, not the first time we talk about it on the show, but
it's a really dedicated episode about about that.
136
So could you introduce the topic to our listeners and basically explain and define what
Bayesian experimental design is?
137
Yeah, of course.
138
So can I actually take a step back and talk a little bit about experimental design first?
139
Yeah.
140
yeah.
141
And then we'll add the Bayesian kind of the Bayesian aspect to it.
142
So, you know, when, when I say, I work on Bayesian experimental design, most people
immediately think lab experiments, right?
143
For example, you're in a chemistry lab and you're trying to synthesize a new drug or a new
compound or something.
144
But actually, you know, the field of experimental design is much, broader than that,
right?
145
And to, you know, give a few concrete examples, you can think about surveys, right?
146
You may need to decide what questions to ask.
147
Maybe you want to tailor your questions as, you know, the survey progresses so that, you
know, you're asking very tailored, customized questions to each of your survey
148
participants.
149
You can think of clinical trials, right?
150
So how do you dose drugs appropriately?
151
Or, you know, when should you test for certain properties of these drugs, things like
absorption and so on?
152
So all of these things can be, you know, cast as a as an experimental design problem, as
an optimal experimental design problem.
153
So in my mind, designing experiments really boils down to optimal or at least intelligent
data gathering.
154
Does that make sense?
155
So we're trying to kind of optimally collect data in order to kind of learn about the
thing that we want to learn about.
156
So some underlying quantity of interest, right?
157
And the Bayesian framework, so the Bayesian experimental design framework specifically
takes an information theoretic approach to what intelligent or optimal means in this
158
context.
159
So as I already mentioned, it is a is a model based approach, right?
160
So we start with an underlying Bayesian model that actually describes or simulates the
outcome of our experiment.
161
And then the optimality part, right?
162
So the optimal design will be the one that maximizes the amount of information about the
thing that we're trying to learn about.
163
Yeah.
164
That makes sense?
165
can actually give a concrete example.
166
Maybe that will make it easier for you and for the listeners, right?
167
So if you think about, you know, the survey, the survey example, right?
168
kind of a simple but I think very easy to understand concept is you know trying to learn
let's say about time value of money preferences of different people right yeah so what
169
does that mean imagine your
170
a behavioral economist, right?
171
And you're trying to understand some risk preferences, let's say, of people.
172
Generally, the way that you do that is by asking people a series of questions of the form,
do you prefer some money now or you prefer some money later?
173
Right?
174
So do you prefer 50 pounds now or you prefer 100 pounds in one year?
175
Right.
176
And then you can choose, are you going to propose 50 pounds or 60 pounds or 100 pounds
now?
177
how much money you're going to propose in what time, right?
178
So you're going to do a hundred pounds in one month or in three months or in one year,
right?
179
So there is like a few choices that you can make.
180
And there is a strong incentive to do that with as few questions as possible because you
end up paying actually the money to the participants, right?
181
So basically,
182
we can start with an underlying Bayesian model that sort of models this type of
preferences of different human participants in this survey.
183
There's plenty of such models from psychology, from behavioral economics.
184
And at the end of the day, what we want to learn is a few parameters, right?
185
You can think about this model almost like a mechanistic
186
model that explains how preferences relate to things like, you know, are described by
things like a discount factor or sensitivity to various other things.
187
And by asking these series of questions, we're learning about these underlying parameters
in our Bayesian model.
188
Did that make sense?
189
Yeah.
190
Yeah.
191
I understand better now.
192
And so I'm wondering, it sounds a lot like, you know, just doing also causal modeling,
right?
193
So you write your causal graph and then based on that, you can have a generative model and
then, and fitting the model to data is just one part, but it's not what you start with to
194
write the model.
195
How is that related?
196
Right.
197
The fields are, in a sense, closely related in the sense that, you know, in order for you
to uncover kind of the true underlying causal graph, let's say if, you know, you start
198
with some assumptions, you don't know if X causes Y or Y causes X or, you know, or
something else, the way that you need to do this is by intervening in the system.
199
Right.
200
So
201
You can only, in a sense, have causal conclusions if you have rich enough data and by rich
enough data we generally mean experimental or interventional data, right?
202
So you're totally right in kind of drawing parallels in this, right?
203
And indeed we may...
204
design experiments that actually maximize information about the underlying causal graph,
right?
205
So if you don't know the graph and you want to uncover the graph, you can set up a
Bayesian experimental design framework that will allow you to, you know, select, let's
206
say, which nodes in my causal graph should I intervene on, with what value should I
intervene on, so that with as few experiments as possible, with as few interventions as
207
possible,
208
can I actually uncover the true, the ground truth, right?
209
The true underlying causal graph, right?
210
And, you know, kind of the main thing that you're optimizing for is this notion of
information content.
211
So how much information is each intervention, each experiment actually bringing us, right?
212
And...
213
And I think that's part of the reason why I find the Bayesian framework quite appealing as
opposed to, I guess, non-Bayesian frameworks.
214
You know, it really centers around this notion of information gathering.
215
And with the Bayesian model, we have a very precise definition of or a precise way to
measure an information content of an experiment.
216
Right.
217
If you think about
218
Imagine again, we're trying to learn some parameters in a model, right?
219
The natural, again, once we have the Bayesian model, the natural way to define information
content of an experiment is to look at, you know, what is our uncertainty about these
220
parameters under our prior, right?
221
So we start with a prior.
222
We have uncertainty that is embedded or included in our prior beliefs.
223
We're going to perform an experiment to collect some data, right?
224
So perform an experiment, collect some data.
225
we can update our prior to a posterior.
226
So that's classic Bayesian inference, right?
227
And now we can compare the uncertainty of that posterior to the uncertainty of our prior.
228
And the larger the drop, the better our experiment is, the more informative our experiment
is.
229
And so the best...
230
or the optimal experiment in this framework would be the one that maximizes this
information gain.
231
So the reduction in entropy, we're going to use entropy as a measure of uncertainty in
this framework.
232
So it is the experiment that reduces our entropy the most.
233
Does that make sense?
234
Yeah.
235
Total sense.
236
Yeah.
237
Total sense.
238
That's amazing.
239
I didn't know.
240
So yeah, I mean, that's, that's pretty natural then to include the causal framework into
that.
241
And I think that's one of the most powerful features of experimental design, because I
guess most of the time what you want to do when you design an experiment is you want to
242
intervene.
243
on a causal graph and see actually if your graph is close to reality or not.
244
So that's amazing.
245
And I love the fact that you can use experimental design to validate or invalidate your
causal graph.
246
That's really amazing.
247
Correct.
248
100%.
249
But I do want to stress that
250
The notion of causality is not necessary for the purposes of describing what Bayesian
experimental design is.
251
I'll give you a couple of other examples, actually.
252
So you may...
253
You may want to do something like model calibration.
254
Let's say you have a simulator with a few parameters that you can tweak, right?
255
So that it, I don't know, produces the best outcomes, right?
256
Or is optimally calibrated for the thing that you're trying to measure, right?
257
It is like, again, I don't think you need, you know, any concepts of causality here,
right?
258
It's you're turning a few knobs.
259
And you know, again, you can formulate this as an experimental design problem where, you
you are trying to calibrate your system with as few kind of no turns as possible.
260
Yeah.
261
Yeah, yeah, yeah.
262
That makes a ton of sense.
263
Something I'm curious about hearing you talk is, and that's also something you've worked
extensively on, is the computational challenges.
264
Can you talk about that?
265
What are the computational challenges associated with traditional bed, so Bayesian
experimental design, and how they affect the feasibility?
266
of bed in real world applications.
267
Yeah.
268
Yeah, that's that's an excellent point.
269
Actually.
270
I Yeah, I see you read some of of my papers.
271
So, all right.
272
So all of these kind of information objectives.
273
So what I just described, you know, we can look at the information content, we can
maximize information, and so on, like, it's all very natural.
274
And it's all very mathematically precise and beautiful.
275
But working with those information, theoretical objectives is quite difficult in practice.
276
And the reason for that is precisely as you say, they're extremely computationally costly
to compute or to estimate, and they're even more computationally costly to optimize.
277
And the careful listener would have noticed that I mentioned posterior inference.
278
Right.
279
So in order to compute the information content of an experiment, you actually need to
compute a posterior.
280
Right.
281
You need to compute a posterior given your data.
282
Now, where the problem lies is that you need to do this before you have collected your
data.
283
Right.
284
Because you designing an experiment and then only you will be performing it and then
observing the outcome and then you can do.
285
the actual posterior update.
286
Now, what we have to then do is look at our prior entropy minus our posterior entropy and
integrate over all possible outcomes that we may observe under the selected experiment.
287
And we have to do that for a number of experiments to actually find the optimal one.
288
So what we end up with is this sort of nesting
289
of expectations.
290
So we have an expectation, we have an average with respect to all possible outcomes that
we can observe.
291
And inside of our expectation, inside of this average, we have this nasty posterior
quantity that, generally speaking, is intractable.
292
Unless you're in a very specific case where you have a conjugate model, where your
posterior is available in close form, you actually don't have access to that posterior.
293
which means that you will need to do some form of approximation, right?
294
Whether it's exact like MCMC or is going to be a variational posterior computation.
295
Again, there is many ways of doing this.
296
The point is that for each design that you may want to try, you need to compute all of
these posteriors.
297
for every sample of your potential outcome, right?
298
Of your possible outcome under the experiment.
299
So what I was just describing is what is known as a doubly intractable quantity, right?
300
So again, this podcast audience is very familiar with Bayesian inference and how Bayesian
inference is intractable in general.
301
Now computing...
302
the EIG, the sort of computing the objective function that we generally use in Bayesian
experimental design is what is known as doubly intractable objective, which is quite
303
difficult to work with in practice, right?
304
Now, what this means for sort of real world applications is that you either need to throw
a lot of compute
305
on the problem.
306
Or you need to do, you know, some, you need to sort of give up on the idea of being
Bayesian optimal, right?
307
You may use some heuristics or something else.
308
And what this problem really becomes limiting is when we start to think about, you know,
running experiments in real time, for example.
309
So the survey example that I started with, you know,
310
you know, asking participants in your survey, do you prefer somebody now or somebody
later?
311
You know, it becomes quite impractical for you to, you know, run all these posterior
inferences and optimize all of these information theoretic objectives in between
312
questions, right?
313
So it's a little bit, you know, I asked you the first question now, let me run by MCMC.
314
Let me optimize some doubly intractable objective.
315
Can you just wait five minutes, please?
316
And then I'll get back to you with the next question.
317
Obviously, it's not something that you can realistically do in practice.
318
So I think, historically, the computational challenge of the objectives that we use for
Bayesian experimental design has really...
319
limited the feasibility of applying these methods in kind of real-world applications.
320
And how, so how did you, which innovations, which work did you do on that front?
321
That make all that better.
322
Right.
323
So there is a few things that I guess we can discuss here.
324
So number one, I mentioned posterior inference, right?
325
And I mentioned we have to do many posterior inference approximations for every possible
outcome of our experiment.
326
Now, I think it was the episode with Marvin.
327
right, where you talked about amortized Bayesian inference.
328
So in the context of Bayesian experimental design, amortized Bayesian inference plays a
very big role as well, right?
329
So one thing that we can do to sort of speed up these computations is to learn a...
330
to learn a posterior that is amortized over all the outcomes that we can observe, all the
different outcomes that we can observe, right?
331
And the beautiful part is that we know how to do that really well, right?
332
So we have all of these very expressive, variational families.
333
that we can pick from and optimize with data that we simulate from our underlying Bayesian
model.
334
So this aspect of Bayesian experimental design definitely touches on related fields of
amortized Bayesian inference and simulation-based inference.
335
So we're using simulations from our model to learn an approximate posterior.
336
that we can very quickly draw samples from, as opposed to having to fit an HMC for every
new data set that we may observe.
337
That makes sense.
338
Yeah.
339
And so I will refer listeners to the episode with Marvin, episode 107, where we dive into
amortized patient inference.
340
put that in the show notes.
341
I also put for reference three other episodes where we mentioned, you know, experimental
design.
342
So episode 34 with Lauren Kennedy, 35 with Paul Burkner and 45 with Frank Harrell, that
one.
343
focuses more on clinical trial design.
344
But that's going to be very interesting to people who are looking to these.
345
And yeah, so I can definitely see how amortized patient inference here can be extremely
useful based on everything you used it before.
346
Maybe do you have an example, especially I saw that during your PhD,
347
You worked on policy-based patient experimental design and you've developed these methods.
348
Maybe that will give a more concrete idea to listeners about what all of these means.
349
Exactly.
350
One way in which we can speed up computations is by utilizing, as I said, amortized
variational inference.
351
Now this will speed up the estimation of our information theoretic objective, but we still
need to optimize it.
352
Now, given that we have to do after each experiment iteration, right?
353
So we have collected our first data point, we have collected our first data point with a
need to...
354
update our model and with this new model under this new model, updated model, we need to
kind of decide what to do next.
355
Now, this is clearly also very computationally costly, right?
356
The optimization step of our information theoretic objective is quite computationally
costly, meaning that it is very hard to do in real time, right?
357
Again, going back to the survey example, you still can do it, right?
358
You can estimate it a little bit more quickly, but you still can't optimize it.
359
And this is where a lot of my PhD work has actually been focused on, right?
360
So developing methods that will allow you to run Bayes Bayesian Optimal Design in real
time.
361
Now, how are we going to do that?
362
So there is a little bit of a conceptual shift in the way that we think about designing
experiments, right?
363
What we will do is rather than choosing
364
the design, the single design that we're going to perform right now, right in this
experiment iteration.
365
What we're going to do is learn or train a design policy that will take as an input our
experimental data that we have gathered so far, and it will produce as an output the
366
optimal design for the next experiment iteration.
367
So our design policy is just a function, right?
368
It's just a function that takes past experimental data as an input and produces the next
design as an output.
369
Does that make sense?
370
Yeah, yeah, yeah.
371
I can see what that means.
372
How do you integrate that though?
373
You know, like I'm really curious concretely.
374
Yeah.
375
what does integrating all those methods, so a multi-spatial inference, variational
inference to the Bayesian experimental design, and then you have the Bayesian model that
376
underlies all of that.
377
How do you do that completely?
378
Yes, excellent.
379
When we say the model, I generally mean the underlying Bayesian model.
380
This is our model that we use to train our
381
let's say, variational amortized posterior.
382
And this is the same model that we're going to train our design policy network.
383
And I already said it's a design policy network, which means that we're going to be using,
again, deep learning.
384
We're going to be using neural networks to actually learn a very expressive function that
will be able to take our data as an input, produce the next design as an output.
385
Now, how we do that concretely?
386
There is, you know, by now a large number of architectures that we can pick that is
suitable for, you know, our concrete problem that we're considering.
387
So one very important aspect in everything that we do is that our policy, our neural
network should be able to take variable size data sets as an input.
388
Right?
389
Because every time we're calling our policy, every time we want a new design, we will be
feeding it with the data that we have gathered so far.
390
Right?
391
And so it is quite important to be able to condition on or take as an input variable
length sequences.
392
Right?
393
And so concretely, how can we do that?
394
Well, you
395
One kind of standard way of doing things is to basically take our experimental data that
we've gathered so far and embed each data point.
396
So we have an X for our design, a Y for our outcome.
397
Take this pair and embed it to a fixed dimensional representation, right, in some latent
space.
398
Let's say with a small neural network, right?
399
And we do that for each
400
individual design outcome pair, right?
401
So if we have n design outcome pairs, we're gonna end up with n fixed dimensional
representations after we have embedded all of this data.
402
Now, how can we then produce the next sort of optimal design for the next experiment
iteration?
403
There is many choices, and I think it will very much depend on the application.
404
So certain Bayesian models, certain underlying Bayesian models are what we call
exchangeable, right?
405
So the data conditional on the parameters can be, the data conditional on the parameters
is IID, right?
406
Which means that the order in which our data points arrive doesn't matter.
407
And again, the survey example.
408
is quite a good example of this precisely, right?
409
Like it doesn't really matter which question we ask first or second, you know, we can
interchange them and the outcomes will be unaffected.
410
This is very different to time series models where, you know, if we design, if we are
choosing the time points at which to take blood pressure, right, for example,
411
If we decide to take blood pressure at t equals five, we cannot then go back and take the
blood pressure at t equals two.
412
So the choice of architecture will very much depend on, as I said, the underlying problem.
413
And generally speaking, we have found it quite useful to explicitly embed the structure
that is known.
414
So if we know that our model is exchangeable, we should be using an appropriate
architecture, which will ensure that the order of our data doesn't matter.
415
If we have a time series, we can use an architecture that takes into account the order of
the data.
416
So for the first one, we have...
417
kind of standard architecture such as I don't know how familiar the audience would be, but
you know, in deep learning, there is an architecture called deep sets, right?
418
So basically, take our fixed dimensional representations and we simply add them together.
419
Very simple, right?
420
Okay, we have our end design outcome pairs.
421
We add them together, they're all of them are in in the same fixed dimensional
representation, we add them together.
422
Now this is our
423
representation or a summary of the data that we have gathered so far.
424
We take that and we maybe map it through another small neural network to produce the next
design.
425
If we have a time series model, then we can, you know, pass everything through an LSTM or
some form of recurrent neural network to then produce the next design.
426
And that will keep sort of the order in
427
and it will take the order into account.
428
Did that answer your question in terms of like how specifically we think about these
policies?
429
Yeah.
430
Yeah, that's fascinating.
431
And so basically, and we talked about that a bit with Marvin already, but the choice of
neural network is very important depending on the type of data because if you have, many
432
time series are complicated, right?
433
Like they already are, even if you're not using a neural network, time is always
complicated to work with.
434
because there is an autocorrelation, right?
435
So you have to be very careful.
436
So basically that means changing the neural network you're working with.
437
then so concretely, like what, you know, for practitioners, someone who is listening to us
or watching us on YouTube, and they want to start implementing BED in their projects,
438
what's practical advice
439
you would have for them to get started?
440
Like how, why, and also when, you know, because there may be some moments, some cases
where you don't really want to use BED.
441
And also what kind of packages you're using to actually do that in your own work.
442
So that's a big question, I know, but like, again, repeat it as you give the answers.
443
Yeah, yeah, yeah.
444
Let me start with kind of...
445
If people are looking to implement BASE in their projects, I think it is quite important
to sort of recognize where BASE experimental design is applicable, right?
446
So it can be applied whenever we can construct an appropriate model for our experiments,
right?
447
So the modeling part, like the underlying BASE model is actually doing a lot of the heavy
lifting in sort of in this framework.
448
simply because this is basically what we're to assess quality of the designs, right?
449
So the model is informing what a valuable information is.
450
And so I would definitely advise not to gloss over that part.
451
If your model is bad, if your model doesn't represent the data generating process, in
reality, the results might be quite poor.
452
Now, I think it's also good to mention that you don't need to know the exact probability
distribution of the outcomes of the experiment, right?
453
So you can, you know, as long as you can simulate, right?
454
So you can have a simulator-based model that simply samples outcomes of the experiment,
given the experiment.
455
which I think, you know, it simplifies things a little bit.
456
You know, don't have to write down exact probability distributions, but still you need to
be able to sample or simulate this outcome.
457
So that would be step number one, right?
458
So ensuring that you have a decent model that you can start sort of experimenting with,
you know, in the sense of like...
459
designing the policies or like training the policies or sort of designing experiments.
460
The actual implementation aspect in terms of software, unfortunately, based on
experimental design is not as well developed.
461
from software point of view as, for example, amortized Bayesian inferences, right?
462
So I'm sure that you spoke about the baseflow package with Marvin, which is a really
amazing sort of open source effort.
463
They have done a great job of implementing many of the kind of standard architectures that
you can basically, you know,
464
pick whatever works or like pick something that is relatively appropriate for your problem
and it will work out, right?
465
I think that is like a super powerful, super powerful framework that includes, know,
latest and greatest architectures in fact.
466
Unfortunately, we don't have anything like this for Bayesian experimental design yet, but
I am in touch with the Baystow guys and I'm definitely looking into
467
implementing some of these experimental design workflows in their package.
468
So I have it on my to-do list to actually write a little tutorial in Baseflow, how you can
use Baseflow and your favorite deep learning framework of choice, whether it's PyTorch or
469
JAX or like whatever, TensorFlow, to train sort of a...
470
a policy, a design policy along with all of the amortized posteriors and all the bells and
whistles that you may need to run, you know, some pipeline like that.
471
Right.
472
So I mentioned the modeling aspect, I mentioned the software aspect.
473
I think thinking about the problem.
474
in like other aspects of like, you going to run an adaptive experiment or are you going to
run a static experiment?
475
Right.
476
So adaptive experiments are much more complicated than static experiments.
477
So in an adaptive experiment, you're always conditioning on the data that you have
gathered so far, right?
478
In a static experiment, you just design a large batch of experiments and then you run it
once, you collect your data and then you do your Bayesian analysis from there.
479
Right.
480
And so
481
I generally always recommend starting with the simpler case, figure out whether the
simpler case works, do the proof of concept on a static or non-adaptive type of Bayesian
482
experimental design.
483
And then, and only then, start to think about, let me train a policy or let me try to do
an adaptive experimental design.
484
pipeline.
485
I think this is a bit of a common pitfall if I may say, like people tend to like jump to
the more complicated thing before actually figuring out kind of the simple case.
486
Other than that, I think, again, I think it's a kind of an active area of research to, you
know, figure out ways to evaluate your designs.
487
I think by now we have
488
pretty good ways of evaluating the quality of our posteriors, for example, right?
489
You have various posterior diagnostic checks and so on that doesn't really exist as much
for designs, right?
490
So what does it like, you know, I've maximized my information objective, right?
491
I have collected as much information as I can, right?
492
According to this information objective.
493
But what does that mean in practice, right?
494
Like there is no kind of real world information that I can...
495
test with, right?
496
Like if you're doing predictions, can, you can predict, can observe and then compare,
right?
497
And you can compute an accuracy score or a root mean squared error or like whatever makes
sense.
498
There doesn't really exist anything like this in, in sort of in, in, in design, right?
499
So it becomes much harder to quantify the success of such a pipeline.
500
And I think it's, it's, it's a super interesting
501
area for development.
502
It's part of the reason why I work in the field.
503
I think there is many open problems that if we figure out, I think we can advance the
field quite a lot and make data gathering an actual thing, principled and robust and
504
reliable so that you run your expensive pipeline, but you end up with
505
you kind of want to be sure that the day that you end up with is actually useful for the
purposes that you want to use it.
506
yeah, did that answer the question?
507
So you have the modeling aspect, you have the software aspect, which we are developing, I
think, you know, we will hopefully eventually get there.
508
Think about your problem, start simple.
509
Try to think about diagnostics.
510
And I think, again, I mentioned, you know, it's very much an open
511
an open problem, but maybe for your concrete problem at hand, you might be able to kind of
intuitively say, this looks good or this doesn't look good.
512
Automating this is like, it's a very interesting open problem and something that I'm
actively working on.
513
Yeah.
514
Yeah.
515
And thank you so much for all the work you're doing on that because I think it's super
important.
516
I'm really happy to see you on the base flow side because yeah, those guys are doing
517
Amazing work.
518
There is the new version that's now been merged on the dev branch, which is back in
agnostics.
519
So people can use it with their preferred deep learning package.
520
So I always forget the names, but TensorFlow, PyTorch, and JAX, I'm guessing.
521
I'm mostly familiar with JAX because that's the one.
522
and a bit of PyTorch because that's the ones we're interacting with in PymC.
523
This is super cool.
524
I've linked to the Baseflow documentation in the show notes.
525
Is there maybe, I don't know, a paper, a blog post, something like that you can link
people to with a workflow of patient experimental design and that way people will get an
526
idea of how to do that.
527
So hopefully by the time the episode is out, I will have it ready.
528
Right now I don't have anything kind of practical.
529
I'm very happy to send some of the kind of review papers that are out there on Bayesian
Experimental Design.
530
hopefully in the next couple of weeks I'll have the tutorial, like a very basic
introductory tutorial.
531
you know, we have a simple model, we have our simple parameters, you know, what we want to
learn, here is how you define your posteriors, here is how we define your policy, you
532
know, and then you switch on base flow, and then you know, voila, you have you have you
have your results.
533
So yeah, I'm hoping to get a blog post like of this of this sort done in in the next
couple of weeks.
534
So once ready, I will thank you.
535
I will thank you with that.
536
Yeah, for sure.
537
yeah, for sure.
538
Can't wait and Marvin and I are gonna start working on setting up a modeling webinar
Amazing which is a you know another format I have on the show So this is like, you know,
539
I'm like Marvin will welcome and share his screen and show how to do the the amortized
patient inference workflow with base flow also using pinc and all that cool stuff now that
540
the new API
541
is merged, we're going to be able to work on that together and set up the modeling
webinar.
542
So listeners, definitely stay tuned for that.
543
I will, of course, announce the webinar a bit in advance so that you all have a chance to
sign up.
544
And then you can join live, ask questions to Marvin.
545
That's going to be super fun.
546
And mainly see how you would do Amortize Bayesian inference, concretely.
547
Great.
548
Amazing.
549
Sounds fun.
550
Yeah, that's going to be super fun.
551
Something I was thinking about is that your work mentions enabling real-time design
decisions.
552
And that sounds really challenging to me.
553
So I'm wondering how critical is this capability in today's data-driven decision-making
processes?
554
Yeah.
555
I do think it really is quite critical, right?
556
In most kind of real world practical aspects, practical problems, you really do need to
run to make decisions fairly quickly.
557
Right?
558
Again, all the surveys as an example, you know, you have anything that involves a human,
and you want to adapt as you're performing the experiment, you kind of need to ensure that
559
things are
560
you're able to run things in real time.
561
And honestly, I think part of the reason why we haven't seen a big explosion or based on
experimental design in practice is partly because we couldn't until recently actually run
562
these things fast enough, both because of the computational challenges, now that we know
how to do amortize inference very well, now that we know how to train policies.
563
that will produce designs very well.
564
I am expecting things to, you know, to improve, right?
565
And to start to see some of these some of these methods applied in practice.
566
Having said that, I do think and please stop me if that's totally unrelated, but to make
things
567
successful in practice, there are a few other things that in my opinion have to be
resolved before, you know, we're confident that we can, you apply, you know, such black
568
boxes in a sense, right, because we have all these neural networks all over the place.
569
And it's not entirely clear whether all of these things are robust to various aspects of
the complexities of the real world.
570
Right.
571
So
572
things like model mis-specification, right?
573
So is your Bayesian model actually a good representation of the thing that you're trying
to study?
574
That's a big open problem again.
575
And again, I'm going to make a parallel to Bayesian inference actually.
576
For inference purposes, model mis-specification may not be as bad as it is for design
purposes.
577
And the reason for that is you will still get valid under some of the assumptions, of
course, you will still get valid inferences or like you will still be close.
578
You still get the best that you can do under the under the the the assumption of a wrong
model.
579
Now, when it comes to design, we have absolutely no guarantees.
580
And oftentimes we end up in very pathological situations where because we're using our
model to inform the data collection.
581
and then to also evaluate, right, fix that model on the same data that we've gathered.
582
If your model is misspecified, you might not even be able to detect the misspecification
because of the way that the data was gathered.
583
It's not IID, right?
584
Like it's very much a non-IID data collection process.
585
And so I think when we talk about practical things,
586
we really, really need to start thinking about how are we going to make our systems or the
methods that we develop a little bit more robust to misspecification.
587
And I don't mean we should solve model misspecification.
588
I think that's a very hard task that is basically unsolvable, right?
589
Like it is solvable under assumptions, right?
590
If you tell me what your misspecification is, you know, we can improve things, but in
general, this is not
591
something that we can sort of realistically address uniformly.
592
But yeah, so again, going back to practicalities, I do think it's of crucial importance to
sort of make our pipeline in diagnostics sort of robust to some forms of
593
mis-specification.
594
Yeah.
595
Yeah, yeah, for sure.
596
And that's where also
597
I really love Amortized Patient Inference because it allows you to do simulation-based
calibration.
598
And I find that especially helpful and valuable when you're working on developing a model
because already before fitting to data, you already have more confidence about what your
599
model is actually able to do and not do and where the possible pain points would be.
600
And I find that.
601
super helpful.
602
And actually talking about all that, I'm wondering where you see the future of Bayesian
experimental design heading, particularly with advancements in AI and machine learning
603
technologies.
604
Wow.
605
Okay.
606
So I do view this
607
type of work.
608
So this type of research is a little bit orthogonal to all of the developments in sort of
modern AI and machine learning.
609
And the reason for this is that we can literally borrow the latest and greatest
development in machine learning and plug it into our pipelines.
610
there is a better architecture to do X, right?
611
Like we can take that architecture and, you know, utilize it for our purposes.
612
So I think
613
you know, when it comes to the future of Bayesian experimental design, given, you know,
all of the advancements, I think this is great because it's kind of helping the field even
614
more, right?
615
Like we have more options to choose from, we have better models to choose from, and kind
of the data gathering aspect will always be there, right?
616
Like we will always want to collect better data for the purposes of, you know, our data
analysis.
617
And so, you know, the design aspect will still be there and with the better models, we'll
just be able to gather better data, if that makes sense.
618
Yeah, that definitely makes sense.
619
Yeah, for sure.
620
And that's interesting.
621
Yeah, I didn't anticipate that kind of answer, that's okay.
622
definitely.
623
see what you mean.
624
Maybe before like, yeah, sorry, but even if you think, you know, now you're everybody's
gonna have their AI assistant, right?
625
Now, wouldn't it be super frustrating if your AI assistant takes three months to figure
out what you like for breakfast?
626
And like, it's experimenting or like, it's just randomly guessing.
627
do you like fish soup for breakfast?
628
Like,
629
How about I prepare you a fish soup for breakfast or like, or I propose you something like
that, right?
630
And so I think again, like this personalization aspect, right?
631
Like again, kind of sticking to, I don't know, personal AI assistance, right?
632
The sooner or the quicker they are able to learn about your preferences, the better that
is.
633
And again, you know, we're learning about preferences.
634
Again, I'm gonna refer back to the...
635
you know, time value of many preference learning, like it is just a more complicated
version of that.
636
Right.
637
And so if your latest and greatest AI assistant is able to learn and customize itself to
your preferences much more quickly than otherwise, you know, that's a huge win.
638
Right.
639
And I think this is precisely where all these sort of principle data gathering techniques
can really shine.
640
Once we figure out, you know, the
641
the sort of the issues that I was talking about, I mean, that makes sense.
642
Maybe to play us out, I'm curious if you have something like applications in mind,
practical applications of bed that you've encountered in your research, particularly in
643
the fields of healthcare or technology that you found particularly impactful.
644
Right.
645
Excellent question.
646
And actually, I was going to mention that aspect as, you know, where you see the future of
Bayesian experimental design.
647
Part of kind of our blind spots, if I may refer to that as sort of blind spots, is that in
our research so far, we have very much focused on developing methods, developing
648
computational methods to sort of make some of these
649
based on experimental design pipelines actually feasible to run in practice.
650
Now, we haven't really spent much time working with practitioners, and this is a gap that
we're actively trying to sort of close.
651
In that spirit, we have a few applications in mind.
652
We sort of apply people.
653
particularly in the context of healthcare, as you mentioned.
654
So clinical trials design is a very big one.
655
So again, things like getting to the highest safe dose as quickly as possible, For, again,
being personalized to the human, given their context, given their various characteristics.
656
is one area where we're looking to sort of start some collaborations and explore this
further.
657
Our group in Oxford have a new PhD student joining that will be working in collaboration
with biologists to actually design experiments for something about cells.
658
I don't know anything about biology, so.
659
I'm not the best person to actually describe that line of work.
660
But hopefully there will be some concrete, exciting applications in the near future.
661
So that's applications in biology.
662
And finally, you know, there's constantly lots of different bits and pieces like, you
know, people from chemistry saying, hey, I have this thing, can we...
663
Can we work on maybe, you know, performing or like setting up a basic experimental design
pipeline?
664
I think the problem that we've had so far or I've had so far is just lack of time.
665
There's just so many things to do and so little time.
666
But I am very much actively trying to find time in my calendar to actually work on a few
applied projects because I do think, you know,
667
It's all like developing all these methods is great, right?
668
I mean, it's very interesting math that you do.
669
It's very interesting coding that you do.
670
But at the end of the day, you kind of want these things to make someone life's better,
right?
671
Like a practitioner that will be able to save some time or save some money or, you know,
improve their data gathering and therefore, you know, downstream analysis much better.
672
and more efficient thanks to some of this research.
673
So I hope that answered your question in terms of concrete applications.
674
think we'll see more of that.
675
But so far, you know, the two things are, yeah, clinical trial design that we're exploring
and some of this biology cell stuff.
676
Yeah, yeah, no, that's worrying.
677
that's, I mean, definitely looking forward to it.
678
That sounds absolutely fascinating.
679
yeah, if you can make that.
680
happening in important fields like that, that's going to be extremely impactful.
681
Awesome, Desi.
682
I've already...
683
Do you have any sort of applications that you think design, basic experimental design
might be suitable?
684
I know you're quite experienced in various modeling aspects based in modeling.
685
So, yeah, do you have anything in mind?
686
Yeah, I mean a lot.
687
So marketing, know is already using that a lot.
688
Yeah.
689
Clinical trials for sure.
690
Also now that they work in sports analytics, well, definitely sports, you know, you could
include that into the training of elite athletes and design some experiments to actually
691
test causal graphs and see if pulling that lever during training is actually something
that makes a difference during
692
the professional games that actually count.
693
yeah, I can definitely see that having a big impact in the sports realm, for sure.
694
Nice.
695
Well, if you're open to collaborations, can do some designing of experiments once you up
your sports models.
696
Yeah.
697
Yeah.
698
Yeah.
699
mean, as soon as I work on that, I'll make sure to reach out because that's going to be...
700
It's definitely something I'm work on and dive into.
701
So that's gonna be fascinating to work on that with you for sure.
702
Sounds fun, yeah.
703
Yeah, exactly.
704
Very, very exciting.
705
well, thanks, Stacey.
706
That was amazing.
707
I think we covered a lot of ground.
708
I'm really happy because Alina had a lot of questions for you.
709
But thanks a lot for keeping your answers very...
710
focused and not getting distracted by all my decorations.
711
Of course, I have to ask you the last two questions I ask every guest at the end of the
show.
712
So first one, if you had unlimited time and resources, which problem would you try to
solve?
713
it's a really hard one.
714
Honestly, I because I know you asked those questions.
715
I was like, what am going to say?
716
I honestly don't know.
717
There's so many things.
718
But
719
Again, I think it would be something of high impact for humanity in general, probably so
in climate change would be something that I would dedicate my unlimited time and
720
resources.
721
That's good answer.
722
That's definitely a popular one.
723
So you're in great company and I'm sure the team already working on that is going to be
very...
724
be happy to welcome you.
725
No, let's hope.
726
I think we need to speed up the solutions.
727
Like seeing what is happening, right?
728
I think it's rather unfortunate.
729
And second question, if you could have dinner with any great scientific mind, dead, alive
or fictional, who would it be?
730
Yeah.
731
So I think it will be Claude Shannon.
732
So, you
733
the godfather of information theory or like actually the father of information theory.
734
Again, partly because a lot of my research is inspired by information theory principles in
Bayesian experimental design, but also outside of Bayesian experimental designs.
735
It sort of underpins a lot of the sort of modern machine learning development, right?
736
And
737
What I think will be really quite cool is that if you were to have dinner with him, if I
were to have dinner with him and basically tell him like, hey, look at all these language
738
models that we have today.
739
Like Claude Shannon was the person that invented language models back in 1948, right?
740
So that's many years ago.
741
And like, literally not even having computers, right?
742
So he would, he calculated things by hand and produced output that actually looks like
English, right?
743
In 1948.
744
And so I think, you know, a brilliant mind like him, you know, just seeing kind of the
progress that we've made since then.
745
And like, we actually have language models on computers that behave like humans.
746
I'll be super keen to hear like, what is next from him.
747
And I think he will have some very interesting answers to that.
748
What is the future for information processing and the path to artificial, I guess people
call it artificial general intelligence.
749
So what would be the path to AGI from here onwards?
750
Yeah, for sure.
751
That'd be a fascinating dinner.
752
Make sure it comes like that.
753
Awesome.
754
Well.
755
Desi, thank you so much.
756
think we can call it a show.
757
I learned so much.
758
I'm sure my listeners did too, because as you showed, is a very, this is a topic that's
very much on the frontier of science.
759
So thank you so much for all the work you're doing on that.
760
And as usual, I put resources and a link to your website in the show notes for those who
want to dig deeper.
761
Thank you again, Desi, for taking the time and being on this show.
762
Thank you so much for having me.
763
It was my pleasure.
764
This has been another episode of Learning Bayesian Statistics.
765
Be sure to rate, review, and follow the show on your favorite podcatcher, and visit
learnbaystats.com for more resources about today's topics, as well as access to more
766
episodes to help you reach true Bayesian state of mind.
767
That's learnbaystats.com.
768
Our theme music is Good Bayesian by Baba Brinkman, fit MC Lass and Meghiraam.
769
Check out his awesome work at bababrinkman.com.
770
I'm your host.
771
Alex and Dora.
772
can follow me on Twitter at Alex underscore and Dora like the country.
773
You can support the show and unlock exclusive benefits by visiting Patreon.com slash
LearnBasedDance.
774
Thank you so much for listening and for your support.
775
You're truly a good Bayesian.
776
Change your predictions after taking information in.
777
And if you're thinking I'll be less than amazing.
778
Let's adjust those expectations.
779
me show you how to be a good Bayesian Change calculations after taking fresh data in Those
predictions that your brain is making Let's get them on a solid foundation