[Alex Andorra]: 00:00:00

Charles Margossian, welcome to Learning Vision Statistics.

[Charles Margossian]: 00:00:06

Hi Alex, thanks for having me.

[Alex Andorra]: 00:00:09

Yeah, thanks a lot for taking the time. I'm super happy to have you here.

[Alex Andorra]: 00:00:14

It's been a while since I wanted to have you on the show and now I managed to

[Alex Andorra]: 00:00:20

find the slot and thank you also to a few patrons who sent me messages

[Alex Andorra]: 00:00:26

to tell me that they would really like to hear about you on the show. So thanks

[Alex Andorra]: 00:00:34

a lot folks for being so proactive and giving me. ideas for the show. So Charles,

[Alex Andorra]: 00:00:44

let's dive in and

[Charles Margossian]: 00:00:47

But

[Alex Andorra]: 00:00:47

as

[Charles Margossian]: 00:00:47

maybe

[Alex Andorra]: 00:00:48

usual...

[Charles Margossian]: 00:00:49

I can bring up a very quick anecdote, which

[Alex Andorra]: 00:00:52

Oh sure.

[Charles Margossian]: 00:00:52

is I think two, three weeks ago, your show came up. And I told my

[Charles Margossian]: 00:00:58

colleagues, I would feel like I have made it in the Bayesian. statistics

[Charles Margossian]: 00:01:03

world if I get an invitation to speak on learning Bayesian statistics. So

[Charles Margossian]: 00:01:09

I'm thrilled to be here and I think sort of the mysterious patrons

[Charles Margossian]: 00:01:14

who have incited this meeting.

[Alex Andorra]: 00:01:19

Yes, I'm sure they will recognize themselves. So, yeah, let's start with your

[Alex Andorra]: 00:01:26

origin story, actually. That's also something I found super interesting. How

[Alex Andorra]: 00:01:32

did you come to the world of statistics and pharmacometrics and epidemiology?

[Charles Margossian]: 00:01:38

Mm-hmm. Right. So I became interested in statistics and data as an undergrad.

[Charles Margossian]: 00:01:47

Actually,

[Alex Andorra]: 00:01:48

Mm-hmm.

[Charles Margossian]: 00:01:48

I was studying physics. I was already in the US. I was at Yale and

[Charles Margossian]: 00:01:52

I was working with an astronomy lab

[Alex Andorra]: 00:01:55

Mm-hmm.

[Charles Margossian]: 00:01:55

on Exoplanet. We had a lot of data.

[Alex Andorra]: 00:01:57

Mm-hmm.

[Charles Margossian]: 00:01:59

So that generally got me interested in data science. But I wasn't

[Charles Margossian]: 00:02:05

introduced to Bayesian methods. And I always was a bit uncomfortable

[Charles Margossian]: 00:02:10

with how we handling uncertainty, what might we do about it. And I was fortunate

[Charles Margossian]: 00:02:17

after I graduated to actually get a job at a, at a biotech company where,

[Charles Margossian]: 00:02:24

so the company was met from research group. They were

[Alex Andorra]: 00:02:27

Mm-hmm.

[Charles Margossian]: 00:02:27

based in Connecticut and the supervisor I got there, William Gillespie.

[Alex Andorra]: 00:02:35

Mm-hmm.

[Charles Margossian]: 00:02:36

is one of the pioneers of Bayesian methods in pharmacometrics. And

[Charles Margossian]: 00:02:42

so he introduced me to Bayesian statistics and it made a lot of sense.

[Charles Margossian]: 00:02:50

And I realized, oh, this is what I have been looking for when I was

[Charles Margossian]: 00:02:53

doing astronomy. Right. And I started, you know, piecing this together.

[Charles Margossian]: 00:02:59

And to be fair, I hadn't done a lot of statistics before. I don't have

[Charles Margossian]: 00:03:04

the experience of struggling with classical statistics for a decade before

[Charles Margossian]: 00:03:11

being rescued by Bayesian statistics. I encountered Bayesian very early

[Charles Margossian]: 00:03:15

on. But even then, the way we counterfeit uncertainty, the fact that

[Charles Margossian]: 00:03:23

everything reduced to essentially one equation was extremely compelling

[Charles Margossian]: 00:03:29

to me. And Bill Gillespie, hired me

[Alex Andorra]: 00:03:37

Mm-hmm.

[Charles Margossian]: 00:03:38

at Matrim Research Group to work on Stan. So this was back in 2015.

[Charles Margossian]: 00:03:43

And at the time, Stan was fairly new, very exciting, very promising.

[Charles Margossian]: 00:03:49

And you had these specialized software in pharmacometrics, but they were

[Charles Margossian]: 00:03:55

not open source. They were not really supporting Bayesian statistics,

[Charles Margossian]: 00:04:02

or at least that was not their priority. And they didn't give you flexibility

[Charles Margossian]: 00:04:06

that a lot of my colleagues were looking for. On the other hand, you had

[Charles Margossian]: 00:04:10

Stan, which was open source, which had great algorithms to do vision modeling,

[Charles Margossian]: 00:04:14

but which lacked a lot of the features required to do pharmacometrics

[Charles Margossian]: 00:04:20

modeling, right? One example was support for differential equation solvers.

[Charles Margossian]: 00:04:27

And all these models are based

[Alex Andorra]: 00:04:29

Yeah.

[Charles Margossian]: 00:04:29

on ODEs. And at the time, Stan had limited support for ODEs. and more

[Charles Margossian]: 00:04:34

generally implicit functions.

[Charles Margossian]: 00:04:39

And then there were some more specialized things like handling, you

[Charles Margossian]: 00:04:42

know, the event schedule of clinical trials. And the project ended up

[Charles Margossian]: 00:04:49

being, we're going to write some general features, we're going to

[Charles Margossian]: 00:04:52

contribute them to Stan, and then we're going to write a specialized

[Charles Margossian]: 00:04:54

extension called Torsten. And that's going to have, you know, more bespoke

[Charles Margossian]: 00:05:00

features targeted at a pharmacometrics audience. And so... That was

[Charles Margossian]: 00:05:09

my exposure to Stan. I can tell you about my first poll request, which

[Charles Margossian]: 00:05:12

Bob Carpenter and Daniel Lee reviewed

[Alex Andorra]: 00:05:15

Mm-hmm.

[Charles Margossian]: 00:05:15

extensively. They were very patient with my C++. And

[Alex Andorra]: 00:05:20

Hehehe

[Charles Margossian]: 00:05:23

my plan, my loose plan had been, well, I'm gonna work for one or two

[Charles Margossian]: 00:05:28

years. I'm gonna learn programming and statistics because I had gained

[Charles Margossian]: 00:05:32

an appreciation for it in astronomy. And then I'm going to do a PhD

[Charles Margossian]: 00:05:37

in physics. But Andrew Gellman, who I had met in an official capacity

[Charles Margossian]: 00:05:45

like through my work, encouraged me to apply to the statistics program

[Charles Margossian]: 00:05:49

at Columbia University, saying, you know, if you do statistics, you'll

[Charles Margossian]: 00:05:54

still be able to work on the natural sciences and the physics that

[Charles Margossian]: 00:05:57

you're interested in, but you

[Alex Andorra]: 00:05:58

Yeah.

[Charles Margossian]: 00:05:58

know, you'll have, maybe you'll be able to make a more unique contribution

[Charles Margossian]: 00:06:04

as a statistician. So I took his word for it. And yeah, and once

[Charles Margossian]: 00:06:10

I had the offer from Columbia, I mean, it was a difficult decision. Um, but

[Charles Margossian]: 00:06:15

I decided, okay, I'm going to do a PhD in statistics. And that was a big

[Charles Margossian]: 00:06:21

change, the field. Um, but I did pursue that. I was able to continue

[Charles Margossian]: 00:06:26

working on Stan and on Stan adjacent, uh, projects. And just a year

[Charles Margossian]: 00:06:31

ago, I completed the PhD. So I would say that's the origin story. Um,

[Charles Margossian]: 00:06:38

that eventually led me to the Flatiron Institute. So that's where

[Charles Margossian]: 00:06:41

I'm currently, it's in New York. We're a non-profit. We focus on applying

[Charles Margossian]: 00:06:46

computational methods to the basic sciences.

[Alex Andorra]: 00:06:49

Mm-hmm.

[Charles Margossian]: 00:06:49

So big emphasis on collaboration. We have people who do astronomy, who do quantum

[Charles Margossian]: 00:06:54

physics, do biology. And so that really resonates with why I wanted

[Charles Margossian]: 00:07:00

to do statistics in the first place, which I wanna solve problems in

[Charles Margossian]: 00:07:05

the sciences.

[Alex Andorra]: 00:07:09

Yeah, I can really relate to that. That was also kind of what happened to

[Alex Andorra]: 00:07:14

me too, even though the first field was political science and not astronomy.

[Charles Margossian]: 00:07:21

Okay.

[Alex Andorra]: 00:07:22

But yeah,

[Charles Margossian]: 00:07:23

Yeah.

[Alex Andorra]: 00:07:24

definitely in the end, applying the methods became more interesting than the

[Alex Andorra]: 00:07:32

field in itself. So continued

[Charles Margossian]: 00:07:35

Right,

[Alex Andorra]: 00:07:36

on that path.

[Charles Margossian]: 00:07:37

right, and I think that, you know, the attitude I had, so I had offers

[Charles Margossian]: 00:07:42

to do a PhD in physics, a PhD in statistics, and a bunch of different

[Charles Margossian]: 00:07:47

fields. And at the time, my attitude was, look, wherever I go, I'm gonna

[Charles Margossian]: 00:07:54

have to do, you know, analyze some data, understand where that data

[Charles Margossian]: 00:07:58

comes from. I'm gonna have to, you know, have the tools to do the

[Charles Margossian]: 00:08:03

analysis or do the statistics.

[Alex Andorra]: 00:08:05

Mm-hmm.

[Charles Margossian]: 00:08:06

So the idea is, I think our loyalty is not to a discipline or to a

[Charles Margossian]: 00:08:10

field, it's really to a problem.

[Alex Andorra]: 00:08:11

Mm-hmm.

[Charles Margossian]: 00:08:12

Whatever

[Alex Andorra]: 00:08:12

Yeah.

[Charles Margossian]: 00:08:12

problem we work on, there's always a lot that we need to learn. We're never

[Charles Margossian]: 00:08:16

experts in a new problem, in a new research problem. And so kind of like,

[Charles Margossian]: 00:08:23

try not to take the field that I mean, whether it's for the PhD or even,

[Charles Margossian]: 00:08:26

right now I'm in computational mathematics, right? You know, whatever that

[Charles Margossian]: 00:08:30

means. that I care more about the problems I work on than the department

[Charles Margossian]: 00:08:36

I'm affiliated with.

[Alex Andorra]: 00:08:38

Yeah, yeah, completely, completely really do that for sure. I like the fact of

[Alex Andorra]: 00:08:45

using the method to solve an interesting problem, whether it is about astronomy,

[Alex Andorra]: 00:08:50

political science, biology,

[Charles Margossian]: 00:08:51

Mm-hmm.

[Alex Andorra]: 00:08:52

epidemiology, it's like, and in the end, I find that even more interesting

[Alex Andorra]: 00:08:56

because you get to work on a variety of topics that you wouldn't have otherwise.

[Alex Andorra]: 00:09:03

And also you

[Charles Margossian]: 00:09:03

Right.

[Alex Andorra]: 00:09:03

learn so much. So that's really

[Charles Margossian]: 00:09:06

Yeah,

[Alex Andorra]: 00:09:06

the cool thing.

[Charles Margossian]: 00:09:07

yeah.

[Alex Andorra]: 00:09:10

Yeah. Thanks a lot for this introduction. And so right now, well, these

[Alex Andorra]: 00:09:16

days you're working at the Flatiron Institute, as you were saying, and it seems

[Alex Andorra]: 00:09:21

to be a very diverse organization. We had Bob Carpenter actually

[Charles Margossian]: 00:09:27

Mm-hmm.

[Alex Andorra]: 00:09:28

recently on the show, so I'm going to link to his episode in the show notes.

[Alex Andorra]: 00:09:34

But you, Charles, what are the topics you are... particularly interested in

[Alex Andorra]: 00:09:40

these days at the flat iron.

[Charles Margossian]: 00:09:44

Yeah, I think that I have a little bit two pools right now. There are

[Charles Margossian]: 00:09:50

a bit on the methodology side

[Alex Andorra]: 00:09:52

Mm-hmm.

[Charles Margossian]: 00:09:53

of things. And one of them is variational inference, which is a form

[Charles Margossian]: 00:10:01

of, it's called approximate Bayesian method, is what it's called. And really

[Charles Margossian]: 00:10:10

what it's... What I'm trying to understand is when should we use

[Charles Margossian]: 00:10:19

an approximate method like variational inference, because it's incredibly

[Charles Margossian]: 00:10:24

popular. It's used in a lot of fields of machine learning.

[Alex Andorra]: 00:10:28

Mm-hmm.

[Charles Margossian]: 00:10:28

And it's used a lot of times in artificial intelligence. And yet you

[Charles Margossian]: 00:10:33

have so many smart, brilliant people who are completely distrustful of variation

[Charles Margossian]: 00:10:41

and inference. And it's also not difficult to construct an example

[Charles Margossian]: 00:10:46

where it really doesn't do the job, where it really fails in a spectacular

[Charles Margossian]: 00:10:51

way. And I think that ultimately it depends on the problem you apply

[Charles Margossian]: 00:10:56

it to. And what we need to do is understand. you know, when can we

[Charles Margossian]: 00:11:02

get away with the approximations that variational inference proposes to do?

[Charles Margossian]: 00:11:08

Why does it sometimes work really well? Or why do we sometimes really

[Charles Margossian]: 00:11:12

get punished for using variational inference? And I can give you a

[Charles Margossian]: 00:11:18

very simple example of that,

[Alex Andorra]: 00:11:20

Yeah.

[Charles Margossian]: 00:11:21

which was, so this was a recent work with. with Lauren Saul, who is

[Charles Margossian]: 00:11:28

also at the Flatiron Institute. He kind of has machine learning there. And

[Charles Margossian]: 00:11:35

there's a statement that goes around, which is to say, well, variation

[Charles Margossian]: 00:11:38

inference will give you good estimates of the expectation value for

[Charles Margossian]: 00:11:43

a target distribution. So that's great. We often care about expectation

[Charles Margossian]: 00:11:48

values, whether

[Alex Andorra]: 00:11:49

Mm-hmm.

[Charles Margossian]: 00:11:49

that's in statistical physics or in Bayesian statistics.

[Alex Andorra]: 00:11:52

Mm-hmm.

[Charles Margossian]: 00:11:53

But on the other hand, it will underestimate uncertainty.

[Alex Andorra]: 00:11:56

Mm-hmm. Okay.

[Charles Margossian]: 00:11:57

Okay, well, what does it mean to underestimate uncertainty? How do

[Charles Margossian]: 00:12:01

we, you know, how do we make sense of a statement like that, which

[Charles Margossian]: 00:12:04

has appeared time and time again, over the last two decades.

[Alex Andorra]: 00:12:09

Mm-hmm.

[Charles Margossian]: 00:12:10

And really we realized that there were two measures of uncertainty

[Charles Margossian]: 00:12:13

that seem to come up again and again, one is the marginal variances and

[Charles Margossian]: 00:12:18

the other one is the entropy

[Alex Andorra]: 00:12:20

Mm-hmm.

[Charles Margossian]: 00:12:21

and so people like the entropy because it's a multivariate notion.

[Charles Margossian]: 00:12:25

of uncertainty and it's a generalization of the marginal variance.

[Charles Margossian]: 00:12:30

So when people say variation on inference underestimates uncertainty, they

[Charles Margossian]: 00:12:36

usually mean you're underestimating the marginal variances and you're underestimating

[Charles Margossian]: 00:12:40

the entropy.

[Alex Andorra]: 00:12:41

Okay.

[Charles Margossian]: 00:12:42

And so what we ended up doing is demonstrating this on the Gaussian

[Charles Margossian]: 00:12:46

case, right? You have a Gaussian target, you're approximating it

[Charles Margossian]: 00:12:49

with a Gaussian with a diagonal covariance matrix. So that's called

[Charles Margossian]: 00:12:55

the factorized approximation or the mean field approximation.

[Alex Andorra]: 00:12:59

Mm-hmm.

[Charles Margossian]: 00:12:59

And indeed you underestimate the marginal variance and you underestimate

[Charles Margossian]: 00:13:02

the entropy. Here's where it gets interesting, which is as you start

[Charles Margossian]: 00:13:07

going to higher dimensions, And for some reason, I don't think people,

[Charles Margossian]: 00:13:11

people have only looked at, you know, really the two dimensional case,

[Charles Margossian]: 00:13:14

because that's the figure that fits on that page. Right. But if you start

[Charles Margossian]: 00:13:18

going to higher dimensions and you take limits where the dimension goes

[Charles Margossian]: 00:13:21

to infinity, you can construct examples where you actually get very, very

[Charles Margossian]: 00:13:29

accurate estimates of the entropy, but you are underestimating the

[Charles Margossian]: 00:13:35

marginal variances in every dimension in an arbitrarily bad man.

[Charles Margossian]: 00:13:41

And so the two notions of uncertainty are not at all equivalent and

[Charles Margossian]: 00:13:44

not at all interchangeable. And what ends up happening is you look at fields

[Charles Margossian]: 00:13:49

where variation inference is applied. So

[Alex Andorra]: 00:13:52

Mm-hmm.

[Charles Margossian]: 00:13:52

for example, in statistical physics, where, where people want to estimate

[Charles Margossian]: 00:13:57

the entropy of an easing model, for example, and that's

[Alex Andorra]: 00:14:00

Mm-hmm.

[Charles Margossian]: 00:14:00

where this factor is this mean field approximation comes

[Alex Andorra]: 00:14:05

Okay.

[Charles Margossian]: 00:14:05

from. Well, here it works just fine. You actually get good estimates

[Charles Margossian]: 00:14:08

of the entropy, right? In certain limits. In machine learning, where

[Charles Margossian]: 00:14:13

you're trying to maximize marginal likelihoods, rather than default Bayesians,

[Charles Margossian]: 00:14:19

actually, you know, you get good estimates of those marginal likelihoods.

[Charles Margossian]: 00:14:23

At least that's our working conjectures. But in

[Alex Andorra]: 00:14:25

Okay.

[Charles Margossian]: 00:14:25

Bayesian statistics, where we have interpretable quantities, and we know

[Charles Margossian]: 00:14:31

those marginal variances mean something for those parameters that have

[Charles Margossian]: 00:14:35

a meaning. Well, here, variation inference might really not or at

[Charles Margossian]: 00:14:40

least not vanilla implementations in it. And that's an example of how, by studying

[Charles Margossian]: 00:14:48

an example, we can start understanding why is it that some people are

[Charles Margossian]: 00:14:53

so enthusiastic about variation inference and other people are so

[Charles Margossian]: 00:14:57

distrustful of it. It really depends on what is the measure of uncertainty

[Charles Margossian]: 00:15:02

that you care about, and that in turn is informed by the problem you

[Charles Margossian]: 00:15:05

want to solve. So I bring this up as an archetype of the work that we're

[Charles Margossian]: 00:15:14

trying to do.

[Alex Andorra]: 00:15:15

Mm-hmm.

[Charles Margossian]: 00:15:15

We wanna understand this method one in one.

[Alex Andorra]: 00:15:18

Mmm. Yeah, that's really interesting.

[Charles Margossian]: 00:15:24

First big topic.

[Alex Andorra]: 00:15:25

Yeah.

[Charles Margossian]: 00:15:27

The other big topic is I still do a lot of MCMC.

[Alex Andorra]: 00:15:31

Mm-hmm.

[Charles Margossian]: 00:15:31

I care a lot about MCMC because as we'll see in pharmacometrics, I

[Charles Margossian]: 00:15:37

really think that is our best tool to solve many problems.

[Alex Andorra]: 00:15:42

Mm-hmm.

[Charles Margossian]: 00:15:45

And here I'm trying to understand some more fundamental questions,

[Charles Margossian]: 00:15:48

right? So people often say, well, you know, yeah, MCMC is great but

[Charles Margossian]: 00:15:54

it's too computationally expensive. That's why we'll

[Alex Andorra]: 00:15:57

Yeah.

[Charles Margossian]: 00:15:58

use approximate methods. And one thing I will argue is actually it's

[Charles Margossian]: 00:16:04

computationally expensive, but we don't really have a very good sense

[Charles Margossian]: 00:16:08

of how much computation we should throw at MCMC. Because ultimately

[Charles Margossian]: 00:16:15

we have three fundamental tuning parameters. One is the number of chains

[Charles Margossian]: 00:16:20

that we use. The other one is how long is the warmup or the burning

[Charles Margossian]: 00:16:26

phase? And then the third one is how long is the sampling phase? And

[Charles Margossian]: 00:16:31

actually,

[Charles Margossian]: 00:16:34

it's not clear what is the optimal computation that you should throw

[Charles Margossian]: 00:16:39

at an MCMC problem. I think people rely on heuristics. More often,

[Charles Margossian]: 00:16:44

they rely on conventions. These are the defaults in Stan, in PMC, in

[Charles Margossian]: 00:16:49

TensorFlow probability. But actually, you know, we need to think about

[Charles Margossian]: 00:16:54

have we used a warm up phase that's too long or too short or

[Alex Andorra]: 00:16:59

Mm-hmm.

[Charles Margossian]: 00:17:00

sampling phase that's too long or too short, right?

[Alex Andorra]: 00:17:02

Yeah.

[Charles Margossian]: 00:17:02

Or how many chains is it useful to do? Especially now we have GPUs,

[Charles Margossian]: 00:17:05

right? In theory, I could run MCMC with 10,000 chains. And people do

[Charles Margossian]: 00:17:12

that now, right? I mean, some people do that. It selects you do that

[Charles Margossian]: 00:17:16

now, right? But what are the implications of doing that, right? I'll

[Charles Margossian]: 00:17:19

tell you an implication. If I have 10,000 chains, that I'm running on

[Charles Margossian]: 00:17:23

a GPU. then I think the sampling phase can be one iteration per chain.

[Charles Margossian]: 00:17:32

We can discuss this and we can argue for or against it, but it changes

[Charles Margossian]: 00:17:36

a little bit our perspective of how much computation we throw at it. Another

[Charles Margossian]: 00:17:42

perspective I like is if you have a strict computational budget,

[Charles Margossian]: 00:17:47

and then you say, well, I'm not going to use MCMC, it's too expensive.

[Charles Margossian]: 00:17:50

Let me run variation inference. I know it's biased. I know it's approximate,

[Charles Margossian]: 00:17:53

but at least it, it finishes running in, you know, within 10 minutes and

[Charles Margossian]: 00:17:59

like, well, run 10 minutes of MCMC. It'll be biased. It'll be approximate

[Charles Margossian]: 00:18:03

and it will finish running in 10 minutes. And then ask yourself, well,

[Charles Margossian]: 00:18:07

how good is this estimate? thinking a little bit more carefully about,

[Charles Margossian]: 00:18:12

you know, how much computation do we really need for MCMC and for the different

[Charles Margossian]: 00:18:18

problems that we might be trying to solve.

[Alex Andorra]: 00:18:20

Mm-hmm.

[Alex Andorra]: 00:18:24

Yeah, that's really interesting. Thanks a lot for that very clear presentation.

[Alex Andorra]: 00:18:29

I really love it. It's actually continue on that path because I wanted to ask

[Alex Andorra]: 00:18:35

you about that a bit later in the show anyways. Yeah, several things that

[Alex Andorra]: 00:18:42

bumped into my mind. First thing is MCMC is also an approximation method. So...

[Alex Andorra]: 00:18:52

Why

[Charles Margossian]: 00:18:52

Yes.

[Alex Andorra]: 00:18:52

do we say, why do we, and I know we usually do that in the field, we define

[Alex Andorra]: 00:18:57

variational inference as an approximation method, which kind of underlies,

[Alex Andorra]: 00:19:03

assume that MCMC is not, but it is. So can you maybe draw the distinction?

[Alex Andorra]: 00:19:07

What makes the difference between the two methods? And why do we call them approximation

[Alex Andorra]: 00:19:12

for variation?

[Charles Margossian]: 00:19:15

Yeah, absolutely. So I think that. people think like a lot of statisticians

[Charles Margossian]: 00:19:21

asymptotically. So you know that asymptotically in what sense when

[Charles Margossian]: 00:19:26

you run a single chain for an infinite number of iterations, MCMC

[Charles Margossian]: 00:19:32

is not only gonna generate samples from stationary distribution, which

[Charles Margossian]: 00:19:38

oftentimes is the posterior distribution, but also multicolor estimators

[Charles Margossian]: 00:19:43

with an arbitrary precision. Your multicolor estimator will converge true

[Charles Margossian]: 00:19:50

expectation value or the true variance or whatever it is you're trying

[Charles Margossian]: 00:19:54

to estimate. Whereas with variational inference, here we have to be a

[Charles Margossian]: 00:19:59

little bit careful because what does it mean for the asymptotic of variation

[Charles Margossian]: 00:20:06

inference? So you might say, okay, I'm gonna run the optimization for

[Charles Margossian]: 00:20:09

an infinite number of iterations. So let's assume that the optimizer

[Charles Margossian]: 00:20:14

does converge. And actually until recently, this was not really shown.

[Charles Margossian]: 00:20:20

There's a recent pre-print that I know Robert Gower and Justin Dumkey

[Charles Margossian]: 00:20:25

have been working on where they actually showed that, yes, under certain

[Charles Margossian]: 00:20:29

conditions, stochastic optimization will converge for variation inference.

[Alex Andorra]: 00:20:34

Mm-hmm.

[Charles Margossian]: 00:20:36

But then even if it doesn't converge, you say, well, let me think about

[Charles Margossian]: 00:20:41

the approximation that minimizes my objective function. So oftentimes

[Charles Margossian]: 00:20:45

the Colbert library divergence. That's what I get asymptotically. Well, that

[Charles Margossian]: 00:20:51

will still not be in general, my target distribution.

[Alex Andorra]: 00:20:55

Mm-hmm.

[Charles Margossian]: 00:20:57

Even asymptotically, I'm still approximate. I think that's why people

[Charles Margossian]: 00:21:01

draw this distinction between MCMC and variational inference. It's

[Charles Margossian]: 00:21:05

really in the asymptotic sense that MCMC is an exact method, whereas

[Charles Margossian]: 00:21:10

VI remains approximate. Even if you've thrown an infinite amount of

[Charles Margossian]: 00:21:14

computation, you're probably not. Right? Now, in practice, we're not asymptotic,

[Charles Margossian]: 00:21:21

right? We work with finite computation. And so I think it's very important

[Charles Margossian]: 00:21:27

to recognize that, yes, MCMC is also an approximate method. It's not

[Charles Margossian]: 00:21:32

unbiased because you don't initialize from the stationary distribution.

[Charles Margossian]: 00:21:37

You do not reach the stationary distribution, right? So when I hear statements

[Charles Margossian]: 00:21:42

like, well, first we wait for MCMC to converge to the stationary distribution.

[Alex Andorra]: 00:21:47

Mm-hmm.

[Charles Margossian]: 00:21:48

I think people have the right intuition, right? I don't think it's

[Charles Margossian]: 00:21:50

a misleading statement, but nonetheless, it's an incorrect statement.

[Charles Margossian]: 00:21:55

What we do is we wait for MCMC to get close enough to their stationary

[Charles Margossian]: 00:21:59

distribution. And the reason why we care about being close enough

[Charles Margossian]: 00:22:02

to the stationary distribution is because we want the bias to be small, right?

[Charles Margossian]: 00:22:08

And so

[Alex Andorra]: 00:22:08

Thank you.

[Charles Margossian]: 00:22:08

when we think about, you know, convergence diagnostic. the way I think

[Charles Margossian]: 00:22:16

we really should start thinking about a quantity like the r hat statistics,

[Charles Margossian]: 00:22:20

for example. And r hat is interesting. r hat has been around for three

[Charles Margossian]: 00:22:24

decades and frankly, there's still debates about what does r hat measure.

[Charles Margossian]: 00:22:32

I mean, this is very existential, right? We have an estimator, but it's not clear

[Charles Margossian]: 00:22:36

what the estimate is. And my perspective, my most recent perspective

[Charles Margossian]: 00:22:43

is that what really matters, the reason I care about convergence is because

[Charles Margossian]: 00:22:46

I want the bias of my Monte Carlo estimator to be sufficiently small.

[Charles Margossian]: 00:22:50

It's not going to be zero, but it has to be sufficiently small. And so

[Charles Margossian]: 00:22:55

can R hat tell me something about how small my bias is? And here's the

[Charles Margossian]: 00:23:02

paradox, which is that R hat the way you compute it, it's a ratio

[Charles Margossian]: 00:23:08

of two standard deviations. Right. So you know that when you measure variance,

[Charles Margossian]: 00:23:12

it doesn't tell you something about bias. Right. And yet, you know, we say

[Charles Margossian]: 00:23:18

our hat tells you if your warmup phase is long enough.

[Alex Andorra]: 00:23:21

Yep.

[Charles Margossian]: 00:23:21

Not everyone agrees that that's what it tells you, but you know, that's

[Charles Margossian]: 00:23:24

my perspective and, um, and I think it's a reasonable perspective,

[Charles Margossian]: 00:23:29

right? But the, the point of the warmup phase, you know, at a, the primary

[Charles Margossian]: 00:23:33

point, not the only point is for the bias to go down. Right? So we're

[Charles Margossian]: 00:23:38

faced with this paradox, this very fundamental question. Can R hat actually

[Charles Margossian]: 00:23:43

give us any useful information? And there was a recent paper that argued that,

[Charles Margossian]: 00:23:48

well, since R hat is really just looking at the variance, it's a one-to-one

[Charles Margossian]: 00:23:53

map. So this was some really nice work by,

[Charles Margossian]: 00:24:01

I know that Kida Vats is one of the co-authors on the paper, and

[Charles Margossian]: 00:24:04

there's another co-author. They call it revisiting the Gellman-Rubin statistic.

[Alex Andorra]: 00:24:09

Mm-hmm.

[Charles Margossian]: 00:24:09

And they argue, well, R hat is just a reframing of variance and of

[Charles Margossian]: 00:24:12

effective sample size. And to me, effective sample size only matters

[Charles Margossian]: 00:24:18

when you're looking at the sampling phase, because it tells you, is

[Charles Margossian]: 00:24:21

your variance low enough? And so we had to think a little bit hard about

[Charles Margossian]: 00:24:26

this because it wasn't completely satisfactory because either it means

[Charles Margossian]: 00:24:30

that r hat is not a useful convergence diagnostic in some sense, or actually

[Charles Margossian]: 00:24:38

there's more going on. And what we realize is you look at the variance

[Charles Margossian]: 00:24:45

that's being measured by r hat. So really what r hat ends up measuring

[Charles Margossian]: 00:24:48

is you're running a bunch of chains. It could be four chains, it could

[Charles Margossian]: 00:24:52

be more. Each chain generates one Monte Carlo estimator. And then you average

[Charles Margossian]: 00:25:00

the per chain Monte Carlo estimator. So now you look at the variance

[Charles Margossian]: 00:25:04

of a single chain Monte Carlo estimator, that variance you can actually

[Charles Margossian]: 00:25:08

decompose it into a non-stationary variance and a persistent variance. And what

[Charles Margossian]: 00:25:13

we realize, and you have to be careful, you have to do that analysis

[Charles Margossian]: 00:25:17

for non-stationary Markov chains. Otherwise

[Alex Andorra]: 00:25:19

Mm-hmm.

[Charles Margossian]: 00:25:19

you completely miss the non-stationary variance.

[Alex Andorra]: 00:25:22

Mm-hmm.

[Charles Margossian]: 00:25:23

And the non-stationary variance is, you know, actually, it's a measure

[Charles Margossian]: 00:25:31

of how well you've forgotten your initial point.

[Alex Andorra]: 00:25:34

Mm-hmm.

[Charles Margossian]: 00:25:35

And you can show in some cases that it decays at the same rate as the

[Charles Margossian]: 00:25:38

squared bias. So you're not directly measuring the squared bias, but

[Charles Margossian]: 00:25:42

because NCMC is what it is, the non-stationary variance gives you a

[Charles Margossian]: 00:25:47

proxy clock for the bias. And so our argument is that, well, what's interesting

[Charles Margossian]: 00:25:52

about our hat is not that it measures, you know, the persistent variance,

[Charles Margossian]: 00:25:57

which you can then relate to the effective sample size, but that it measures

[Charles Margossian]: 00:26:02

the non-stationary variance, right? And this then led us to, you know, we're

[Charles Margossian]: 00:26:05

coming up with revisions of our hat, which more directly measure the

[Charles Margossian]: 00:26:10

non-stationary variance rather than the total variance. So that we

[Charles Margossian]: 00:26:13

actually get an estimator that unambiguously tells you something about

[Charles Margossian]: 00:26:18

the length of the warm up phase. And then you ask the question of

[Charles Margossian]: 00:26:22

the length of the sampling phase in a second and separate state.

[Charles Margossian]: 00:26:30

So, sorry, it's conceptual explanation. So that this is a paper that we

[Charles Margossian]: 00:26:36

have a preprint out called the nested r hat.

[Alex Andorra]: 00:26:39

Mm-hmm.

[Charles Margossian]: 00:26:40

And this is joint work with, so Andrew Gellman, Akiva Terry, Matt

[Charles Margossian]: 00:26:44

Hoffman, so some of the usual suspects. We also have Pavel Junssov

[Charles Margossian]: 00:26:48

and Yonel Viyouduran.

[Alex Andorra]: 00:26:50

Mm-hmm.

[Charles Margossian]: 00:26:50

So we've all worked on this together. And

[Charles Margossian]: 00:26:57

A cool anecdote here is that what we were really interested in is those

[Charles Margossian]: 00:27:03

regimes where we're running hundreds of chains in parallel, or thousands

[Charles Margossian]: 00:27:06

of chains in parallel, like GPU-friendly MCMC. And what this made

[Charles Margossian]: 00:27:12

us do is instead of thinking asymptotics in the limit where we have an infinitely

[Charles Margossian]: 00:27:18

long chains, which is how asymptotics for MCMC have worked for the

[Charles Margossian]: 00:27:22

past, you know, five or six decades, right? because that's what people

[Charles Margossian]: 00:27:26

did. They ran long chains, right?

[Alex Andorra]: 00:27:28

Yep.

[Charles Margossian]: 00:27:29

And so even though asymptotics are property of infinity, we want to somehow

[Charles Margossian]: 00:27:36

get close to asymptotic regime, right? That's where we care about this

[Charles Margossian]: 00:27:39

asymptotic analysis. And here we thought, well, let's take asymptotics

[Charles Margossian]: 00:27:43

in another direction. Let's say we have a finite number of chains,

[Charles Margossian]: 00:27:47

but what happens when we have an infinite number of chains? And then

[Charles Margossian]: 00:27:51

suddenly you can do asymptotic analysis. on non-stationary Markov chains.

[Charles Margossian]: 00:27:56

Right? So the problem is if I take an asymptotic in the length of

[Charles Margossian]: 00:28:01

the chain, well, I've made my chain stationary. And then there are

[Charles Margossian]: 00:28:03

only so many properties that I can study. If I take asymptotics in

[Charles Margossian]: 00:28:08

the other direction, which is the number of chains, and

[Alex Andorra]: 00:28:10

Mm-hmm.

[Charles Margossian]: 00:28:10

suddenly I can start making statements about non-stationary Markov chains,

[Charles Margossian]: 00:28:14

I can elicit terms such as, you know, non-stationary variance. And

[Charles Margossian]: 00:28:19

this was a cool example where the hardware you know, trying to work

[Charles Margossian]: 00:28:24

with, you know, GPUs running a lot of chains. Actually, I think is really

[Charles Margossian]: 00:28:29

changing our theoretical approach and our conceptual understanding of

[Charles Margossian]: 00:28:36

MCMC. And I think we're going to get a lot of breakthroughs from this

[Charles Margossian]: 00:28:40

kind of perspective.

[Alex Andorra]: 00:28:41

Hmm. Yeah, super interesting. I put the two papers you mentioned in the show

[Alex Andorra]: 00:28:47

notes. And so that makes me think, basically, what would you say right

[Alex Andorra]: 00:28:56

now practically for people, when would variational inference usually be

[Alex Andorra]: 00:29:04

most helpful, especially in comparison to MCMC?

[Charles Margossian]: 00:29:12

Yeah, so great. So I think there are two things to unpack here. One is

[Charles Margossian]: 00:29:19

when can variational inference still give you accurate answers?

[Alex Andorra]: 00:29:23

Mm-hmm.

[Charles Margossian]: 00:29:24

And two, when do you actually not need an accurate answer?

[Charles Margossian]: 00:29:29

Right? So, in the first question, it really depends on the family of

[Charles Margossian]: 00:29:33

variation and inference that you're willing to use, the family of approximation.

[Charles Margossian]: 00:29:38

And also it turns out the objective function. So I mentioned earlier

[Charles Margossian]: 00:29:42

that when I have this mean field approximation, and I'll just remind

[Charles Margossian]: 00:29:46

what mean field means is, I'm assuming that all my latent variables

[Charles Margossian]: 00:29:50

are independent. Now we don't believe that that's true in practice,

[Charles Margossian]: 00:29:54

but that makes the computation much cheaper. And when you have, millions of

[Charles Margossian]: 00:29:59

observations, millions of parameters, you need an algorithm where

[Charles Margossian]: 00:30:03

the cost scales linearly with the number of observations. And if I don't

[Charles Margossian]: 00:30:07

have this mean field assumption, I get things that can scale quadratically or

[Charles Margossian]: 00:30:12

cubically.

[Charles Margossian]: 00:30:16

And so when I do this approximation, well, I'm gonna get some things

[Charles Margossian]: 00:30:20

wrong, but maybe I'll get the things that I care about right, which

[Alex Andorra]: 00:30:23

Mm-hmm.

[Charles Margossian]: 00:30:23

could be the first moment or it could be the entropy, right? if you change

[Charles Margossian]: 00:30:30

the objective function, which actually the KL divergence, you can kind

[Charles Margossian]: 00:30:33

of reverse it. You can show that you get arbitrarily poor estimates

[Charles Margossian]: 00:30:38

of the entropy, but good estimates of the marginal variances. Right,

[Charles Margossian]: 00:30:43

so it turns out that the choice of objective function that you use to

[Charles Margossian]: 00:30:47

measure, you know, the disagreement between your approximation and

[Charles Margossian]: 00:30:50

your target matters. So there's a real question of, you know, what

[Charles Margossian]: 00:30:55

are the quantities you care about? because we don't care about the

[Charles Margossian]: 00:30:58

whole distribution. We care about some summaries of the posterior

[Charles Margossian]: 00:31:01

distribution.

[Alex Andorra]: 00:31:03

Mm-hmm.

[Charles Margossian]: 00:31:03

And then that informs when you use them. So I think that's the first

[Charles Margossian]: 00:31:06

question. And

[Alex Andorra]: 00:31:07

Mm-hmm.

[Charles Margossian]: 00:31:07

I wanna emphasize that there's a lot of great work by Tamara Broderick

[Charles Margossian]: 00:31:11

and her group, Justin Dumke and his colleagues about trying to get

[Charles Margossian]: 00:31:19

more accurate variation inference. And sometimes that works really

[Charles Margossian]: 00:31:23

well. Which is a bit of a... I can't really give you a more precise

[Charles Margossian]: 00:31:29

prescription than that, because we have to go into the details of the

[Charles Margossian]: 00:31:32

different problems.

[Charles Margossian]: 00:31:36

But then the second point I made is sometimes you don't need a really

[Charles Margossian]: 00:31:39

accurate answer. So what are examples of that? So in machine learning,

[Charles Margossian]: 00:31:45

let's say you're just training a model, and maybe you're more interested

[Charles Margossian]: 00:31:48

in either a more complicated model or using more data than improving the

[Charles Margossian]: 00:31:53

accuracy of the inference. And then you

[Alex Andorra]: 00:31:55

Mm-hmm.

[Charles Margossian]: 00:31:55

look at something like performing a task, a classification prediction

[Charles Margossian]: 00:32:00

and so forth under a computational budget,

[Alex Andorra]: 00:32:03

Mm-hmm.

[Charles Margossian]: 00:32:03

right? It turns out that it's better to have a sophisticated model with

[Charles Margossian]: 00:32:09

very approximate inference than a less sophisticated model with more accurate

[Charles Margossian]: 00:32:14

inference.

[Alex Andorra]: 00:32:15

Mm-hmm.

[Charles Margossian]: 00:32:15

Now, it's hard to know, and my big problem with variation inference

[Charles Margossian]: 00:32:20

is it's hard to know which regime you're gonna be in. and to actually

[Charles Margossian]: 00:32:24

justify it. And even once you've run variation inference, you don't have

[Charles Margossian]: 00:32:29

that many diagnostics that can tell you, well, you know, you're doing

[Charles Margossian]: 00:32:34

okay, but you would do much better if you improve the inference. So

[Charles Margossian]: 00:32:37

I think that it's an open question. But then the other example that I

[Charles Margossian]: 00:32:43

wanna bring up where we don't always need accurate inference is when

[Charles Margossian]: 00:32:47

we're developing models. So this takes back into this idea of the Bayesian

[Charles Margossian]: 00:32:52

workflow. Uh, that now has been championed. Uh, you know, so Andrew

[Charles Margossian]: 00:32:57

Gellman and colleagues wrote a lot about it. Michael Bettencourt wrote

[Charles Margossian]: 00:33:01

a lot about it. David Bly wrote about a lot about it. You know, arguably

[Charles Margossian]: 00:33:04

George Box, right. Wrote a lot about it. And, um, and you know, if you,

[Charles Margossian]: 00:33:12

if you ever work on an applied project, you sit down, you come up

[Charles Margossian]: 00:33:15

with the first iteration of your model and, uh, arguably there are a

[Charles Margossian]: 00:33:20

lot of problems with that model. And you don't need super accurate inference

[Charles Margossian]: 00:33:26

to diagnose the problems with this model. You do a quick fit, a quick

[Charles Margossian]: 00:33:29

approximation, and usually something obvious is gonna pop up.

[Alex Andorra]: 00:33:33

Mm-hmm.

[Charles Margossian]: 00:33:33

Then you revise the model. Then you do again, quick inference, okay? And

[Charles Margossian]: 00:33:38

you keep refining and refining. And only once you have actually a

[Charles Margossian]: 00:33:41

polished version of the model, do I think that it makes sense to, you

[Charles Margossian]: 00:33:44

know, get out the big gun and the very accurate inference. And I

[Charles Margossian]: 00:33:49

think that, you know, if we talk about pharmacometrics and epidemiology,

[Charles Margossian]: 00:33:52

I'll give you some very precise examples of those situations.

[Alex Andorra]: 00:33:57

Yeah, for sure. We'll do that in a few minutes. But yeah, thanks a lot for

[Alex Andorra]: 00:34:04

that tour. That makes a lot of sense, actually, all of that. I will use

[Charles Margossian]: 00:34:09

I realize my

[Alex Andorra]: 00:34:09

those.

[Charles Margossian]: 00:34:09

answers are very long, but your questions are very deep.

[Alex Andorra]: 00:34:13

Yeah, yeah, yeah. No, no, that's really good. I mean, that's why the podcast

[Alex Andorra]: 00:34:17

is for also, you know, going deep into the explanation that you cannot really

[Alex Andorra]: 00:34:24

do in a paper, right? In the papers, usually it's a more technical audience

[Alex Andorra]: 00:34:29

first.

[Charles Margossian]: 00:34:30

Mm-hmm.

[Alex Andorra]: 00:34:31

And so like, you're not going to really explain the difference between volitional

[Alex Andorra]: 00:34:35

inference and MCMC in a paper, because if the audience already is supposed

[Alex Andorra]: 00:34:38

to know that, well, why would you do that in a paper? So.

[Charles Margossian]: 00:34:41

Yeah, and frankly the paper, so what ends up being the discussion

[Charles Margossian]: 00:34:47

that ends up being in the paper is usually the discussion you've

[Charles Margossian]: 00:34:50

had with the reviewer, which is a subset, which is really a subset

[Charles Margossian]: 00:34:55

of everything you would like to discuss. So, yeah, it's nice to have

[Charles Margossian]: 00:35:00

a more free format

[Alex Andorra]: 00:35:02

Yeah.

[Charles Margossian]: 00:35:02

to really think about those questions. And what I will say is that

[Charles Margossian]: 00:35:07

actually the one format that was... that I really like where all these

[Charles Margossian]: 00:35:11

questions come up is, you know, is teaching, is like workshops.

[Alex Andorra]: 00:35:14

Yeah.

[Charles Margossian]: 00:35:15

And so when you do a workshop on Bayesian modeling, you do a workshop on

[Charles Margossian]: 00:35:19

Stan, on PMC or something like that, all the questions that I've brought

[Charles Margossian]: 00:35:25

up, you know, how long should the sampling phase be? How long should

[Charles Margossian]: 00:35:28

the warmup phase be? Should I use this or that algorithm? Those are questions

[Charles Margossian]: 00:35:32

that a person sitting at a workshop Intro to Stan, intro to your

[Charles Margossian]: 00:35:40

favorite language would ask. And so these end up being forums where

[Charles Margossian]: 00:35:46

we do discuss these fundamental questions. Because even though they're deep,

[Charles Margossian]: 00:35:53

they're elementary nonetheless. And I mean that in the most positive sense

[Charles Margossian]: 00:35:57

of the word elementary possible.

[Alex Andorra]: 00:35:59

Yeah, now completely you have completely outmasked the way I pick questions

[Alex Andorra]: 00:36:09

for the show. I'm just doing the same as you did. Yeah, I teach a lot of

[Alex Andorra]: 00:36:14

workshops too. And these questions are basically the questions that a lot of beginners

[Alex Andorra]: 00:36:19

ask, where it's like they often have used variational inference because for

[Alex Andorra]: 00:36:25

some reason, especially when they come from the classical machine learning

[Alex Andorra]: 00:36:29

world, then using variational inference makes more sense to them because

[Alex Andorra]: 00:36:34

it's closer from home, basically

[Charles Margossian]: 00:36:36

Mm-hmm.

[Alex Andorra]: 00:36:36

closer to home. And so yeah, like in the end the natural question is when

[Alex Andorra]: 00:36:41

should I use variational inference? Why should I use it? Why should I even bother

[Alex Andorra]: 00:36:45

with MCMC? Things like that. So I have a lot of the questions I've been

[Charles Margossian]: 00:36:50

Yeah.

[Alex Andorra]: 00:36:50

asking.

[Charles Margossian]: 00:36:53

And these are totally open questions. I mean, correct me. Maybe you

[Charles Margossian]: 00:36:56

have an answer that I missed. You know, we have heuristics and we

[Charles Margossian]: 00:37:02

have good pointers and we have good case studies. But so much of this

[Charles Margossian]: 00:37:08

remains unanswered. I think

[Alex Andorra]: 00:37:11

Yeah, yeah,

[Charles Margossian]: 00:37:12

not

[Alex Andorra]: 00:37:13

yeah.

[Charles Margossian]: 00:37:13

unapproachable. Let me be clear. Totally approachable. And you can,

[Charles Margossian]: 00:37:16

you know, it's not crippling. We can totally still do a Bayesian modeling.

[Charles Margossian]: 00:37:21

But there are open questions that linger.

[Alex Andorra]: 00:37:24

Yeah. And then you have to strike the right balance between when you're answering

[Alex Andorra]: 00:37:30

such a question from a beginner, where you want to be like intellectually honest

[Alex Andorra]: 00:37:38

and saying,

[Charles Margossian]: 00:37:38

Mm-hmm.

[Alex Andorra]: 00:37:39

yeah, like these are still open questions, but at the same time, you don't

[Alex Andorra]: 00:37:42

want them to walk away with the feeling that, well, this is completely...

[Alex Andorra]: 00:37:50

like is completely undefined and I cannot really use these methods because

[Alex Andorra]: 00:37:55

no, um, there is no clear rule of about what I would use in when

[Charles Margossian]: 00:38:00

Mm-hmm.

[Alex Andorra]: 00:38:01

and why. So, uh, like that's always a, an important balance to strike and

[Alex Andorra]: 00:38:07

not, not always an easy one.

[Charles Margossian]: 00:38:09

Yeah, yeah, I absolutely relate to that.

[Alex Andorra]: 00:38:15

Yeah, so actually before we dig into a bit more of the applications, I'm

[Alex Andorra]: 00:38:26

curious basically on the work you're doing right now because I really love

[Alex Andorra]: 00:38:32

the fact that you're both working on MCMC and on approximate patient inference.

[Alex Andorra]: 00:38:37

So I'm wondering what are the frontiers? currently in that field of algorithms

[Alex Andorra]: 00:38:46

that you find particularly excited about. You already mentioned basically the

[Alex Andorra]: 00:38:52

progress of the hardware, which opens a lot of avenues. I'm curious if there

[Alex Andorra]: 00:38:58

are other things you have your eye on.

[Charles Margossian]: 00:39:04

Yeah, I think that...

[Charles Margossian]: 00:39:12

So. Well, I think hardware is an important question. And I think it's

[Charles Margossian]: 00:39:24

a difficult question.

[Alex Andorra]: 00:39:26

Mm-hmm.

[Charles Margossian]: 00:39:26

And I wanna talk, I think, you know, maybe I wanna say a little bit more

[Charles Margossian]: 00:39:29

because I think

[Alex Andorra]: 00:39:29

Mm-hmm.

[Charles Margossian]: 00:39:29

frontier is good. And not only that, I think it's an ambiguous frontier.

[Alex Andorra]: 00:39:34

Mm-hmm.

[Charles Margossian]: 00:39:35

Because I don't think, you know, to me, it's not clear. you know,

[Charles Margossian]: 00:39:40

how much we're gonna get out of hardware for MCMC, for example,

[Charles Margossian]: 00:39:49

and you know, what are gonna be the limits of that? And so what I'm

[Charles Margossian]: 00:39:54

excited is that now we have GPUs, now we have several algorithms that

[Charles Margossian]: 00:40:01

are GPU friendly. And I'll explain a little bit what that means,

[Charles Margossian]: 00:40:06

but essentially, So at a very fundamental level, what it means is

[Charles Margossian]: 00:40:09

you can run a lot of Markov chains

[Alex Andorra]: 00:40:12

Mm-hmm.

[Charles Margossian]: 00:40:13

in a synchronized time. And so you're not waiting for the slowest chain,

[Charles Margossian]: 00:40:18

basically. That's kind of the intuition here. And people are like,

[Charles Margossian]: 00:40:25

well, this is great. If you run a lot of chains, we can really make

[Charles Margossian]: 00:40:28

the sampling phase much shorter. Like again, so let's say your target

[Charles Margossian]: 00:40:34

effective sample size is 2000. And actually what effective sample

[Charles Margossian]: 00:40:39

size you should target. That's another interesting and very fundamental

[Charles Margossian]: 00:40:42

question. And it turns out a question where different people have

[Charles Margossian]: 00:40:47

different opinions on

[Alex Andorra]: 00:40:48

Mm-hmm.

[Charles Margossian]: 00:40:49

within the Bayesian community. But let's say now for sake of argument, you're

[Charles Margossian]: 00:40:53

targeting an effective sample size of 2000. And I'm not saying this

[Charles Margossian]: 00:40:58

is what you should do or not do, but let's just say 2000. Then once

[Charles Margossian]: 00:41:02

you run 2000 chains and you warm them up enough, right? But really

[Charles Margossian]: 00:41:08

you only need one good sample per chain,

[Alex Andorra]: 00:41:12

Thanks for watching!

[Charles Margossian]: 00:41:12

which means your sampling phase can be one iteration. And then all your

[Charles Margossian]: 00:41:16

questions about how long should the warmup be, it's no longer about

[Charles Margossian]: 00:41:21

adapting the kernels so that I have a low auto correlation during the

[Charles Margossian]: 00:41:25

sampling phase. Actually the auto correlation only matters in so far

[Charles Margossian]: 00:41:30

as it reduces your bias. All right, and so

[Alex Andorra]: 00:41:33

Mm-hmm.

[Charles Margossian]: 00:41:33

I think that suddenly we've greatly simplified the question of diagnosing

[Charles Margossian]: 00:41:40

how much computation we need to throw at the algorithm. If we have

[Charles Margossian]: 00:41:45

a lot of variance, then suddenly it just becomes about bias decay.

[Charles Margossian]: 00:41:48

And the sampling problem becomes much closer to the optimization problem.

[Charles Margossian]: 00:41:54

Right. And then I can go into questions. So then there are interesting

[Charles Margossian]: 00:41:57

things where you actually run a lot of chains. You can pool information

[Charles Margossian]: 00:42:01

between the different chains to

[Alex Andorra]: 00:42:02

Mm-hmm.

[Charles Margossian]: 00:42:02

make the warmup phase shorter. And for some

[Alex Andorra]: 00:42:05

Yeah.

[Charles Margossian]: 00:42:05

problems, like you have these multimodal problems, like it's exponentially

[Charles Margossian]: 00:42:10

shorter, like you're never going to get a good answer with a single chain

[Charles Margossian]: 00:42:15

that doesn't use cross adaptation. Right. And so here, I want to give

[Charles Margossian]: 00:42:18

a shout out to Marie-Lou Gabrielle and her work on MCMC that uses normalizing

[Charles Margossian]: 00:42:24

flow for adaptation.

[Alex Andorra]: 00:42:26

Mm-hmm.

[Charles Margossian]: 00:42:27

Right. That's the thing, that's a technique where they actually, you

[Charles Margossian]: 00:42:30

know, run 10,000 workers or chains. And I remember when I was talking

[Charles Margossian]: 00:42:35

to her and colleagues, I was like, well, you know, once you've, you're

[Charles Margossian]: 00:42:38

already running 10,000 chains to jump between the modes, you actually

[Charles Margossian]: 00:42:42

don't need a long sampling phase at all,

[Alex Andorra]: 00:42:44

Mm-hmm.

[Charles Margossian]: 00:42:44

right? So that's one aspect of it. But then even for more ordinary problems,

[Charles Margossian]: 00:42:52

you can show that the time it takes you to reduce the bias goes down

[Charles Margossian]: 00:42:56

with the number of chains, because you are pooling information between

[Charles Margossian]: 00:43:00

the Markov chains. And this is not something that we really understand.

[Alex Andorra]: 00:43:03

Hmm.

[Charles Margossian]: 00:43:03

Um, and so, you know, where I see the frontier is that actually,

[Charles Margossian]: 00:43:08

if I run a lot of chains, I also get more accurate diagnostics. My computations

[Charles Margossian]: 00:43:11

of our hat and its generalization become much more reliable. And I think the

[Charles Margossian]: 00:43:15

Holy grail would be something where we don't have users specify, um, the

[Charles Margossian]: 00:43:22

length of the warmup phase or the length of the sampling phase. We have

[Charles Margossian]: 00:43:25

them think about what is your target ESS? That's the number of chains

[Charles Margossian]: 00:43:29

that you run. And then we're going to automatically stop the warmup

[Charles Margossian]: 00:43:32

phase. when we hit a certain target, right? And then suddenly,

[Charles Margossian]: 00:43:40

we're starting to do optimal computation for MCMC. And I think that to do

[Charles Margossian]: 00:43:44

optimal computation, at least in the way that I've described it, we

[Charles Margossian]: 00:43:48

need those GPUs. And at the same time, I think that there are a lot

[Charles Margossian]: 00:43:52

of problems that are not gonna be amiable to GPUs, right? It's still,

[Charles Margossian]: 00:43:56

there's still this fundamental sequential component. which is the bias has

[Charles Margossian]: 00:44:02

to go down, the warmup needs to happen, right? At some point, adding

[Charles Margossian]: 00:44:06

more chains is not gonna help you. Whether the speed up you're gonna

[Charles Margossian]: 00:44:11

get on this is not gonna be arbitrarily large, right? And then the benefit

[Charles Margossian]: 00:44:16

you're gonna get from various reduction by running more chains, well,

[Charles Margossian]: 00:44:20

once you've read. If your target ESS is 2000, maybe it doesn't help to

[Charles Margossian]: 00:44:25

run 10,000 chains, right?

[Alex Andorra]: 00:44:27

Yeah.

[Charles Margossian]: 00:44:27

At least not immediately, right? So those are very clear, you know, questions

[Charles Margossian]: 00:44:34

that arise about, you know, ultimately, what are, how far are we

[Charles Margossian]: 00:44:37

going to be able to go with this ES, this, you know, running many

[Charles Margossian]: 00:44:42

chains. But what I want to emphasize is that there's a, you know, there's

[Charles Margossian]: 00:44:45

a computational game, which people think about, our algorithms are

[Charles Margossian]: 00:44:48

faster. I also think there's a conceptual game. And there is the opportunity

[Charles Margossian]: 00:44:55

to make MCMC more black box.

[Alex Andorra]: 00:44:58

Mm-hmm.

[Charles Margossian]: 00:44:59

We're less reliant on some very fundamental tuning algorithms.

[Charles Margossian]: 00:45:06

And therefore, you know, we kind of get into this regime where the computation

[Charles Margossian]: 00:45:12

that we're using is just the right amount. And that's really, you know,

[Charles Margossian]: 00:45:16

if I have to say, like, I have another one or two years as a postdoc,

[Charles Margossian]: 00:45:20

very optimistically, that's the problem I'd like to solve. Right? Like,

[Charles Margossian]: 00:45:24

we remove the fundamental tuning parameters of MCMC. And there are other

[Charles Margossian]: 00:45:28

approaches towards that. I'm not going to pretend that this is the only

[Charles Margossian]: 00:45:32

angle to tackle this problem. Let me be absolutely clear. I think it's a

[Charles Margossian]: 00:45:36

very, very promising.

[Alex Andorra]: 00:45:38

Yeah, yeah,

[Charles Margossian]: 00:45:40

Yeah,

[Alex Andorra]: 00:45:40

for sure. That's fascinating.

[Charles Margossian]: 00:45:41

what I will say is since we're going to go into applications is one

[Charles Margossian]: 00:45:44

limitation

[Alex Andorra]: 00:45:44

Mm-hmm.

[Charles Margossian]: 00:45:44

with GPUs is they're very bad at solving ODEs.

[Alex Andorra]: 00:45:47

Mm-hmm.

[Charles Margossian]: 00:45:48

And a lot of the problems that I care about, you

[Alex Andorra]: 00:45:52

Have

[Charles Margossian]: 00:45:52

know,

[Alex Andorra]: 00:45:52

a day!

[Charles Margossian]: 00:45:52

have likelihoods. And in order to evaluate those likelihoods, you

[Charles Margossian]: 00:45:55

need to solve an OD.

[Alex Andorra]: 00:45:57

Mm-hmm.

[Charles Margossian]: 00:45:58

So I don't think we're at a stage where running a thousand chains

[Charles Margossian]: 00:46:03

on a GPU is going to solve all the problems in pharmacometrics that I

[Charles Margossian]: 00:46:08

am, you know. deeply invested in, right? That said, you know, we have clusters

[Charles Margossian]: 00:46:15

of CPUs where maybe we can run 60 to 120 chains and that can get us

[Charles Margossian]: 00:46:21

some of the way.

[Alex Andorra]: 00:46:24

Hmm, yeah. Yeah, I mean, when you were talking about that, I was thinking

[Alex Andorra]: 00:46:30

that'd be super cool to have, you know, in Stan or Pimcey afterwards, like

[Alex Andorra]: 00:46:35

at some point, optimize the way the number of chains and samples that are

[Alex Andorra]: 00:46:41

taken instead of having them, because right now it's like, okay, we're gonna

[Alex Andorra]: 00:46:45

run as many chains as we can with the GP or CPU we have. So that's

[Charles Margossian]: 00:46:50

Mm-hmm.

[Alex Andorra]: 00:46:50

already kind of automated. But the number of samples not really automated

[Alex Andorra]: 00:46:57

is just a rule of thumb. Where it's like, OK, we think in general, these number

[Alex Andorra]: 00:47:02

of samples work well. In PyMC, it's 1,000 per chain after warming up. But

[Alex Andorra]: 00:47:10

what would be

[Charles Margossian]: 00:47:11

Thanks

[Alex Andorra]: 00:47:11

super

[Charles Margossian]: 00:47:11

for watching!

[Alex Andorra]: 00:47:11

cool is like, OK, PyMC or Stan, see what's there, the resources. And then it's

[Alex Andorra]: 00:47:18

like, OK, so given the complexity of the posterior that we can see right now,

[Alex Andorra]: 00:47:23

we are going

[Charles Margossian]: 00:47:23

Yeah.

[Alex Andorra]: 00:47:23

to run that many chains and that many sample per chain. That'd be super

[Alex Andorra]: 00:47:29

cool, because also that would be something that would be easier for the beginners,

[Alex Andorra]: 00:47:33

because they are

[Charles Margossian]: 00:47:33

Mm-hmm.

[Alex Andorra]: 00:47:34

really, really sometimes very anxious about having a lot of samples, even

[Alex Andorra]: 00:47:41

though you know, no, you don't need 10,000 samples per chain for that simple

[Alex Andorra]: 00:47:47

regression, but it's hard to explain.

[Charles Margossian]: 00:47:50

Yeah, and so what ends up, also what ends up happening, very real

[Charles Margossian]: 00:47:55

experience, and I do it myself, is the simple models that probably

[Charles Margossian]: 00:47:59

don't need that many iterations. I run them for a lot of iterations because

[Charles Margossian]: 00:48:02

that's computationally cheap to do. And then the hard models. where

[Charles Margossian]: 00:48:08

each iteration is very expensive and the posterior distribution is

[Charles Margossian]: 00:48:12

much more complicated, I actually end up, and where I would need more

[Charles Margossian]: 00:48:15

iterations, I end up running less iterations.

[Alex Andorra]: 00:48:18

Yeah.

[Charles Margossian]: 00:48:20

Right, and I think a lot of people will sympathize with that. That's

[Charles Margossian]: 00:48:26

my experience interacting with practitioners. I'll give you another example

[Charles Margossian]: 00:48:31

of things I've seen my colleagues in epidemiology do is that when

[Charles Margossian]: 00:48:34

their model starts getting really complicated, they start using somewhat

[Charles Margossian]: 00:48:38

less over dispersed initializations.

[Alex Andorra]: 00:48:41

Mm-hmm.

[Charles Margossian]: 00:48:42

Even though we know that that's what we need for the convergence diagnostic

[Charles Margossian]: 00:48:46

to be reliable. And I'll tell you a little bit more about that because

[Charles Margossian]: 00:48:50

actually that's another question is like, what does it mean for initialization

[Charles Margossian]: 00:48:53

to be over dispersed? And I have some answers to that are not quite,

[Charles Margossian]: 00:48:58

you know, the regular answers. Um, but that's a huge problem. Like

[Charles Margossian]: 00:49:03

when models get hard and again, those ODE based models. Right? You cannot

[Charles Margossian]: 00:49:09

solve those ODE's if you throw insane parameter values at your model.

[Charles Margossian]: 00:49:16

And so now people have to have to make compromises. And I think that,

[Charles Margossian]: 00:49:20

you know, especially statisticians were a little bit cavalier. We try to, we tend

[Charles Margossian]: 00:49:23

to be conservative, uh,

[Alex Andorra]: 00:49:25

Mm-hmm.

[Charles Margossian]: 00:49:26

because maybe in a way, you know, that's the role of the statistician

[Charles Margossian]: 00:49:28

and the theorist is to be conservative and to be, and to play it safe.

[Charles Margossian]: 00:49:33

But when the safe and the conservative heuristics, um become impractical

[Charles Margossian]: 00:49:39

uh we need to think harder about you know okay if we don't want to be

[Charles Margossian]: 00:49:46

too conservative but we still want to be safe what do we need to do

[Charles Margossian]: 00:49:52

and that's where

[Alex Andorra]: 00:49:53

Mm-hmm.

[Charles Margossian]: 00:49:53

these questions of you know optimal warm-up length optimal sampling

[Charles Margossian]: 00:49:56

phase optimal over dispersed initialization uh really come in into play

[Charles Margossian]: 00:50:03

because if you're too conservative in your prescriptions You might

[Charles Margossian]: 00:50:07

make your editor happy, but actually practitioners are going to have

[Charles Margossian]: 00:50:11

a really hard time following those prescriptions.

[Alex Andorra]: 00:50:17

Yeah,

[Charles Margossian]: 00:50:17

And then they do

[Alex Andorra]: 00:50:18

yeah.

[Charles Margossian]: 00:50:18

things. I'm not saying that they do silly things instead, right? But

[Charles Margossian]: 00:50:24

they do other things that are maybe less principled.

[Alex Andorra]: 00:50:29

Mm-hmm. Yeah. Yeah, fascinating. I could spend the whole episode on these topics.

[Alex Andorra]: 00:50:36

I really love it. Thanks a lot for diving so deep into these, Charles. But

[Alex Andorra]: 00:50:42

let's get a bit more practical here and talk about

[Charles Margossian]: 00:50:46

Uh huh.

[Alex Andorra]: 00:50:46

what you do, basically, with epidemiology and pharmacometrics. So first,

[Alex Andorra]: 00:50:52

can you define pharmacometrics for us? I personally don't know what that is,

[Charles Margossian]: 00:50:58

Yeah.

[Alex Andorra]: 00:50:58

how that differs from epidemiology and what do patient statistics bring to epidemiology

[Alex Andorra]: 00:51:06

and pharmacometrics.

[Charles Margossian]: 00:51:09

Yeah, so pharmacometrics, I mean, the way I would think about pharmacometrics

[Charles Margossian]: 00:51:14

is pharmacometrics is to pharmacology, what econometrics is to economics.

[Alex Andorra]: 00:51:19

Mm-hmm.

[Charles Margossian]: 00:51:19

There's some people who want to emphasize that they're using quantitative

[Charles Margossian]: 00:51:24

methods.

[Alex Andorra]: 00:51:25

Mm-hmm.

[Charles Margossian]: 00:51:27

Now, the particular field of pharmacometrics that I've worked on is called

[Charles Margossian]: 00:51:38

PKPD modeling, so pharmacokinetics and pharmacodynamics. And essentially, what

[Charles Margossian]: 00:51:46

happens is, let's say there's a new treatment that's being developed.

[Charles Margossian]: 00:51:50

So it could be a new drug compound or it could be a number of things.

[Charles Margossian]: 00:51:54

We're usually interested in two questions. One, how does the drug get

[Charles Margossian]: 00:51:57

absorbed and how does it diffuse in the patient's body? That's called

[Charles Margossian]: 00:52:01

the pharmacokinetics. And two, once the drug diffuses in the body,

[Charles Margossian]: 00:52:07

what does it do to the body? And so that includes targeting a disease,

[Charles Margossian]: 00:52:14

but also side effects. I was worried about how toxic the treatment

[Charles Margossian]: 00:52:20

might be. And what people are trying to do with these models is based

[Charles Margossian]: 00:52:27

on early data from clinical trials. So either on human beings or even on

[Charles Margossian]: 00:52:32

animals, they try to predict. what is going to happen when we look at a broader

[Charles Margossian]: 00:52:37

population of individuals and when we kind of like start changing the

[Charles Margossian]: 00:52:43

treatment. So you have some, you know, some medical treatments can

[Charles Margossian]: 00:52:49

get very complicated. You have questions of, you know, you know,

[Charles Margossian]: 00:52:53

I have a certain drug compound. How much do I administer? How often

[Charles Margossian]: 00:52:59

do I administer it? Is it better to take, you know, half a dose? every

[Charles Margossian]: 00:53:04

half hour or only a single dose every hour. And so you have all these

[Charles Margossian]: 00:53:10

combinatorics of possibilities. If I really increase the dose, do I immediately

[Charles Margossian]: 00:53:16

get better effects? Or does it saturate? We often have these nonlinear

[Charles Margossian]: 00:53:22

effects. These are called Michaelis-Menten models, where at some point,

[Charles Margossian]: 00:53:28

adding more dose erases the cost and doesn't help the patient.

[Alex Andorra]: 00:53:34

Yeah.

[Charles Margossian]: 00:53:36

Right. And one way to do it would be

[Charles Margossian]: 00:53:42

you run a ton of clinical trials and then you hope that something works

[Charles Margossian]: 00:53:46

out that's extremely expensive and time consuming and, you know, well, Maybe

[Charles Margossian]: 00:53:53

it's safer for the humans than the animals. Let me put it this way. Or

[Charles Margossian]: 00:54:01

based on a bit of data, and then a really mechanistic model where you

[Charles Margossian]: 00:54:06

actually really bake in some of your expertise as a pharmacologist,

[Charles Margossian]: 00:54:13

as a biomedical engineer, you try to understand the underlying system

[Charles Margossian]: 00:54:17

so that when you're trying out different regimens, you can really

[Alex Andorra]: 00:54:20

Mm-hmm.

[Charles Margossian]: 00:54:20

predict. what are gonna be the doses that are more promising? So it's

[Charles Margossian]: 00:54:25

very useful to do exploratory analysis. And then it's also useful to do,

[Charles Margossian]: 00:54:32

once a drug hits the market, you actually collect very imperfect data, you

[Charles Margossian]: 00:54:36

actually have uncertainty, but you still want to keep learning about the

[Charles Margossian]: 00:54:40

dose and the dosing regimen, right, from

[Alex Andorra]: 00:54:42

Mm-hmm.

[Charles Margossian]: 00:54:42

data from hospitals. And sometimes the data is rare, you have rare diseases,

[Charles Margossian]: 00:54:46

rare conditions, and that kind of thing. And so that is... If you will,

[Charles Margossian]: 00:54:50

then...

[Charles Margossian]: 00:54:54

the domain of pharmacometrics. And what's extremely interesting is that within

[Charles Margossian]: 00:54:59

that domain, you have, so first of all, you have what's called mechanistic

[Charles Margossian]: 00:55:04

models. And what I mean by that is the parameters are interpretable.

[Charles Margossian]: 00:55:07

The relationship between models

[Alex Andorra]: 00:55:08

Mm-hmm.

[Charles Margossian]: 00:55:08

is interpretable. Now, at contrast that with a neural network, for

[Charles Margossian]: 00:55:11

example, where I might get good predictions, but then if I want to

[Charles Margossian]: 00:55:15

do out of sample predictions, which is actually really what we want to do,

[Charles Margossian]: 00:55:21

right? in pharmacometrics, right? Examples of other samples would be a

[Charles Margossian]: 00:55:25

different dosing regimen, or I've tested the drug on an adult population.

[Charles Margossian]: 00:55:30

What happens if I do it for children, right? That's the kind of pediatric

[Charles Margossian]: 00:55:34

questions. We need to bake in the mechanistic understanding, which

[Charles Margossian]: 00:55:39

doesn't exclude a role in neural networks. I think these can also

[Charles Margossian]: 00:55:43

play a role, but I'll leave that for now. But then you have various

[Charles Margossian]: 00:55:50

degrees details in the mechanism. You have some equations that are

[Charles Margossian]: 00:55:54

very simple. So the two-compartment model with first-order absorption says the

[Charles Margossian]: 00:55:59

human body is three compartments. There's the gut, where the drug arrives when

[Charles Margossian]: 00:56:03

you already administer it. There is the central compartment where

[Charles Margossian]: 00:56:09

the drug diffuses quickly. So usually that includes the blood. And then

[Charles Margossian]: 00:56:13

maybe there's a peripheral compartment, so tissues where the drug diffuses

[Charles Margossian]: 00:56:17

more slowly. That's obviously not a very detailed description of the human

[Charles Margossian]: 00:56:21

body. And then you have models that are more complicated. So at Matron,

[Charles Margossian]: 00:56:27

Matthew Riggs and some colleagues, they work on this bone mineral

[Charles Margossian]: 00:56:29

density, where actually they had a lot of different parameters. And

[Charles Margossian]: 00:56:34

now instead of having a system of three differential equations, you

[Charles Margossian]: 00:56:37

have 30 differential equations. You have a ton of parameters, but you have

[Charles Margossian]: 00:56:41

a lot of information, prior information, about these parameters.

[Alex Andorra]: 00:56:45

Mm-hmm.

[Charles Margossian]: 00:56:45

And then you have people who really throw differential equations with

[Charles Margossian]: 00:56:51

you know, hundreds of states at them and, you know, thousands of

[Charles Margossian]: 00:56:55

interpretable parameters. And frankly, I don't think we have the Bayesian

[Charles Margossian]: 00:57:00

computation to fit those models, even though in theory, they lend themselves

[Charles Margossian]: 00:57:05

extremely well to a Bayesian analysis, right? I think that realistically

[Charles Margossian]: 00:57:09

we're somewhere in the semi-mechanistic regime. So these are models

[Charles Margossian]: 00:57:14

that have some level of sophistication, but already we pay a dire price

[Charles Margossian]: 00:57:18

for this sophistication, which is that the computation. can take hours

[Charles Margossian]: 00:57:22

or days to fit. And so there's like really this potential for, better

[Charles Margossian]: 00:57:31

Bayesian computation can really allow people to deploy better models

[Charles Margossian]: 00:57:37

and more sophisticated. The other big aspect of pharmacometrics is

[Charles Margossian]: 00:57:43

usually we have trials with data from different patients, there's

[Charles Margossian]: 00:57:48

heterogeneity

[Alex Andorra]: 00:57:48

Mm-hmm.

[Charles Margossian]: 00:57:48

between the patients or similarity

[Alex Andorra]: 00:57:50

Yeah.

[Charles Margossian]: 00:57:50

between patients, so that lends itself very well to hierarchical modeling.

[Alex Andorra]: 00:57:53

Yeah,

[Charles Margossian]: 00:57:54

And

[Alex Andorra]: 00:57:54

for

[Charles Margossian]: 00:57:54

we

[Alex Andorra]: 00:57:54

sure.

[Charles Margossian]: 00:57:54

know hierarchical modeling is hard, right? It tends to create these posterior

[Charles Margossian]: 00:57:59

distributions with geometries that are very frustrating. And I spent a lot

[Charles Margossian]: 00:58:05

of time worrying about hierarchical models. I've done a lot of work

[Charles Margossian]: 00:58:08

on the... nested Laplace approximations. So

[Alex Andorra]: 00:58:11

Mm-hmm.

[Charles Margossian]: 00:58:11

that's another nice example of an approximation. It's not variation inference

[Charles Margossian]: 00:58:16

has very complimentary qualities. And what the nested Laplace approximation

[Charles Margossian]: 00:58:22

allows you to do is marginalize out the latent variables in a hierarchical

[Charles Margossian]: 00:58:26

model.

[Charles Margossian]: 00:58:29

And people often explain nested Laplace as, oh, it's great, it reduces

[Charles Margossian]: 00:58:32

the dimension of your problem. And then we can throw a quadratic

[Charles Margossian]: 00:58:36

at the remaining parameters. We've had models where we had thousands of

[Charles Margossian]: 00:58:43

hyperparameters, those were genetic problems where we were using horseshoe

[Charles Margossian]: 00:58:48

parameters to select Gs. So even once you marginalize out, you know,

[Charles Margossian]: 00:58:54

the latent variables, you still have a high dimensional problem.

[Charles Margossian]: 00:58:56

So we threw Hamiltonian Monte Carlo at it, but it still made a big

[Charles Margossian]: 00:59:01

difference because we simplified the geometry of the posterior distribution

[Charles Margossian]: 00:59:06

by doing this marginalizations for the hierarchical models. So I'm very excited

[Charles Margossian]: 00:59:11

about the prospect of having, you know, and that's the plus approximation

[Charles Margossian]: 00:59:15

in Stan.

[Alex Andorra]: 00:59:16

Thanks

[Charles Margossian]: 00:59:16

We

[Alex Andorra]: 00:59:16

for

[Charles Margossian]: 00:59:16

have

[Alex Andorra]: 00:59:16

watching!

[Charles Margossian]: 00:59:16

a prototype. that works really well. We have some really cool automatic

[Charles Margossian]: 00:59:20

differentiation supporting it.

[Alex Andorra]: 00:59:22

Hmph.

[Charles Margossian]: 00:59:23

But, and you know, the problem is again, I wanna try this on OD based

[Charles Margossian]: 00:59:28

models.

[Alex Andorra]: 00:59:29

Yeah.

[Charles Margossian]: 00:59:30

I don't get a good approximation. I don't get efficient automatic differentiation.

[Charles Margossian]: 00:59:33

I get something that's unstable. Now the simple examples where I got

[Charles Margossian]: 00:59:36

it working, it actually gave surprisingly accurate results, right?

[Charles Margossian]: 00:59:42

But this is again an example where here's this awesome. algorithm and statistical

[Charles Margossian]: 00:59:47

methods, and it just gets frustrated by the nature of the problems we

[Charles Margossian]: 00:59:52

encounter in pharmacometrics. Even though, you know, these are hierarchical

[Charles Margossian]: 00:59:58

models and these methods are designed for hierarchical models. But again,

[Charles Margossian]: 01:00:01

if your likelihood is not a general linear model, yeah, suddenly

[Charles Margossian]: 01:00:06

those approximations become much more tricky. And that's why, that's why,

[Charles Margossian]: 01:00:11

you know, I think that We have to use MCMC for these models.

[Alex Andorra]: 01:00:19

Hmm.

[Alex Andorra]: 01:00:22

Yeah, that's interesting. So in these cases, yeah, that makes it a bit clearer,

[Alex Andorra]: 01:00:27

I think, for people in the practical cases where you would have to do that trade-off,

[Alex Andorra]: 01:00:36

basically. How do you choose between the trade-off between the different

[Alex Andorra]: 01:00:39

methods? I think it's very important.

[Charles Margossian]: 01:00:41

And that said, I do want to say that there's some really cool approximations

[Charles Margossian]: 01:00:45

that people

[Alex Andorra]: 01:00:46

Mm-hmm.

[Charles Margossian]: 01:00:46

do deploy in pharmacometrics.

[Alex Andorra]: 01:00:49

Mm-hmm.

[Charles Margossian]: 01:00:49

I recently read a...

[Charles Margossian]: 01:00:54

I'm an anonymous reviewer, so I'm not going to give too many details.

[Alex Andorra]: 01:00:58

Ha, sure.

[Charles Margossian]: 01:01:00

But what I like, you know, it's, there were questions of what if we fix

[Charles Margossian]: 01:01:05

those parameters or what if, you know, we draw these parameters from

[Charles Margossian]: 01:01:10

their priors because they're removed away enough from the data,

[Alex Andorra]: 01:01:14

Mm-hmm.

[Charles Margossian]: 01:01:15

but maybe they're not influenced that much by the data, so the posterior

[Charles Margossian]: 01:01:18

stays close to the prior, but nonetheless, we need that uncertainty for

[Charles Margossian]: 01:01:22

the interpretable quantities in the middle. Right.

[Alex Andorra]: 01:01:25

Hm-hm.

[Charles Margossian]: 01:01:26

And so people are coming up with these compromises. And of course, now

[Charles Margossian]: 01:01:33

we're again in the business of we have these awesome computational constraints.

[Charles Margossian]: 01:01:39

People come up with these approximations either of the inference or even,

[Charles Margossian]: 01:01:44

you know, they say, well, let's use a simpler model. We know there's

[Charles Margossian]: 01:01:47

a more complicated model out there but maybe we still get all the answers

[Charles Margossian]: 01:01:50

that we need with the simpler model. Right. And so now we get again

[Charles Margossian]: 01:01:54

in this, you know, the question of understanding what are the simplifications

[Charles Margossian]: 01:01:58

that we get away with? What are the ones where we pay a heavy price?

[Charles Margossian]: 01:02:03

Can we actually quantify the price we're paying? Can we diagnose when

[Charles Margossian]: 01:02:07

the simplification is too dire or not? And as far as I can tell,

[Charles Margossian]: 01:02:13

the answer is, you know, we're not at the stage of diagnosing the problem,

[Charles Margossian]: 01:02:18

but at least now people are taking this problem seriously enough that

[Charles Margossian]: 01:02:21

they're building these case studies. Now based on these case studies

[Charles Margossian]: 01:02:25

where, you know, we do try out the different models and we do fit

[Charles Margossian]: 01:02:31

the complicated methods and we're able to say something about the simpler

[Charles Margossian]: 01:02:34

methods because we did fit the complicated methods. So it's an academic

[Charles Margossian]: 01:02:38

exercise in a way.

[Alex Andorra]: 01:02:39

Mm-hmm.

[Charles Margossian]: 01:02:39

But at least it gives us, you know, that's how it starts. And that's

[Charles Margossian]: 01:02:43

how we're going to start developing the intuition and the heuristics

[Charles Margossian]: 01:02:47

to then deal with, you know. deploying the approximations and the

[Charles Margossian]: 01:02:53

simplifications. In practice.

[Alex Andorra]: 01:02:57

Yeah, that makes sense. And so actually, do you have an example of a research

[Alex Andorra]: 01:03:08

project that where you applied patient statistics in this field of pharmacometrics

[Alex Andorra]: 01:03:14

and where really patient statistics helped you uncover very important insights?

[Alex Andorra]: 01:03:24

thanks to all those methods that you've talked about.

[Charles Margossian]: 01:03:28

Yeah. So I've never led a project in pharmacometrics. I've always collaborated

[Charles Margossian]: 01:03:38

with pharmacologists.

[Alex Andorra]: 01:03:40

Mm-hmm.

[Charles Margossian]: 01:03:40

So, you know, and it's true that my work has been more methodological,

[Charles Margossian]: 01:03:45

has been more developing towards in itself. That's it. What I can talk

[Charles Margossian]: 01:03:51

about is some of the interactions I've had with pharmacometricians and some of

[Charles Margossian]: 01:03:55

the works were maybe a more contributor than say, the project lead.

[Charles Margossian]: 01:04:03

And I'll give you one example. And then I think that if we have time

[Charles Margossian]: 01:04:06

and we talk a little bit about epidemiology, I have a very

[Alex Andorra]: 01:04:09

Mm-hmm.

[Charles Margossian]: 01:04:10

good example in epidemiology.

[Alex Andorra]: 01:04:12

Ah, okay, I was...

[Charles Margossian]: 01:04:13

But this is a, this was super cool actually. This is, this is a bit

[Charles Margossian]: 01:04:16

anecdotal.

[Alex Andorra]: 01:04:17

Mm-hmm.

[Charles Margossian]: 01:04:18

I don't know that there's a preprint on this now, but I was in Paris,

[Charles Margossian]: 01:04:21

I was visiting in CERN, which does a lot of medical work. And I was

[Charles Margossian]: 01:04:30

visiting France Mautre's group, and they have fantastic people there.

[Charles Margossian]: 01:04:34

And I'm talking with Julie Bertrand, and she's interested in pharmacogenetics,

[Charles Margossian]: 01:04:43

right? So we're... We have some genome sequencing or some gene tracking

[Charles Margossian]: 01:04:49

that's involved, and we're trying to see how a patient reacts to a treatment,

[Charles Margossian]: 01:04:53

to a certain condition. And in that case, what ended up happening,

[Charles Margossian]: 01:05:01

so one of the things that was frustrating is they're trying to identify

[Charles Margossian]: 01:05:05

what are the genes that are meaningful, that seem to have a meaningful

[Charles Margossian]: 01:05:08

connection to the outcome of the treatments. And they couldn't get anything

[Charles Margossian]: 01:05:15

that is statistically significant in the traditional sense, right?

[Charles Margossian]: 01:05:21

There wasn't one gene or one SNP that unembarrassed bigously stood

[Charles Margossian]: 01:05:27

out as, yes, this is a meaningful snip and it should intervene in

[Charles Margossian]: 01:05:30

oilier analysis.

[Alex Andorra]: 01:05:32

Mm-hmm.

[Charles Margossian]: 01:05:33

And what they had done was a Bayesian analysis where they had used

[Charles Margossian]: 01:05:39

a horseshoe prior.

[Alex Andorra]: 01:05:40

Mm-hmm.

[Charles Margossian]: 01:05:40

And so what the horseshoe prior does is it does like, it's a regularization

[Charles Margossian]: 01:05:46

tool that does a soft selection. And the way that soft selection

[Charles Margossian]: 01:05:50

manifests is there is a quantity that you can look at. and it will

[Charles Margossian]: 01:05:55

usually be bimodal. And one mode will indicate that the covariate that

[Charles Margossian]: 01:06:01

corresponds to this snip is completely

[Charles Margossian]: 01:06:09

regressed to zero. So that's an indication that this is not a very

[Charles Margossian]: 01:06:15

useful explanatory variable. And then the second mode tells you actually

[Charles Margossian]: 01:06:19

this variable matters and it's not regressed to zero. And so why

[Charles Margossian]: 01:06:24

there are two modes is because there is uncertainty. But the very

[Charles Margossian]: 01:06:28

cool thing is that even though there was no single snip that stood

[Charles Margossian]: 01:06:31

out as this is the meaningful snip, uh, you had two snips that came up and

[Charles Margossian]: 01:06:36

they both had a bimodal, right? So what that means is you couldn't definitely

[Charles Margossian]: 01:06:41

say that, uh, either snip mattered, right? But you say, okay, with some

[Charles Margossian]: 01:06:47

probability, either of the snips can matter.

[Alex Andorra]: 01:06:51

Mm-hmm.

[Charles Margossian]: 01:06:51

And here's where it gets very interesting is that actually, because

[Charles Margossian]: 01:06:53

you have a multivariate posterior distribution, you can go a bit further

[Charles Margossian]: 01:06:56

and you realize that the two snips are anti-correlated, right? And so

[Charles Margossian]: 01:07:02

what that means is when you have a lot of posterior mass at one mode

[Charles Margossian]: 01:07:09

for one snip that says, this variable matters, don't regress it to

[Charles Margossian]: 01:07:13

zero, the other covariate would always get regressed to zero

[Alex Andorra]: 01:07:19

Hmm.

[Charles Margossian]: 01:07:19

and vice versa, right?

[Alex Andorra]: 01:07:21

Mm-hmm.

[Charles Margossian]: 01:07:21

So what the multivariate analysis tells you and what this proper treatment

[Charles Margossian]: 01:07:25

of uncertainty tells you is like, yeah, you can't say that one snip

[Charles Margossian]: 01:07:29

is statistically significant in the traditional sense, but now you have

[Charles Margossian]: 01:07:33

this more comprehensive treatment of uncertainty that tells you, but

[Charles Margossian]: 01:07:38

you know what? It has to be one of those two. You can't tell for

[Charles Margossian]: 01:07:42

sure which one, but it has to be one of those two. And that's a nice

[Charles Margossian]: 01:07:45

example where we're really, not just looking at the maximum likelihood

[Charles Margossian]: 01:07:53

estimator or even the expectation value or just a variance, we're

[Charles Margossian]: 01:07:57

really looking at essentially the quantiles, the extreme quantiles

[Alex Andorra]: 01:08:03

Mm-hmm.

[Charles Margossian]: 01:08:04

of the posterior distribution. And excuse me.

[Charles Margossian]: 01:08:13

Sorry about that.

[Charles Margossian]: 01:08:17

we're looking at multiple variables and

[Alex Andorra]: 01:08:20

Mm-hmm.

[Charles Margossian]: 01:08:20

the uncertainty across those multiple variables, right? So I think

[Charles Margossian]: 01:08:23

that's a very neat example and you know. paper to look forward to when

[Charles Margossian]: 01:08:29

it comes out or again. Again, this was anecdotal. This was a conversation

[Charles Margossian]: 01:08:33

in the laboratory,

[Alex Andorra]: 01:08:34

Yeah.

[Charles Margossian]: 01:08:35

but I got very excited about that exam.

[Alex Andorra]: 01:08:38

Yeah, I mean, that sounds super exciting. Thanks a lot for sharing that. And

[Alex Andorra]: 01:08:44

yeah, like if when the paper is out, please get in touch and then that'd

[Alex Andorra]: 01:08:49

be a fun thing to talk about again. And actually, so we're running a bit

[Alex Andorra]: 01:08:58

long, but do you still have a few more minutes because I still have a few

[Alex Andorra]: 01:09:03

questions for you. Or you want

[Charles Margossian]: 01:09:06

Yeah,

[Alex Andorra]: 01:09:06

to

[Charles Margossian]: 01:09:06

yeah,

[Alex Andorra]: 01:09:06

close out?

[Charles Margossian]: 01:09:07

I can stick around for a bit. Yeah.

[Alex Andorra]: 01:09:09

Okay, awesome. Yeah. So, uh, yeah, I'd like to, to talk to you about, uh,

[Alex Andorra]: 01:09:14

priors and then maybe, um, Torsten, uh, or an epidemiology example, uh, or both

[Alex Andorra]: 01:09:22

of them. Um, so yeah, maybe, um, basically you have, uh, been working on,

[Alex Andorra]: 01:09:32

on Torsten, which is, if I understood correctly, uh, pharmacometrics application

[Alex Andorra]: 01:09:38

of Stan models. So that was interesting to me because the Bayesian field

[Alex Andorra]: 01:09:47

has been evolving really, really fast lately, especially with the new techniques,

[Alex Andorra]: 01:09:53

the new software tools, and you've definitely been part of that effort,

[Alex Andorra]: 01:09:59

Charles. So I'm wondering if any recent developments that have particularly

[Alex Andorra]: 01:10:05

excited you, especially in your field of pharmacometrics and epidemiology,

[Alex Andorra]: 01:10:12

and also in relationship with what you're doing with Tolston.

[Charles Margossian]: 01:10:20

Yeah, okay, so let me talk about, let me give one example, right?

[Alex Andorra]: 01:10:26

Mm-hmm, yeah.

[Charles Margossian]: 01:10:27

So not a comprehensive answer, but an administrative answer.

[Alex Andorra]: 01:10:32

Yeah, it's great.

[Charles Margossian]: 01:10:32

Um, and so one, one fundamental, you know, yet another fundamental question

[Charles Margossian]: 01:10:39

that would come up at a workshop, um, is okay. I have this ODE integrator,

[Charles Margossian]: 01:10:47

right? I need to solve an ODE to evaluate my likelihood and the ODE's

[Charles Margossian]: 01:10:52

come with certain tuning parameters. In

[Alex Andorra]: 01:10:54

Mm-hmm.

[Charles Margossian]: 01:10:54

particular, what is the precision with which you should solve your

[Charles Margossian]: 01:10:58

ODE? And. And that is going to have an impact on the quality of your

[Charles Margossian]: 01:11:06

entrance and also the run time. Right, because if you solve the OD

[Charles Margossian]: 01:11:12

with a very strict tolerance, it takes much longer to solve that OD. Right.

[Charles Margossian]: 01:11:22

And so again, that comes back to the question of how much computation

[Charles Margossian]: 01:11:25

should we throw at the problem. And For the longest time, I didn't

[Charles Margossian]: 01:11:30

have a good answer. And maybe I still don't to that question. And the

[Charles Margossian]: 01:11:37

way this manifested is either, you know, at workshops when teaching

[Charles Margossian]: 01:11:41

the subject, but even when we were writing the Stan manual, we have

[Charles Margossian]: 01:11:47

a page on OD integrators. We are, we have a, you know, we state, these

[Charles Margossian]: 01:11:51

are the default values that we use, but we don't have any clear recommendations

[Charles Margossian]: 01:11:57

on what is the precision you should use. And we kind of assume that

[Charles Margossian]: 01:12:03

the user knows their ODE well enough that they'll know which ODE integrator

[Charles Margossian]: 01:12:09

to pick and what tolerance to set.

[Alex Andorra]: 01:12:11

Mm-hmm.

[Charles Margossian]: 01:12:15

which is not realistic. But it's just we didn't have an answer to that

[Charles Margossian]: 01:12:21

question. And so recently, there was a paper. And so I know that.

[Charles Margossian]: 01:12:33

Let me look up exactly who the... I know Aki is a co-author on it,

[Charles Margossian]: 01:12:37

Aki the Tarii.

[Alex Andorra]: 01:12:38

Mm-hmm.

[Charles Margossian]: 01:12:41

But I want to give a shout out to the lead author.

[Alex Andorra]: 01:12:45

Yeah, for sure. And we'll put that also in the show notes for this episode.

[Charles Margossian]: 01:12:51

Yeah, so there you go. So Juho Timonen is the lead authors, is the

[Charles Margossian]: 01:12:54

lead author, and then there are a bunch of other people on it. So

[Charles Margossian]: 01:12:56

their paper is an important sampling approach for Bayesian ODIs. And so

[Charles Margossian]: 01:13:03

essentially, what they realize is, if when you're solving, when you're

[Charles Margossian]: 01:13:09

using a numerical integrator to evaluate your likelihood, really you're

[Charles Margossian]: 01:13:14

not computing the true likelihood, you're computing an approximation

[Charles Margossian]: 01:13:17

of this likelihood. And we have a lot of tools in statistics when we're

[Charles Margossian]: 01:13:26

not dealing with the exact likelihood, but some approximation to this

[Charles Margossian]: 01:13:30

likelihood, notably in the field of importance sampling.

[Alex Andorra]: 01:13:34

Mm-hmm.

[Charles Margossian]: 01:13:35

And what they came up with is a way to use those tools that exist

[Charles Margossian]: 01:13:40

in importance sampling. to actually check whether the approximate likelihood

[Charles Margossian]: 01:13:48

and therefore the tuning parameter of the ODE integrators are as precise

[Charles Margossian]: 01:13:52

enough

[Alex Andorra]: 01:13:53

Mm-hmm.

[Charles Margossian]: 01:13:53

or not, right? And so that gives you a diagnostic that you can use.

[Charles Margossian]: 01:14:01

It's not a completely perfect diagnostic and I think I'm still trying

[Charles Margossian]: 01:14:04

to test that idea and play around with it and see how well it really

[Charles Margossian]: 01:14:08

works. So I wanna try it out on pharmacometrics problems. Um, right

[Charles Margossian]: 01:14:12

now I'm writing, uh, I've been tasked with writing, um, a tutorial on

[Charles Margossian]: 01:14:18

Torsten. So we released part one a while ago. Now we're writing part

[Charles Margossian]: 01:14:21

two and we promised that in part two, we'd explain how to tune the ODEs.

[Alex Andorra]: 01:14:25

Mm-hmm.

[Charles Margossian]: 01:14:25

Um, except we only know so much how to tune the ODEs. So I am trying

[Alex Andorra]: 01:14:30

Yeah.

[Charles Margossian]: 01:14:30

this method. I am trying this method, um, on the ODEs, but it's just

[Charles Margossian]: 01:14:35

getting me, you know, thinking about. are the tolerances we're using

[Charles Margossian]: 01:14:39

too conservative? Are they too strict? We actually get important

[Charles Margossian]: 01:14:43

speed ups. I'm teaching a course this September in Leuven, the Advanced

[Charles Margossian]: 01:14:50

Summer School in Basie Methods, where I'm gonna have the students

[Charles Margossian]: 01:14:54

on an epidemiology problem, on a pharmacokinetic problem, try out different

[Charles Margossian]: 01:14:58

tolerances and see the differences and then build this diagnostic

[Charles Margossian]: 01:15:02

that's based on important sampling to check whether the precision with

[Charles Margossian]: 01:15:08

which they're solving their ODEs is making meaningful changes to

[Charles Margossian]: 01:15:13

the inference, right?

[Alex Andorra]: 01:15:14

Mm-hmm.

[Charles Margossian]: 01:15:15

And so again, I think that this is one tuning parameter where either

[Charles Margossian]: 01:15:18

we're using ODE solvers that are not precise enough or ODE solvers

[Charles Margossian]: 01:15:22

that are too slow. We're not being optimal in our computation. And this

[Charles Margossian]: 01:15:29

is preventing us from either getting accurate answers or deploying

[Charles Margossian]: 01:15:33

models with the sophistication that we would want to. And so that's a development

[Charles Margossian]: 01:15:38

that I'm excited about. The one caveat that I will throw in this is

[Charles Margossian]: 01:15:43

that right now we're still thinking about it as a single tuning parameter.

[Charles Margossian]: 01:15:48

When what I've observed in practice is that the behavior of the ODE

[Charles Margossian]: 01:15:56

can change widely depending on where you are in the parameter space.

[Alex Andorra]: 01:16:00

Mm-hmm.

[Charles Margossian]: 01:16:01

So for certain parameter values, you don't need to be super precise.

[Charles Margossian]: 01:16:06

And for other parameter values, you need a lot more precision or you

[Charles Margossian]: 01:16:10

need a different type of integrator because the OD just behaves in

[Charles Margossian]: 01:16:15

a different way.

[Alex Andorra]: 01:16:16

Yeah.

[Charles Margossian]: 01:16:17

Now very concretely, how does this manifest? So I don't wanna say it's

[Charles Margossian]: 01:16:20

hopeless, but what ends up happening is during the warmup phase,

[Charles Margossian]: 01:16:27

where we start the Markov chains far off in the parameter space, or we

[Charles Margossian]: 01:16:32

haven't tuned the MCMC sampler, so the Markov chains are still jumping

[Charles Margossian]: 01:16:37

left and right.

[Alex Andorra]: 01:16:38

Mm-hmm.

[Charles Margossian]: 01:16:39

During the warm-up phase, we are more vulnerable to those extreme parameter

[Charles Margossian]: 01:16:46

values. Whereas during the sampling phase, we get away with less strict

[Charles Margossian]: 01:16:51

body solvers. So I think that somehow, what I would like to do is two

[Charles Margossian]: 01:16:56

things. One, I would like to have a very automatic way of running this

[Charles Margossian]: 01:16:59

diagnostics in Torsten. But I also wanna give users control over what

[Charles Margossian]: 01:17:05

OD solver do they use at different stages of MCMC. They think that

[Charles Margossian]: 01:17:10

makes a crucial difference. Another way to approach this problem is coming

[Charles Margossian]: 01:17:16

up with good initializations. If I can start... my MCMC near, you know, within

[Charles Margossian]: 01:17:24

the parameter space where I might land from the stationary distribution.

[Charles Margossian]: 01:17:29

And I know that here, the parameter values are a bit less absurd.

[Charles Margossian]: 01:17:33

And so solving the ODEs is a bit more feasible computationally. If

[Charles Margossian]: 01:17:38

I can start there, then maybe I'm skipping the early regions that really

[Charles Margossian]: 01:17:43

frustrate my ODE integrator and my MCMC sample. And the way this manifests

[Charles Margossian]: 01:17:48

in practice is you'll have, you know, you run, let's say you run

[Charles Margossian]: 01:17:51

eight chains. You have six of them that finish quickly, and then you

[Charles Margossian]: 01:17:55

have two of them that are lagging because they're stuck somewhere

[Charles Margossian]: 01:17:59

during the warmup phase. They're encountering this region where the

[Charles Margossian]: 01:18:02

parameter values are a little bit absurd. Your OD is super hard to

[Charles Margossian]: 01:18:07

solve, and that's eating up all your computation. And the truth

[Alex Andorra]: 01:18:11

Mm-hmm.

[Charles Margossian]: 01:18:11

is, you know, at least... And the way we do things right now, we always

[Charles Margossian]: 01:18:16

wait for the slowest chain. By

[Alex Andorra]: 01:18:18

Mm-hmm.

[Charles Margossian]: 01:18:18

the way, we don't have to do it, right? So I'm excited about methods

[Charles Margossian]: 01:18:22

to come up with good initializations. And I think that this is a place where variational

[Charles Margossian]: 01:18:28

inference can be good, right? So the, especially

[Alex Andorra]: 01:18:30

Aha.

[Charles Margossian]: 01:18:30

now the pathfinder variational inference, right,

[Alex Andorra]: 01:18:33

Mm-hmm.

[Charles Margossian]: 01:18:36

was originally designed to produce good initializations for NCNC. And

[Charles Margossian]: 01:18:42

so getting good initializations, that's a great example of, I need a good answer,

[Charles Margossian]: 01:18:47

but not a super precise answer. Right. Uh, and so, you know, if somehow

[Charles Margossian]: 01:18:53

pathfinder can help me skip the regions that frustrate OD integrators,

[Charles Margossian]: 01:18:59

I think that's a big win for pharmacometrics. Again, that's, that's

[Charles Margossian]: 01:19:03

something we have to test and really try out. But I will say now going

[Charles Margossian]: 01:19:07

back, I'm going to make another connection back to our hat. which is

[Charles Margossian]: 01:19:12

when we think about over dispersion. really what over dispersion means.

[Charles Margossian]: 01:19:18

So we have shown a bit formally that what makes our hat reliable and

[Charles Margossian]: 01:19:23

we define reliability in a formal sense is the initial variance has

[Charles Margossian]: 01:19:28

to be large relative to the initial bias. Now, if you have an initialization,

[Charles Margossian]: 01:19:33

like you draw your sample from your prior and that's reliable,

[Alex Andorra]: 01:19:38

Mm-hmm.

[Charles Margossian]: 01:19:38

and then you throw variation inference, right? If variation inference reduces

[Charles Margossian]: 01:19:42

your squared bias more than it reduces your variance, it turns out

[Charles Margossian]: 01:19:46

you preserve the property of reliability. So there's actually a sense

[Charles Margossian]: 01:19:50

that we might be able to get good initializations for MCMC without

[Charles Margossian]: 01:19:55

compromising, uh, the reliability of our diagnostics for convergence,

[Charles Margossian]: 01:20:01

right? And these are all the pieces that I think can come together and

[Charles Margossian]: 01:20:05

really help a great deal with, you know, pharmacometrics, but more generally,

[Charles Margossian]: 01:20:11

uh, with OD based models. and even more generally, models based on implicit

[Charles Margossian]: 01:20:16

functions. And I do include things that use nested Laplace approximations

[Charles Margossian]: 01:20:20

in that, because that's an optimization problem, that's an implicit

[Charles Margossian]: 01:20:23

function. It has the same kind of misbehaviors that an ODE has, but also

[Charles Margossian]: 01:20:28

technologies that we develop for ODE is in a Bayesian context, better initializations,

[Charles Margossian]: 01:20:34

different tolerances, important sampling corrections would apply

[Charles Margossian]: 01:20:38

to nested Laplace stuff. So those are the things that I'm excited about

[Charles Margossian]: 01:20:46

Um, but it's going to take time.

[Alex Andorra]: 01:20:49

Hey,

[Charles Margossian]: 01:20:49

I just

[Alex Andorra]: 01:20:50

yeah.

[Charles Margossian]: 01:20:50

want to, I just want to be perfectly honest. It takes time to really,

[Charles Margossian]: 01:20:54

you know, uh, you know, between the paper and. The software implementation

[Charles Margossian]: 01:21:00

and, uh, you know, clear description in the manual that the users can,

[Charles Margossian]: 01:21:06

can follow. It takes a lot of time.

[Alex Andorra]: 01:21:10

Yeah, yeah, for sure. I mean, that stuff is really at the frontier of the

[Alex Andorra]: 01:21:12

research. So it does make sense that it takes time to percolate

[Charles Margossian]: 01:21:17

Yeah, yeah.

[Alex Andorra]: 01:21:17

from how to find a solution to, okay, this is how we can implement it and

[Alex Andorra]: 01:21:23

reproduce it reliably.

[Charles Margossian]: 01:21:26

Yeah, exactly. And I'm most concerned about how speculative I am for

[Charles Margossian]: 01:21:30

some of the ideas I'm sharing. But I think these are directions where

[Charles Margossian]: 01:21:36

it's worth pushing the research. That has the potential to have a

[Charles Margossian]: 01:21:41

really big impact.

[Alex Andorra]: 01:21:42

Yeah, yeah, for sure, for sure. And so I put in the show notes the Pathfinder

[Alex Andorra]: 01:21:50

paper, actually. That made me think that I should do an episode about the

[Alex Andorra]: 01:21:56

Pathfinder paper, basically, what that is about and what that means concretely.

[Charles Margossian]: 01:22:02

Mm-hmm.

[Alex Andorra]: 01:22:02

So yeah, I'll try to do that

[Charles Margossian]: 01:22:06

Yeah.

[Alex Andorra]: 01:22:06

in

[Charles Margossian]: 01:22:06

And

[Alex Andorra]: 01:22:06

the

[Charles Margossian]: 01:22:06

if you haven't

[Alex Andorra]: 01:22:07

near

[Charles Margossian]: 01:22:07

reached

[Alex Andorra]: 01:22:07

future.

[Charles Margossian]: 01:22:07

out to Lu Zhang, you know, I mean, it, you know, first of all, the paper

[Charles Margossian]: 01:22:14

is great. But actually sitting down and discussing this with her,

[Charles Margossian]: 01:22:20

you know, at the blackboard or whatever, like, she has so many ideas that

[Charles Margossian]: 01:22:25

have not appeared in the paper itself. I think, you know, if I can recommend

[Charles Margossian]: 01:22:33

a guest. If you haven't had her already, I don't know if you... But

[Charles Margossian]: 01:22:37

yeah, Lu

[Alex Andorra]: 01:22:37

No,

[Charles Margossian]: 01:22:37

Zheng, I think,

[Alex Andorra]: 01:22:38

I didn't.

[Charles Margossian]: 01:22:38

would be fantastic to interview.

[Alex Andorra]: 01:22:40

Yeah, yeah, exactly. I was actually thinking about inviting her on the podcast

[Alex Andorra]: 01:22:45

to talk about Pathfinder because she's the lead author on the paper and also

[Alex Andorra]: 01:22:51

all the other authors have been on the show, Bob

[Charles Margossian]: 01:22:54

Ah

[Alex Andorra]: 01:22:55

Carpenter,

[Charles Margossian]: 01:22:55

cool.

[Alex Andorra]: 01:22:56

Andrew

[Charles Margossian]: 01:22:56

Cool.

[Alex Andorra]: 01:22:56

Gellman, Akive Tareiso. Lu Zheng is missing, so definitely

[Charles Margossian]: 01:23:01

Mm-hmm.

[Alex Andorra]: 01:23:02

need to correct that. So yeah, in the near future, definitely try. to have

[Alex Andorra]: 01:23:08

that episode that would be a very interesting one. So maybe can you talk

[Alex Andorra]: 01:23:15

Charles about... So I'll give you two avenues and you pick the one you prefer

[Alex Andorra]: 01:23:23

because I don't want to take too much of your time but basically I'm curious

[Alex Andorra]: 01:23:28

either about hearing an example of your work in the epidemiology field Or you

[Alex Andorra]: 01:23:37

can talk a bit more in general about a very, very common question that students

[Alex Andorra]: 01:23:45

always ask me, and it's about priors. So basically, how do you choose the

[Alex Andorra]: 01:23:51

prior, how do you approach the challenge of choosing appropriate prior distributions,

[Alex Andorra]: 01:23:56

and especially when you're dealing with complex models. So these are the two avenues

[Alex Andorra]: 01:24:01

I have in mind. And feel

[Charles Margossian]: 01:24:03

Okay.

[Alex Andorra]: 01:24:04

free to... Pick one or... Pick both.

[Charles Margossian]: 01:24:09

Okay, so let me say that about priors, I think that...

[Charles Margossian]: 01:24:15

I want to get better answers,

[Alex Andorra]: 01:24:17

Yeah.

[Charles Margossian]: 01:24:17

a better answer to this question. And, you know, like the next time

[Charles Margossian]: 01:24:21

workshop that I'm giving, I don't have a module on priors that I find

[Charles Margossian]: 01:24:26

satisfactory. And so I'm still undergoing this journey. But what I will

[Charles Margossian]: 01:24:30

do is I'll give, I'll talk about epidemiology and I will talk about

[Charles Margossian]: 01:24:34

the prior that we use there. So that will be, that would be an example.

[Charles Margossian]: 01:24:39

And, you know, I like to think. through examples, I like to think

[Alex Andorra]: 01:24:44

Mm-hmm.

[Charles Margossian]: 01:24:45

through the anecdotal, as complementary to the formal, right? I'm

[Charles Margossian]: 01:24:51

a big fan of fairy tales and fables, simple stories that have good themes.

[Alex Andorra]: 01:24:58

Yeah.

[Charles Margossian]: 01:25:01

So what happened in epidemiology, and this will be a good example, is, well,

[Charles Margossian]: 01:25:06

the pandemic happened, and suddenly COVID, we're all going home.

[Charles Margossian]: 01:25:16

And we had colleagues in epidemiology, in particular Julien Rioux,

[Charles Margossian]: 01:25:20

who actually met Julien Rioux at the StankCon. He was in Cambridge,

[Charles Margossian]: 01:25:26

he was a PhD student at the time and he demonstrated his model and

[Charles Margossian]: 01:25:31

he was using those ODI based models. So now instead of having a drug

[Charles Margossian]: 01:25:36

compound that flows between different parts of the body, what you have

[Charles Margossian]: 01:25:40

is... you separate the population, the human population, to these

[Charles Margossian]: 01:25:46

different compartments. So susceptible individuals, infected individuals,

[Charles Margossian]: 01:25:51

recovered individuals, and then the individuals flow between the compartments.

[Charles Margossian]: 01:25:57

And there are a bit more layers. But basically the mathematical formalism

[Charles Margossian]: 01:26:03

that I had familiarized myself with in the context of pharmacometrics

[Charles Margossian]: 01:26:08

turned out to be very relevant to... certain classes of epidemiological

[Charles Margossian]: 01:26:13

models. And essentially, Julien was working on an early model of COVID-19.

[Charles Margossian]: 01:26:21

They were trying to estimate the mortality rate. There was a lot of uncertainty

[Charles Margossian]: 01:26:26

in the data. There were a lot of

[Charles Margossian]: 01:26:31

things to correct for. Right. So, for example, early on, not everyone

[Charles Margossian]: 01:26:37

got tested and testing was not widely available.

[Alex Andorra]: 01:26:41

Mm-hmm.

[Charles Margossian]: 01:26:42

Who got tested? The people with severe symptoms. So now if you think about

[Charles Margossian]: 01:26:46

you're trying to estimate the mortality rate

[Alex Andorra]: 01:26:49

Yeah.

[Charles Margossian]: 01:26:51

and according to your data, the people who catch the disease are

[Charles Margossian]: 01:26:55

only the ones who have severe symptoms, then it looks like a lot of

[Charles Margossian]: 01:26:59

people are dying from the disease. I mean, a lot of people were dying

[Charles Margossian]: 01:27:03

from the disease, but that inflates the number. There's a bias

[Alex Andorra]: 01:27:06

Yeah,

[Charles Margossian]: 01:27:06

because you're

[Alex Andorra]: 01:27:06

yeah,

[Charles Margossian]: 01:27:06

only

[Alex Andorra]: 01:27:06

for sure.

[Charles Margossian]: 01:27:07

testing the people who are sick. You're not testing the people with

[Charles Margossian]: 01:27:11

mild symptoms or even with no symptoms.

[Alex Andorra]: 01:27:13

No, for sure. So, clear sampling by ASEA.

[Charles Margossian]: 01:27:17

The other bias was some of the people who had caught the virus had

[Charles Margossian]: 01:27:22

not died yet. So you count them as living, but that doesn't mean they

[Charles Margossian]: 01:27:26

survived the disease. So that's a bias

[Alex Andorra]: 01:27:27

Yeah.

[Charles Margossian]: 01:27:28

in another direction. And that's an example where you actually have

[Charles Margossian]: 01:27:32

a somewhat mechanistic model. That's based on the epidemiology of

[Charles Margossian]: 01:27:37

how a disease transmits and circulates in a population. Then on top

[Charles Margossian]: 01:27:41

of that, you need to build a measurement model to account for, you know,

[Charles Margossian]: 01:27:45

how the data is collected. Right. But at the end of the day, none of the,

[Charles Margossian]: 01:27:50

you know, we were not able to draw any conclusions. unless we understood

[Charles Margossian]: 01:27:58

what was the rate of people who were symptomatic or had severe symptoms.

[Charles Margossian]: 01:28:03

Right. And so there's one parameter in the model, which is the asymptomatic

[Charles Margossian]: 01:28:06

rate. And so now you have two options in a classic classical statistics

[Charles Margossian]: 01:28:11

framework. Either you fix the parameter and then you're making a

[Charles Margossian]: 01:28:16

strong assumption and maybe you try different values of the fixed

[Charles Margossian]: 01:28:19

parameter. Right. Or you just say, well, I don't know this. So really

[Charles Margossian]: 01:28:24

I have no idea what the mortality rate is because maybe the entire

[Charles Margossian]: 01:28:27

population was infected or maybe only a small fraction was infected

[Charles Margossian]: 01:28:31

and everyone in that small fraction had the severe disease, right? And

[Charles Margossian]: 01:28:37

so we needed an in-between, between saying we don't know anything and saying

[Charles Margossian]: 01:28:43

we know everything. And this is why I think we're basing shines, which

[Charles Margossian]: 01:28:47

is we can quantify uncertainty. We have more nuanced statements. through the

[Charles Margossian]: 01:28:52

language of probability about what our state of knowledge is. And actually

[Charles Margossian]: 01:28:58

what had happened is there were some instances where we had measured asymptomatic

[Charles Margossian]: 01:29:05

rates. The example was a cruise ship, the Diamond Princess, in the

[Charles Margossian]: 01:29:12

coast of Japan. So they had identified some cases of COVID-19. They

[Charles Margossian]: 01:29:16

put the cruise ship in quarantine and they test everybody, regardless

[Charles Margossian]: 01:29:19

of whether they had symptoms or not. So now you have a small population

[Charles Margossian]: 01:29:23

and based on that small population, you get an estimate of the

[Alex Andorra]: 01:29:28

Yeah.

[Charles Margossian]: 01:29:28

asymptomatic rate. And then you had one or two incidents where people

[Charles Margossian]: 01:29:34

do that. There were some cities where they had done some other experiments

[Charles Margossian]: 01:29:41

and measured some other data. And so then you bring all that information

[Charles Margossian]: 01:29:46

and you use that information. to construct priors and then to propagate uncertainty

[Charles Margossian]: 01:29:52

into the model, right? The reason we're able to make predictions and

[Charles Margossian]: 01:29:55

then to

[Charles Margossian]: 01:29:59

calculate things like the mortality rate with the appropriate uncertainty

[Charles Margossian]: 01:30:03

is because we had a prior on the asymptomatic rate. And that's a very

[Charles Margossian]: 01:30:08

nice example. This is more an example of why it was crucial to have

[Charles Margossian]: 01:30:13

a prior rather than, you know. How should you construct priors in general,

[Charles Margossian]: 01:30:17

right? This is a bit of a specific case,

[Alex Andorra]: 01:30:19

Yep.

[Charles Margossian]: 01:30:19

but it's a very good example. And so I'll recommend two papers. One is the

[Charles Margossian]: 01:30:25

one by Julien Rioux, which was, so this was disappeared in PLOS Medicine.

[Charles Margossian]: 01:30:34

And it's with a lot of contributors, but it's an estimation of seroscoped

[Charles Margossian]: 01:30:37

tumor mortality during the early stages of an epidemic. And then the

[Charles Margossian]: 01:30:41

other paper, that goes a bit more into what are the lessons that we

[Charles Margossian]: 01:30:46

learned from this, from a Bayesian workflow perspective, is Bayesian

[Charles Margossian]: 01:30:49

workflow for disease transmission modeling in Stan. And so here the first author

[Charles Margossian]: 01:30:54

was Leo Grintjerdsch.

[Alex Andorra]: 01:30:57

Mm-hmm.

[Charles Margossian]: 01:30:59

And then we had also Lisa Semenova and Julien Rioux as co-authors.

[Alex Andorra]: 01:31:03

Mm-hmm.

[Charles Margossian]: 01:31:04

And this is a beautiful paper that really goes into the, you know,

[Charles Margossian]: 01:31:09

here's the prior reuse. Here's the first version of the model. Here

[Charles Margossian]: 01:31:12

are the limitations with this model. Here's how we diagnose the models.

[Charles Margossian]: 01:31:16

Here's the next iterations. And we go through all the iterations. And I

[Charles Margossian]: 01:31:19

like to show the numbers that the model that eventually we used to model

[Charles Margossian]: 01:31:23

COVID-19 was the 15th iteration. And along the way we had the model

[Charles Margossian]: 01:31:29

that took three days to run. And we had to change the way we wrote

[Charles Margossian]: 01:31:35

the Stan model to improve the computation. So that had to do with how

[Charles Margossian]: 01:31:39

the OD was parameterized and how the automatic differentiation was happening.

[Charles Margossian]: 01:31:42

We got it from three days to two days. And that's useful not, sorry,

[Charles Margossian]: 01:31:48

three days to two hours, not two days, two hours, drastic speed up.

[Charles Margossian]: 01:31:55

And so that not only was good because the inference is faster, but that's

[Charles Margossian]: 01:31:58

what allowed us to then use more sophisticated versions of the model.

[Charles Margossian]: 01:32:05

And so

[Alex Andorra]: 01:32:05

Yeah.

[Charles Margossian]: 01:32:05

all that is described in that. Bayesian workflow for disease transmission.

[Charles Margossian]: 01:32:11

paper.

[Alex Andorra]: 01:32:13

Hmm, oh yeah, need

[Charles Margossian]: 01:32:14

So

[Alex Andorra]: 01:32:14

that.

[Charles Margossian]: 01:32:15

that's my epidemiology fairy tale. And I don't mean

[Alex Andorra]: 01:32:20

Yeah.

[Charles Margossian]: 01:32:20

that in the sense that it had a happy ending or that everything was

[Charles Margossian]: 01:32:24

nice and glowing. I mean that this is a very nice story that touches

[Charles Margossian]: 01:32:29

upon a lot of interesting things.

[Alex Andorra]: 01:32:32

Yeah, yeah, for sure. Definitely going to use link to this paper in the show

[Alex Andorra]: 01:32:40

notes. Already have it on the Stan website. Actually, you've done a case

[Alex Andorra]: 01:32:47

study on this, so that's perfect with all the co-authors. So I'm going to put

[Alex Andorra]: 01:32:52

that right now in the show notes. And maybe last question before letting you

[Alex Andorra]: 01:32:58

go, Charles. I'm breaking records these days on the episodes. Like episode 89

[Alex Andorra]: 01:33:06

is going out this week actually. And it's so far the longest episode. Uh, it's

[Alex Andorra]: 01:33:11

about two hours. And right now we're like approaching this record, uh, Charles.

[Alex Andorra]: 01:33:17

So, uh, like well done. And at the same time.

[Charles Margossian]: 01:33:20

Okay. There's a fantastic podcast and I forget the name. It's a German

[Charles Margossian]: 01:33:24

podcast.

[Alex Andorra]: 01:33:25

Uh huh.

[Charles Margossian]: 01:33:28

I can't believe I forget the name. I'll try and email you what it is,

[Charles Margossian]: 01:33:31

but basically they do interviews with these, you know, intellectuals

[Charles Margossian]: 01:33:35

and well established, you know, politicians and well, I'm not saying

[Charles Margossian]: 01:33:38

all politicians are intellectuals, but people have opinions and thoughts and there's

[Charles Margossian]: 01:33:43

no time limit to the interview. And they just go on and on and on and on and

[Charles Margossian]: 01:33:47

on. And they just have so many topics to discuss. It's really, I'm

[Charles Margossian]: 01:33:54

not saying you should do that with me, but you could imagine like,

[Charles Margossian]: 01:33:56

you know, someone like, you know, Some of the other co-authors that

[Charles Margossian]: 01:34:01

have come up, I feel like you could talk six hours with them and you

[Charles Margossian]: 01:34:06

would just

[Alex Andorra]: 01:34:06

Oh

[Charles Margossian]: 01:34:06

pick

[Alex Andorra]: 01:34:06

yeah,

[Charles Margossian]: 01:34:06

their

[Alex Andorra]: 01:34:06

for

[Charles Margossian]: 01:34:07

brand

[Alex Andorra]: 01:34:07

sure.

[Charles Margossian]: 01:34:07

and they would have so much insights to

[Alex Andorra]: 01:34:09

Yeah.

[Charles Margossian]: 01:34:09

share.

[Alex Andorra]: 01:34:11

No, for sure.

[Charles Margossian]: 01:34:12

So

[Alex Andorra]: 01:34:12

Like, I mean, most

[Charles Margossian]: 01:34:14

they

[Alex Andorra]: 01:34:14

of

[Charles Margossian]: 01:34:14

are.

[Alex Andorra]: 01:34:14

the time the limitation is the, is the guest's own, own time.

[Charles Margossian]: 01:34:19

Right, right,

[Alex Andorra]: 01:34:20

Yeah.

[Charles Margossian]: 01:34:21

right. But the

[Alex Andorra]: 01:34:22

Yeah, for

[Charles Margossian]: 01:34:22

dev,

[Alex Andorra]: 01:34:22

sure.

[Charles Margossian]: 01:34:23

yeah, I think it's really cool, this idea of, yeah, we're gonna do, what

[Charles Margossian]: 01:34:26

if we didn't have a time limit on the interview? What would happen?

[Charles Margossian]: 01:34:28

And

[Alex Andorra]: 01:34:29

Exactly.

[Charles Margossian]: 01:34:30

eventually somebody gets hungry and that's what happens, but. Ha ha

[Charles Margossian]: 01:34:33

ha.

[Alex Andorra]: 01:34:34

Yeah, yeah, exactly. Yeah, but so basically before asking you the last two

[Alex Andorra]: 01:34:40

questions, I'm also wondering, because you also teach, so that's super interesting

[Alex Andorra]: 01:34:47

to me. And I often hear that a lot of practitioners and beginners in particular

[Alex Andorra]: 01:34:59

might be hesitant or even intimidated. to adopt patient methods because they perceive

[Alex Andorra]: 01:35:05

them as complex. So I don't necessarily agree with that underlying assumption,

[Alex Andorra]: 01:35:13

but without necessarily disputing it, what do you do in those cases? Basically,

[Alex Andorra]: 01:35:20

what resources or strategies do you recommend to those who want to learn

[Alex Andorra]: 01:35:25

and apply patient techniques in the work, but might be intimidated or hesitant?

[Charles Margossian]: 01:35:35

Yeah, okay, so I think that usually when I teach.

[Charles Margossian]: 01:35:43

Most of the time it's people who are already interested in Bayesian

[Charles Margossian]: 01:35:46

methods,

[Alex Andorra]: 01:35:47

Mm-hmm.

[Charles Margossian]: 01:35:48

especially if it's a workshop on stand, the people who do sign up, they already

[Charles Margossian]: 01:35:53

have adhered to Bayesian methodology. So I don't know that I've really

[Charles Margossian]: 01:35:56

had to convince that many people. Definitely when I was a TA at Columbia,

[Alex Andorra]: 01:36:02

Mm-hmm.

[Charles Margossian]: 01:36:03

I had those conversations a bit more with the

[Alex Andorra]: 01:36:05

Mm-hmm.

[Charles Margossian]: 01:36:05

students, especially when I TA'd the PhD level. that everyone has to

[Charles Margossian]: 01:36:11

take applied statistics and

[Alex Andorra]: 01:36:13

Hm.

[Charles Margossian]: 01:36:13

not everyone is gonna do Bayesian, right? And so we have those conversations.

[Charles Margossian]: 01:36:18

But I think that ultimately what it is, is not that Bayesian is complex,

[Charles Margossian]: 01:36:24

it's analysis is complex. Analysis is difficult. And when you use less

[Charles Margossian]: 01:36:33

complicated methods,

[Charles Margossian]: 01:36:37

like, maximum likelihood estimates or point estimates. And these are

[Charles Margossian]: 01:36:43

not simple. I really don't want to undermine the difficulty related to

[Charles Margossian]: 01:36:49

those methods and the fact that they are useful in a lot of applications.

[Charles Margossian]: 01:36:54

But anyway, those methods are simple and they will work for simple analysis.

[Charles Margossian]: 01:37:03

But if an analysis requires you quantify uncertainty, to propagate uncertainty,

[Charles Margossian]: 01:37:08

to take some decisions with imperfect data. And then someone says,

[Charles Margossian]: 01:37:14

well, I don't wanna be Bayesian because that's complicated, but I

[Charles Margossian]: 01:37:18

still want to quantify uncertainty and I still want to propagate uncertainty

[Charles Margossian]: 01:37:22

and I still want to make predictions. Well, suddenly the classical methods,

[Charles Margossian]: 01:37:26

you really have to do a lot of gymnastic to get them to do what you're

[Charles Margossian]: 01:37:32

interested in. And And so the classical methods also become complicated.

[Charles Margossian]: 01:37:37

And that's more because you're trying to use them for, uh, you know,

[Charles Margossian]: 01:37:44

to do a sophisticated analysis. And so I think that what matters

[Charles Margossian]: 01:37:48

is, you know, to really meet practitioners, uh, where they are, what

[Charles Margossian]: 01:37:56

is the problem they're interested in? And. you know, does the, you

[Charles Margossian]: 01:38:01

know, is it, is it a problem where, you know, the, the complexity of the

[Charles Margossian]: 01:38:05

analysis would be handled in a relatively straightforward way by a Bayesian

[Charles Margossian]: 01:38:10

analysis? And the answer could be yes, it could also be no. And if it's

[Charles Margossian]: 01:38:13

no, that's fine. You know, I think that again, we have, we have to be

[Charles Margossian]: 01:38:18

loyal to, to the problem, more so than to the field or to, or to the

[Charles Margossian]: 01:38:23

method, but that's how I would start this conversation. And so when,

[Charles Margossian]: 01:38:26

you know, when I'm sitting with pharmacometricians, And by the way, a

[Charles Margossian]: 01:38:29

lot of them don't use Bayesian. I think I can talk for 30 minutes

[Charles Margossian]: 01:38:37

about what do we want to get out of a pharmacokinetic analysis without

[Charles Margossian]: 01:38:41

bringing up what kind of statistics we do. And once we've established

[Charles Margossian]: 01:38:44

the common goals, then we think about what are the methods that are

[Charles Margossian]: 01:38:48

going to get there. That's how I would do it. Frankly, I haven't had that

[Charles Margossian]: 01:38:53

much experience doing it. So take what I say with a grain of salt.

[Charles Margossian]: 01:38:58

Yeah, but I really think it's, Bayesian is complicated because it confronts

[Charles Margossian]: 01:39:02

you with the complexity of data analysis. Right? It doesn't

[Alex Andorra]: 01:39:06

Yeah.

[Charles Margossian]: 01:39:06

introduce the complexity. Let me put it this way.

[Alex Andorra]: 01:39:09

Yeah, yeah. No, completely agree. And that usually is... I usually give an answer

[Alex Andorra]: 01:39:21

along those lines. So

[Charles Margossian]: 01:39:23

Mm-hmm.

[Alex Andorra]: 01:39:23

that's interesting. To see that we have converged, even though we didn't, we

[Alex Andorra]: 01:39:31

didn't, you know, talk about that before the show. so that people don't

[Alex Andorra]: 01:39:34

start having a conspiracy theories about Charles and I trying to push back the

[Alex Andorra]: 01:39:39

message.

[Charles Margossian]: 01:39:40

This is not rehearsed.

[Alex Andorra]: 01:39:42

Exactly. Awesome. Well, is there a topic I didn't ask you about that you'd like

[Alex Andorra]: 01:39:51

to mention before we close up the show?

[Charles Margossian]: 01:39:57

Um, no. Let me keep

[Alex Andorra]: 01:40:01

Perfect.

[Charles Margossian]: 01:40:01

it simple.

[Alex Andorra]: 01:40:02

Yeah. I mean, we've talked about a lot

[Charles Margossian]: 01:40:05

There

[Alex Andorra]: 01:40:05

of

[Charles Margossian]: 01:40:05

are

[Alex Andorra]: 01:40:05

things.

[Charles Margossian]: 01:40:05

a lot of topics that I think we could have very interesting conversations

[Charles Margossian]: 01:40:09

about, but I also think that we're starting to hit the time constraints.

[Alex Andorra]: 01:40:18

Awesome. Well, let's go to show then. Me too, I could keep asking you a lot

[Alex Andorra]: 01:40:25

of things, but let's do another episode. Another day. That'd be a fun thing.

[Alex Andorra]: 01:40:32

Or if one day you have a model you've been working about and that you think

[Alex Andorra]: 01:40:38

would be beneficial for listeners to go through. I have this new format now,

[Alex Andorra]: 01:40:46

which are the modeling webinars, where basically you would come on the show

[Alex Andorra]: 01:40:51

and share your screen and show us your code and do some live coding about

[Alex Andorra]: 01:40:58

basically a project that you've been working on. And so if one day you have

[Alex Andorra]: 01:41:04

something like that, feel free to get in touch and we'll get that organized

[Alex Andorra]: 01:41:10

because... It's a really cool new format and I really love it. In August,

[Alex Andorra]: 01:41:16

we've had Justin Boyce showcasing how to do Bayesian modeling workflow in the

[Alex Andorra]: 01:41:25

biostatistics world. So even if you're not a biostatistician, it's a really

[Alex Andorra]: 01:41:32

useful thing because basically, as we were saying, Bayes are basically methods.

[Alex Andorra]: 01:41:38

Even though the field might... not be yours, the methods are definitely transferable

[Alex Andorra]: 01:41:44

in the workflow. So that was a very fun one. And then we've got one coming

[Alex Andorra]: 01:41:50

up in September in theory with Benjamin Vincent and we're going to dive into

[Alex Andorra]: 01:41:57

the new do operator that we have in Pimc and how to use that. So that's

[Alex Andorra]: 01:42:04

going to be a very fun one too.

[Charles Margossian]: 01:42:06

Yeah, I love the idea of that format, and I'm definitely going to

[Charles Margossian]: 01:42:10

check out the two webinars.

[Alex Andorra]: 01:42:14

Yeah,

[Charles Margossian]: 01:42:14

The

[Alex Andorra]: 01:42:14

yeah.

[Charles Margossian]: 01:42:14

one that's already there and the one that's coming up. Actually, the

[Charles Margossian]: 01:42:17

two topics sound extremely interesting.

[Alex Andorra]: 01:42:21

Yeah, I'll send that to you. And yeah, for sure that that's something I've

[Alex Andorra]: 01:42:25

been starting to do. And so if you have one such analysis one day, feel

[Alex Andorra]: 01:42:30

free to reach out that'd be a fun one. Um,

[Charles Margossian]: 01:42:34

Absolutely.

[Alex Andorra]: 01:42:35

so before letting you go, of course, I'm going to ask you the last two questions

[Alex Andorra]: 01:42:40

I ask every guest at the end of the show. One, if you had unlimited time

[Alex Andorra]: 01:42:46

and resources, which problem would you try to solve?

[Charles Margossian]: 01:42:51

Oh yeah, I thought about this and

[Alex Andorra]: 01:42:54

Hehe.

[Charles Margossian]: 01:42:55

I feel like so much of my work is about working under computational

[Charles Margossian]: 01:42:59

constraints,

[Alex Andorra]: 01:43:00

Mm-hmm.

[Charles Margossian]: 01:43:01

you know, and time constraints and resource constraints and suddenly you relax

[Charles Margossian]: 01:43:06

all of this. And...

[Charles Margossian]: 01:43:15

I think. For me, they're really big questions.

[Charles Margossian]: 01:43:24

let's say at a curiosity level, right? I think there are, you know,

[Charles Margossian]: 01:43:28

there are utilitarian questions and philanthropic questions that

[Charles Margossian]: 01:43:32

I might prioritize. But my thinking right now is, I really love the

[Charles Margossian]: 01:43:39

problems I worked on in astronomy. I am mind blown by some of the stuff

[Charles Margossian]: 01:43:45

my colleagues do in cosmology, where they're trying to understand, you know,

[Charles Margossian]: 01:43:51

the early universe. They have models with six parameters that apparently

[Charles Margossian]: 01:43:55

explain the entire structure of the universe. I like to understand that

[Charles Margossian]: 01:44:04

a little bit better. I love the work I did on exoplanets. I think

[Charles Margossian]: 01:44:09

thinking about... Yeah, is there life on other planets? How does it

[Charles Margossian]: 01:44:14

manifest? My advisor when I was an undergrad, she...

[Charles Margossian]: 01:44:22

She put it a good way, she said, you know, astronomy helps us think

[Charles Margossian]: 01:44:25

about our position in the universe, our place in the universe.

[Alex Andorra]: 01:44:30

Mm-hmm.

[Charles Margossian]: 01:44:30

And you know, yes, I have unlimited time and resources in this completely

[Charles Margossian]: 01:44:34

ideal scenario. I think I would gravitate towards these really, really

[Charles Margossian]: 01:44:39

big questions, which by the way, I can still get involved in

[Alex Andorra]: 01:44:46

Heh

[Charles Margossian]: 01:44:46

right

[Alex Andorra]: 01:44:46

heh.

[Charles Margossian]: 01:44:46

now as a researcher in Flat Iron, but it's true that there's more

[Charles Margossian]: 01:44:50

competition for that. Um, and just, there's a scene that I, um,

[Charles Margossian]: 01:45:03

Yeah, okay. If I had thought about this more, I would have just started

[Charles Margossian]: 01:45:06

there. But there's a scene that I really like. This goes back to cosmology,

[Charles Margossian]: 01:45:10

to astronomy, you know, which is this idea of a theory of everything

[Charles Margossian]: 01:45:13

and, you know, a very fundamental model.

[Alex Andorra]: 01:45:15

Mm-hmm.

[Charles Margossian]: 01:45:17

And, you know, maybe it all reduces to one equation or one set of equations.

[Charles Margossian]: 01:45:23

And what I really wonder is, you know, if we did have that model and

[Charles Margossian]: 01:45:27

that theory, how much insight would we actually get from it? because

[Charles Margossian]: 01:45:33

you can have a simple system with simple particles and simple interaction

[Charles Margossian]: 01:45:37

rules. And that doesn't mean you understand the emerging behavior of

[Charles Margossian]: 01:45:41

the system. And if I had unlimited resources and I can actually figure

[Charles Margossian]: 01:45:45

out what that equation is and then run the simulations with infinite

[Charles Margossian]: 01:45:48

computation and then study the behavior. And actually, at a very conceptual

[Charles Margossian]: 01:45:55

level, understand how much insight we get out of this.

[Alex Andorra]: 01:45:58

Yeah.

[Charles Margossian]: 01:45:58

And the example I like to give is, The rules of chess are simple,

[Charles Margossian]: 01:46:04

but just because you understand the rules of chess doesn't mean you understand

[Charles Margossian]: 01:46:06

chess.

[Alex Andorra]: 01:46:07

Mm-hmm.

[Charles Margossian]: 01:46:09

And I think that there is a, there is a tension with the reductionist

[Charles Margossian]: 01:46:14

view of physics and the state of the world that we live in, that is

[Charles Margossian]: 01:46:22

kind of related to that. And that sometimes the simplifications that

[Charles Margossian]: 01:46:26

we use, involvement of probability theory of statistics. That's still

[Charles Margossian]: 01:46:32

useful, not just because we don't have the computation to run all the

[Charles Margossian]: 01:46:35

simulations based on fundamental equations, but because it actually is more

[Charles Margossian]: 01:46:40

intelligible to us. And yeah, I think we've only made the time and

[Charles Margossian]: 01:46:44

resources. You could really explore that. Once

[Alex Andorra]: 01:46:48

Hmm.

[Charles Margossian]: 01:46:48

you can run all the simulations you want, what are the actual models that teach

[Charles Margossian]: 01:46:53

you something and that gives you insight?

[Alex Andorra]: 01:46:56

Mm hmm.

[Charles Margossian]: 01:46:57

Let's try

[Alex Andorra]: 01:46:58

Yeah.

[Charles Margossian]: 01:46:58

that.

[Alex Andorra]: 01:47:00

Yeah, that sounds like a fun one for sure. And second question, if you could

[Alex Andorra]: 01:47:06

have dinner with any great scientific mind, dead, alive or fictional, who would it

[Alex Andorra]: 01:47:13

be?

[Charles Margossian]: 01:47:17

I read the question yesterday and I really have...

[Charles Margossian]: 01:47:24

I, um...

[Charles Margossian]: 01:47:30

I like the idea of, that's not a definite answer. Let's put it as an answer.

[Charles Margossian]: 01:47:35

But I like the idea of talking, you know, with one of the founders

[Charles Margossian]: 01:47:40

of hypothesis testing. So, you know, Naaman Pearson, Fisher, and,

[Charles Margossian]: 01:47:46

you know, and by the way, just because I'm having the dinner with them,

[Charles Margossian]: 01:47:49

that doesn't mean I condone everything they've done and their character

[Charles Margossian]: 01:47:53

and their behavior. Just to follow this closure. But I think that I'd

[Charles Margossian]: 01:48:02

be interested to ask them, why were you thinking when you came up with

[Charles Margossian]: 01:48:08

those ideas? And what do you make of how this method that you've

[Charles Margossian]: 01:48:13

developed has been used and misused

[Alex Andorra]: 01:48:16

Mm-hmm.

[Charles Margossian]: 01:48:18

in current days? And what do

[Alex Andorra]: 01:48:20

Mm-hmm.

[Charles Margossian]: 01:48:20

you, and also like, I don't know how much Bayesian methods were on

[Charles Margossian]: 01:48:25

their radar, but what would they think of Bayesian methods now that

[Charles Margossian]: 01:48:28

we have the computations that are available. So kind of, you know, someone

[Charles Margossian]: 01:48:32

from the early 20th, late 19th century, one of those, you know, statisticians

[Charles Margossian]: 01:48:39

that's described as having done something foundational, but also that

[Charles Margossian]: 01:48:43

has worked with, you know, on a field, on a branch of statistics that

[Charles Margossian]: 01:48:48

historically has been opposed to the branch of statistics that I

[Charles Margossian]: 01:48:53

work on. And I think that could be, you know, hopefully a pleasant. certainly

[Charles Margossian]: 01:49:00

an engaging conversation for dinner.

[Alex Andorra]: 01:49:05

Yeah, very interesting answer and very original. You're the first one to

[Alex Andorra]: 01:49:10

answer that. And I had not even thought about that, but that definitely makes

[Alex Andorra]: 01:49:16

sense.

[Charles Margossian]: 01:49:17

Again, an

[Alex Andorra]: 01:49:17

Very

[Charles Margossian]: 01:49:17

answer.

[Alex Andorra]: 01:49:17

interesting.

[Charles Margossian]: 01:49:18

If having dinner with Laplace is on the table, I'm not saying I wouldn't

[Charles Margossian]: 01:49:22

take that.

[Alex Andorra]: 01:49:24

Yeah, for sure, for sure. But yeah, that's definitely super interesting.

[Alex Andorra]: 01:49:29

And I do remember that I, I mean, in my recollection, which of course is

[Alex Andorra]: 01:49:37

very fuzzy, has any Homo sapiens memory. We talked about that in episode 51

[Alex Andorra]: 01:49:46

with Aubrey Clayton. And we took it about his book. Bernoulli's fallacy

[Alex Andorra]: 01:49:54

and the crisis of modern science. I'll put that in the show notes. And I seem

[Alex Andorra]: 01:49:59

to remember also from his book that the founder of hypothesis testing definitely

[Alex Andorra]: 01:50:09

had an active role in putting aside patient statistics at the time. That

[Alex Andorra]: 01:50:13

it was definitely very, very motivated also in that regard, but I don't

[Alex Andorra]: 01:50:20

remember one. why on the top of my head. But yeah. So yeah, I'll put that

[Alex Andorra]: 01:50:26

into the show notes. I should really listen to this episode also personally.

[Alex Andorra]: 01:50:30

This was a very interesting one. So I recommend it to people who, when

[Alex Andorra]: 01:50:36

I get started with the podcast, actually, I think it's a very good first one

[Alex Andorra]: 01:50:40

to understand a bit more, basically, why you would like to think a bit more about

[Alex Andorra]: 01:50:48

the foundations of the podcast. hypothesis testing and

[Charles Margossian]: 01:50:52

Mm-hmm.

[Alex Andorra]: 01:50:52

why it's interesting to think about other frameworks and why the Bayesian

[Alex Andorra]: 01:50:56

framework is an interesting one.

[Charles Margossian]: 01:50:59

Oh yeah, that sounds great. Yeah. So maybe I could have dinner with, remind

[Charles Margossian]: 01:51:03

me the name

[Alex Andorra]: 01:51:04

Aubrey

[Charles Margossian]: 01:51:04

of your

[Alex Andorra]: 01:51:04

Clayton.

[Charles Margossian]: 01:51:04

guest. Yeah. So I mean, add that to my answer.

[Alex Andorra]: 01:51:11

Awesome. Well, thanks a lot, Charles, for taking the time. As usual, I put

[Alex Andorra]: 01:51:17

resources and a link to your website in the show notes for those who want

[Alex Andorra]: 01:51:22

to dig deeper. Thank you again, Charles, for taking the time and being on this

[Charles Margossian]: 01:51:26

Yeah,

[Alex Andorra]: 01:51:26

show.

[Charles Margossian]: 01:51:27

thanks Alex for the invitation and also the service to the community that

[Charles Margossian]: 01:51:33

I think this podcast is. It's really a fantastic resource. So thank