Episode

Let me show you how to be a good bee.

This episode is a special one.

It was recorded live on stage at StanCon 2026 in Uppsala, Sweden, in front of a room full of Stan users and developers.

My guests are two pillars of the Stan world.

Paul Bürkner, professor of computational statistics at TU Dortmund and creator of BRMS, is back on the show after episode 35, and Steve Bronder, software engineer at the Flatiron

Institute.

And Stan core developer joins us for the first time.

With Paul, we dig into amortized Bayesian inference, why some models don't have a usable likelihood, and why simulation-based calibration isn't enough to trust a neural posterior.

With Steve, we talk about what's holding GPU computing back in Stan, the push to move the whole algorithm onto the GPU, and his work with the Eigen team on GPU-ready linear

algebra.

Along the way, we cover some to zero constraints.

Leave one group out cross-validation, a very clear definition of the Jacobian adjustment, how both of them use AI tools, and some great questions from the audience.

A big thank you again to the StanCon organizers, especially Sara Hamis, Måns Magnusson, and Aki Vehtari for inviting the podcast to StanCon.

This is Learning Bayesian Statistics, episode 167, recorded August 20, 2026.

So you had to be a good beesie And change your predictions after taking information And if you're thinking I'll be less than amazing Let's adjust those expectations What's a

Bayesian?

It's someone who cares about evidence

Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible.

I'm your host, Alex Andorra.

You can follow me on Twitter at Alex underscore Andorra, like the country, for any info about the show.

learnbayesstats.com is Laplace to be.

Show notes, becoming a corporate sponsor, unlocking Bayesian merch, supporting the show on Patreon, everything is in there.

That's learnbayesstats.

dot com.

If you're interested in one on one mentorship, online courses or statistical consulting, feel free to reach out and book a call at topmate.io slash alex underscore andorra.

See you around, folks, and best Bayesian wishes to you all.

Hello my dear Bayesians.

Just a short announcement before continuing with the episode that I will be at the Carnegie Mellon Sports Analytics Conference in Pittsburgh, Pennsylvania on October 23rd

and 24th, 2026.

And on October 23rd, I will actually teach a workshop about Bayesian inference.

You guessed it.

And also causal inference and how to use Bayesian and causal inferences.

To infer the effect of a player on his team.

So if you wanna come meet me and also attend my workshop, that would be awesome to see you there.

And the idea will be to teach you how to causally infer the effect of a player and the effect of a team and how to disentangle them with one model, which will be the soccer

factor model that I'm working on with Max Göbel.

It will be actually the updated version of it.

So if you're curious about causal inference,

Gaussian processes, time series, and PyMC, I am sure there will be something for you to learn there.

You don't have to be interested in sports analytics or football or soccer to get something from it.

If you are, well that's great because that's a sports analytics conference.

But if you want to apply that in your marketing models or biology models or whatever, you will definitely get something from that because what counts is really the concepts and

The time series data and the structure that we will be looking at.

I put all the details in the show notes.

You will have the link to actually sign up for the conference.

So hopefully, see you there on Friday, October 23rd.

And also a big thank you to Ron Yurko for inviting me in the podcast over there because actually my workshop will also double as the Learn Bayes Stats episode.

You can find more about

Ron and his background and his research actually in sports analytics in episode one forty of the podcast.

That was a very fun one, so I definitely recommend you tune in for that one.

And on that note, let's go on with the show.

Welcome back.

Now a special uh live recording of Learning Bayesian Statistics Podcast.

Uh it's my favorite podcast.

Uh and Alex Andorra is the uh creator of the post podcast.

Uh I highly recommend it.

And it's been great to uh hear thoughts from many uh Bayesians and it's I'm certain that it's been also helpful for uh making Stan and uh Stan developers and different parts of

Stan uh like more widely known, spreading spreading the information.

Uh

And then we have uh Paul Bürkner, Steve Bronder as the guests for this podcast.

But please, Alex, take over.

Yeah, thank you so much, Aki, for the the great introduction.

I'm very happy to hear it's your favorite podcast.

it's definitely the only Bayesian podcast, so by definition it's the best and the worst at the same time.

So that's uh that's a very rare qualification.

Uh if you don't know the podcast, that's very hurtful.

But you have uh QR code at the very top here.

that will bring you to the website and you'll see that we have now one hundred and sixty three episodes since twenty nineteen.

Aki has been on there, Paul and now Steve, uh and so that's why the the podcast is is great.

Definitely not thanks to me.

so you'll see lots of episodes there.

next week is gonna be episode one hundred and sixty four.

That's gonna be a very special one.

I recommend tuning in for that one.

And the second QR code is something very new.

Uh I've started uh an Instagram account for the show to give a bit of the behind the scene feel because I get a lot of um questions about that.

So if you wanna see Charles Margossian kicking my ass at football, Paul Bürkner being very good at football, actually me being very bad, you can you can go in there and you'll

see that later.

Um but today

We're gonna talk with Paul and Steve.

We're gonna open to QA more or less ten minutes before the end of the recording.

So if you have questions, write them down and uh and we'll get to them at the end.

But right now, let's focus on that.

Feel free to, you know, like upload, make noise, things like that, because this will be uh real

Podcast episode.

So if you want to be on the show, that's uh it's your time now to be a heckler or something like that, you know.

Okay, so let's start.

Um I recommend tuning in to episode 35.

I think that's the one where you were on, Paul.

So but that was a long time ago.

So I'm very happy to have you back today.

It's definitely awesome.

Um but if you want to have like the full background and history of Mr.

Bürkner.

That's gonna be that one because today we're not gonna do that.

Uh however, Steve, it's your it's your first time.

Um how are you feeling?

A little anxious to be recorded.

I always feel little anxious.

I'm kind of a background character, I feel like, in a lot of Stan stuff.

And so it's kind of like I don't usually show up on main stage.

Well, I'm I'm very happy that uh you're here today and that you're here even though you're a bit anxious.

Thank you so much.

because yeah, you're doing so so many amazing things that we just saw a bit, we had a bit of a window in there.

But I always like to dive a bit more into, you know, who the people are on the show.

So yeah, Steve, can you give us your origin story?

Basically, what do you do today?

Because I think you have a very original job, you know.

It's just like what what do you do?

I'm a Stan core developer.

What what's your job and how did you end up doing that?

Yeah, so um I am a software engineer at the Flatiron Institute, working under Bob.

Um a lot of my job is maintaining and helping uh building things out for Stan, as well as like when we have new contributors helping shepherd them into becoming more long-term

developers, if someone has a feature making sure it comes in and it's up to the standards so that we can be able to maintain it in the long term.

in terms of uh how I got to where I am, uh

In undergrad, I actually started as a political science and journalism major.

Re.

Um Yeah.

I like thought I would just like I just really like wanted to be like an on-the-beat style, like, yeah, like what's really happening in politics.

Uh and after a while I just kind of like I had a bunch of this sounds so silly.

I had a bunch of friends who were like electrical, or they were like engineers or like electricians.

And they made like $50 an hour.

And I was like, look, that's at that time I was like, like that's sick.

I kind of want to I kind of want that.

And so I was like, you like I don't know if I like school or not.

I am going to pick a major where like I have a very high likelihood of dropping out.

like I'm gonna actually try, but like I am gonna be open to failing and like that's okay.

So I did econ and math.

and I ended up really liking them.

So I actually completed my degree and everything and then uh just fell in love with like forecasting and stuff like that.

Um I then went from undergrad uh to uh Columbia to do a masters

in their quantitative methods in the social sciences program that Andrew Gelman helped run.

and at that time I took a class on GPU computing and deep learning because I was very pretentious at the time and I said, well I'm gonna take these classes just so I can talk

about them at parties and tell people how stupid they are.

And I uh did these classes like okay GPUs are cool.

But like

I was like, you would need so much compute to make deep learning actually work.

I just like don't think this is gonna be a real thing.

No one's gonna ever do this.

And that was the most wrong I've ever been in my entire life.

Like I readily do admit that.

m

Yeah.

Yeah, yeah, you're right.

Like I I have to make up for my mistakes.

Um but uh I uh was doing this GPU stuff and Ben Goodrich was my master's thesis advisor on a very non-Bayesian thing, bless him.

Uh and he was Well, we want some GPU stuff in Stan.

Would you be willing to help us try to work on this?

And I said, totally.

Um and I wrote like the first implementation of Stan's GPU code.

And like as we were about to release it, uh Rok Češnovar showed up and was like, hey, like we're like also working on a GPU for Stan thing.

Like what do you think about our code?

And I was like, like this is so much more efficient than my code.

but it was like written a little bit, I was like, you know, like let's work on making this let's integrate all of my stuff together.

so that's kind of how I started working on Stan.

Um and then I went into finance and I was doing structured product stuff for a while and it was fun.

And just kind of working on Stan like in my at home in my underwear at night.

and then ah

For after a while I was kind of complaining to stuff about Bob and he was like, like, well, why don't you come up to Columbia?

Uh we can like you know have you come and work for a while.

And then Aki Aki also let me take a sabbatical in Finland, which is like one of the best times I've had in my life, and I'm very thankful.

Love Helsinki.

and so I uh kind of got back into being like a main contributor for Stan again, and then kind of moved to Flatiron from Columbia, um kind of f following Bob.

Uh

And so that's kind of how I got to where I am today.

This is a long winded way to say it, apologies.

No, that's that's perfect.

That was great.

Thanks.

Um yeah, I didn't know you had started that way, so you you really like that was a very senior senior way.

So that's great.

I mean the I'll what I really like about these paths usually is that you end up doing something because you really want to do it and really like it, which I think is definitely

your case.

Um what's interesting though is that I find your

Profile is much more geared towards, well as you were saying, software engineering and really kind of like, you know, the the background of of Stan and what we're using and how

we're actually getting uh the the results.

Uh more than on the statistical and modeling side and actually getting into that because you wanted to do Bayesian statistics.

Is that is that yeah, am I correct here or feel free to say I'm completely wrong.

ah

I think you're right on a lot of points.

Like I think that um my undergrad was very focused on stat stuff.

But then over time I kind of just ended up liking software engineering and like the computational side more.

and so that kind of ended up being my focus, yeah.

I still love stat stuff.

I do like a bunch of weird little like stats.

Most of my day-to-day like weird C++ stuff.

Um but like I still have like weird side projects that I just are like being able to riff and like fun little Bayesian models.

Yeah, yeah.

Okay.

Uh I think later we can come back on that uh you know GPU stuff that you that you mentioned.

I think that'd be interesting because that's also top of mind for a lot of people right now, obviously.

but now Paul, let's uh let's talk a bit more about you.

So uh I'm guessing most of the people you know you interact with in the Bayesian world, they identify you as the BRMS guy.

Is that is that correct?

Like do you get recognized sometimes in the hallways and you're like, Aren't you the BRMS guy?

It would certainly be good for my ego, but I'm not sure this has happened so much.

em

Maybe because you didn't put the picture put a headshot of you on the README of BRMS, you know, and then people start recognizing.

could be, it could also be that I kind of moved out a little bit of out of the fields, like, I don't know, psychology where I'm originally from, for example, where BRMS is a lot

used, people may or may not recognize me and moved more like to computer science, deep learning aspects, where nobody probably very few people know Stan in the first place.

Only if they know to compare MCM, CSAM +, nobody knows BRMS or do they care about R or something.

So it's like, it's always humbling to go to a new community.

where you are kind of completely like a blank person, eh you have zero standing.

Yeah, that was the that was the segue actually of my question.

So yeah, when you came on the show that was to talk about BRMS, very BRMS-focused.

But you do a lot of things.

And like so we've seen one of the presentations, you were one of the co authors.

I know you do a lot of stuff on GPs, uh HSGPs, you love them also.

and you'll obviously do a lot of things more on deep learning now, especially with Stefan Radev and his group.

Um so

Yeah, if you're interested in that by the way, amortized Bayesian inference, definitely recommend checking that out.

the BayesFlow package.

Um we have a bunch of episodes about that on the on the website, so definitely listen to them.

But yeah, like you you wanna talk about us, um you wanna talk about that, uh maybe like what you're doing in that in that field right now.

Yeah, I can talk a little bit about the history how I got there.

So when I was like working on BRMS and Stan and all sorts around that, things ended up working so well, you know, that I got bored.

So I had to move away from Stan a little bit because it used to and still works so well.

Still surprising to me.

And so I met Stefan, one of my close collaborators now in a BRMS workshop before COVID pandemic, right before COVID pandemic.

He didn't need to be in the workshop at all.

I don't know why he

went there, he do the psychology PhD.

And then he showed me this paper on the original kind of, using essentially uh neural networks as a replacement for MCMC.

So the statistical Bayesian models remain the same, but our estimation is now done with neural networks rather than with MCMC, which has advantages because we can train on

simulation data so we don't need likelihood densities or prior densities.

And we get real-time inference if we do it well, but of course we are having all the pain that neural networks come with, lack of guarantees, unstable training, et cetera, et

cetera.

He was showing me one of the papers which recently got rejected by WeirdReviewer2.

So I was reading that, I was really excited, and then we were like, yeah, mean, there's some COVID pandemic happening, COVID something happening in China, or like in Northern

Italy, will not really hit us, then two weeks afterwards, everything will close.

Also great at predictions.

Hehehehe

And then from there we worked on a lot.

I was just bringing actually the expectation that I want these kind of amortized Bayesian inference methods to make them as reliable and robust as MCMC.

I have no idea whether we ever achieve this, but it's like a goal that we can work towards.

uh But multiple orders of magnitude faster.

um

in the sense that after like neural network training, which may of course be slow, then you get a new data set, you have the posterior distribution, and then you get another data

set, you have the posterior distribution.

So that's really impressive.

So the promise is huge.

And the challenges are also huge, so that's why I was excited to work on that.

But I also know that Stefan and I, of course, have been talking a lot to people, like also who are like, Stan adjacent or even like part of the Stan team, of course.

And you know, for a lot of things like where we have all the likelihood densities available, like everything is kind of coding up in Stan, we don't have like, I don't

know, tens or hundreds of data sets that we need to run the same model on.

We don't need.

uh We are just happy with Stan or other probabilistic programming language, so there's a certain community that I'm also building methods for, and I'm not sure if they're ever

going to be applied or appreciated by the community, but we will see.

Yeah, I mean and right now so basically what Paul said is is my just the let's say definition of amortized Bayesian inference and why that would be even helpful.

Have you now that you've been working on that for a few years now, have you seen any any interesting applications of these kind of methods, especially using BayesFlow of course?

Yeah, we do have a lot, especially when we have models where we don't have the likelihood densities available or easily available and we just, but we can efficiently simulate from

these models.

So we then simulation-based Bayesian inference, which we did have like approximate Bayesian computation for decades, but it didn't really scale beyond, I don't know, a few

parameters.

And neural networks really humiliated when they came and humiliated all the benchmarks and made SBI really feasible and scalable.

And as a side project, we got this amortized

property that gives us real time basing.

And so all the kind of models like physics models or sometimes models from psychology or biology or chemistry where we don't have these analytic likelihood densities which you

don't see at StanCon of course.

um For these the kind of simulation based number test methods are like really like

the state of the art, if you will.

em But of course, they would still not reach the reliability specifically for out of distribution data for which the model is misspecified, uh then MCMC would have in

comparison if we could apply MCMC.

So a lot of the core applications where this has been really successful is in places where we cannot efficiently do MCMC in the first place.

Because if we did, probably MCMC would still win at least in the accuracy and the

the trustworthiness that we get out of it.

It's my goal to improve that trustworthiness of the neural methods so that we get faster inference but similarly accurate.

Yeah.

Can you

I gonna is there is there methods yet or measures that you can use to say, well this didn't quite work as we wanted it to?

So um in a way, So basically, there's, I mean, of course, know simulation-based calibration, right?

And for MCMC, it's kind of, or variance of this is kind of gold standard in a way.

But it only works, and you know that I'm just giving for context, it only works, of course, if we are simulating from the exact same model uh that then we are essentially

fitting,

But of course if that works, let's say with MCMC implemented in Stan or wherever, then we have reasonable trust that also for real world data which may look a little bit

different than the simulated data, still we get an accurate posterior approximation.

So now for kind of neural simulation-based amortized inference, of course we can run uh SBC, but it's more of a convergence diagnosis because we have been training the neural

networks with exactly this kind of simulated data.

And we also know that on the neural network

works are only good to predict something that is close enough to the data that seen during training.

So while for SBC we may easily pass, for example, we don't know if it will work for real-world data.

uh

if the real world data is somehow out of distribution of the simulated data.

Because our models are usually misspecified in some way or form.

So that there's, and this is actually what we're working with, to trying to diagnose not when the data, when the model is misspecified for a given data, but when it is

misspecified in ways that will affect our...

accuracy of the analytic posterior, which of course we don't know.

So it's kind of a black box and the way we're doing it kind of, we are having two neural networks.

One is actually the posterior estimation and the other one is um summarizing the data into ideally, like informative, ideally sufficient statistics.

So we can check whether we are out of distribution in the space of sufficient statistics and if we are, chances are our posterior approximation will also be bad, which kind of

gives us diagnostics and we have a paper soon coming out of this that extends previous methods.

of hours and sometimes it works and sometimes it doesn't and um so I'm still not happy.

We with embedded the plus we also had similar issues as well.

Yes.

Thanks Steve for the the assist.

Uh I appreciate that.

And uh yeah, actually to make sure people understand if that could be helpful in their case, you know, SBI, so simulation based inference and amortized Bayesian inference.

can you explain because you said they were especially useful when there is no likelihood.

Right.

So

Why would that even happen?

You know, like can you explain to people what that means to not have a likelihood available?

Yeah, so for example, think your model is expressed while like a differential equation.

I don't know, some epidemiological models about, you know, we can use COVID, right, about the changes in cases and the people who are like infected and immune, et cetera.

And of course, if the models are simple enough, these differential equations either have like an analytic solution or we can use kind of some kind of differentiable ODE solvers as

we have in Stan.

It will slow down.

things but it will kind of work but as our differential equations expressing our mechanistic model let's say in physical chemistry or biology gets increasingly complicated

we may not have

solvers in the way that we can solve through them and compute the densities and specifically the gradients in any numerically stable fast enough manner.

So it's not that we would have no way of ever computing the likelihood density or prior density for that matter.

It's just that it becomes so computationally expensive and numerically unstable that de facto it's like we say it's intractable in a way so that the numerical approximations

don't work, but simulations from these models.

easily doable differential equations, like partial, whatever differential equations, one aspect, the other one, you have like big latent variable models, like latent variables in

psychological sense, you have, I don't know, theta your parameters, then they go to Z1, then they go to Z2, they go to Z3 and Z4 and Z5, and then you have Y in the end.

And you don't care about Z1 to Z6, they are huge, there are thousands of latent variables, you don't want to estimate all of them.

So you would like to use the marginal likelihood, marginalised

all of these latent variables and you have just no way of computing this.

So then it becomes de facto intractable because there's so much stuff in between that you just want to, you know, not having to deal with.

And if you simulate, you simulate your theta, you simulate all your latent variables, you simulate your y, you throw the latent variables away, never look at them again, and then

you have the relations of theta and y in simulations, that's all you need.

Yeah, yeah, thanks.

I think that was very clear, but we we'll see if we have more questions about that.

Feel free to ask later.

Um so now

Coming back to you to you uh Steve, I heard that neural networks they are quite expensive to to feed, right?

So often we need a GPU for that.

So you made fun of yourself earlier for making that forecast about GPUs, deep learning, it would never work.

I'm not sure even though it turned out to be a wrong forecast, I'm not sure it was a bad forecast, probabilistically.

Right, maybe at the time when you were making that forget that was actually the most uh the most reasonable one.

So I guess now your like your vision of that has changed.

Can you can you tell us what's your yeah, how you're thinking about GPU computing these days and and yeah, where where do you see that coming, uh especially when it comes to

Stan?

Um yes.

So my my forecast was in twenty sixteen.

So was like I think if I had done a little more homework, maybe I could have made it more informed, but anyway.

Um how I think about GPU computing is mainly that like as I kind of said in like the talk that we had before this, the main thing that stops Stan's current implementation from

being as good as we want it to be is that it really just resides inside of the function that we use for calculating uh the

uh log likelihood where what you really want is something where the entire algorithm is moved over to the GPU.

Because for the RAM to communicate with the CPU it is pretty fast.

Um but for the RAM to then communicate with the GPU it is just very, very slow.

Um and so you want to because that's why you kind of want the whole algorithm over there.

Um right now the kind of way I'm thinking about it for at least with Stan is that there are some ways that we can do that and port a lot of our back-end stuff to uh the GPU that

could take a lot of time.

Uh Andre I was just speaking with had some schemes and ideas that I think are great.

Um

In terms of overall GPU computing stuff, I'm actually working with the Eigen team on a nice sort of uh C++ linear algebra library that would the the end goal for people who don't

know, Eigen's a C++ library for linear algebra that essentially like if you use scientific software, it is almost surely that does linear algebra, it is almost surely using Eigen.

It has a very pretty API front end and me and another guy are trying to figure out how we can get this front end to also work for GPUs.

Um so that's kind of what I'd like is to have like a lower level um GPU access schema.

'cause things like PyTorch and stuff you can totally just go do.

GPU stuff right now.

For some particular applications, like for autodiff, we do have to be really careful with memory management.

And that means like we need to be able to actually go in and say, like, I need memory for this now, it must stay over here.

Um and things like PyTorch don't quite give us like the accessibility that we need.

and so right now that's kind of how I think about it.

there's also stuff like JAX, where like I I use JAX and I like it.

Um it means that like you can't write for loops, for instance, which is

Like I, as I said before, like I like for loops, I think they're cool.

Um, but like they're incredibly hard to optimize over.

so for Stan, we're working with another contributor.

Me and me and Brian are working with another contributor to allow for people to write for loops, and then in some cases, we can collapse them such that they could then be ported

over to the GPU.

Uh JAX also has these vmap style functions, and we're also trying to port those into Stan because the

Then once we see those, we can better reason about can this go over to the GPU or not.

ah And so it is really a lot of backend machinations to allow users to write code just in a very natural way that can still be ported over to GPUs.

Does that answer your question?

I yeah.

Yeah.

and is that something you're gonna focus on especially in the coming month or what or what's another thing that's gonna take most of your time, you think, in the in the coming

month?

right now this sort of um Eigen GPU library has extremely beta.

it's like super we're very in like the rough edges of making things how we want, figuring out what the API should look like.

Um in the coming months it's definitely gonna be a big focus for me, just because like I think it'd be nice to have for a bunch of other like we're building it so that Stan and a

few other libraries we're like very specifically targeting can go use it.

Um

And so that is like my kind of main focus and then uh doing it such that then we can essentially reuse a bunch of our current code, but then over on the GPU.

okay, yeah.

Sounds like a yeah, hard problem to solve.

Is yes, it's there's a lot of not nice words to say for it, but it's gonna be a big effort for sure.

Yeah.

but it's it's fun and I like doing it.

Yeah, yeah, no, for sure.

I mean uh of course.

And something I liked so when you just did the the presentation earlier with uh with Brian about um yeah, like basically update on where Stan is, I took some notes because there

were some interesting things in there for me, like the topics I uh that are dear to my heart, especially of course the sum to zero vector and matrix.

So I worked uh on that with Adrian Seyboldt, who's uh just over there who

spearheaded that effort on the on the PyMC side.

If you're curious about that folks, episode seventy-four with Adrian.

That's the first one.

He explains what you what a sum to zero vector even is and why you would even want to do that.

And it turns out, as you were saying, you definitely want to do that a lot of the time.

It's gonna make your models much more efficient.

And then we had Sean Pinkney.

I don't see you here, Sean.

ah But

Yeah, I think it was episode one hundred thirty three.

So if you wanna hear how that was developed on the Stan side, uh definitely give that a listen.

But yeah, I'm curious, so you talked very briefly about that in your presentation and you were saying that you were yeah, you were saying definitely a a big advantage of using that

on a on a bunch of models.

Um yeah, do you do you wanna add anything on that?

Um

I think Sean did an awesome job of kind of explaining it.

Uh he I I love Sean.

Every once in a while we'll just talk on the phone and he'll be like, yeah, like by the way, like here's this like I'll mention some like weird problem to him and then like a

year later like, by the way, like I like totally figured out how to do this and I'm like, that's insane.

and he's just a really fun guy.

and so uh I think he probably said a lot of it uh more than I could or I think Paul would probably be able to talk about no, you give me the look of this is this math is also

beyond me.

Yeah, I agree.

I agree.

Whenever I think about it, the basic parts are clear and then I want to implement it in BRMS and then Adrian made me aware again of some of the edge cases when things get like we

get multiple grouping factors and they interact and then things get really really messy and then suddenly I'm scared of implementing it.

So there I hope that someone whose math I trust will tell me if whatever and someone ends up doing in BRMS or whatever is correct.

Mm-hmm.

still a lot of talk with like I think there's been some discussions right at StanCon about like well like what if you have like correlations between the like parameters inside of

the sum to zero vector?

Like how do you 'cause it's just as simple as like, okay, like multiply it by a Cholesky correlation Cholesky of a correlation matrix, or like do you want to do some other schema

to be able to set priors?

There's still I think stuff we have to figure out before we can like how do you say standardize it in something like BRMS?

Um but like I think it's very doable.

It's it's foreseeable.

Yeah, yeah, yeah, yeah.

so just to set some context here for people who are not familiar with sum to zero, it basically each time you're using a categorical predictor, for instance, in in a

regression.

The classic way of dealing with that and have your model not over-parametrized is to do I think it's called reference encoding, right?

And so you take like you consider one of the categories as uh as your placebo if you want, and then everything is in comparison to that reference.

But often turns out there is no reference category, and so the sum to zero uh handles that over-parameterization by

Um fixing the mean, right, Adrian, or the sum.

Yeah, that's the I mean I think it it turns out to be the same.

So um it's very intimidating to explain that when Adrian is in the room, by the way.

so yeah, but anyways, uh that allows you to do to not have your model being over parameterized without having to fix a reference category.

So for instance, super helpful if like I used to do a lot of electoral forecasting.

you don't have a reference party, in France for instance.

So then you just look at the at the mean of the parties.

And so that's great, but then yeah, as you were saying, when you start having a lot of categorical predictors in your models and I don't know, you have a multinomial likelihood,

then it starts being uh it starts getting messy.

But um it's extremely powerful and you you you are able to do a lot of of very interesting stuff with that.

So I definitely encourage you folks to

Use that in your models more often because more often than not, it will very probably um sample much faster.

Uh I think I had another point about that, but I forgot.

Uh yeah, actually uh a question I had when I saw your slides earlier, which is a term we hear all the time in the Bayesian world, but I think is almost never defined.

So maybe one of you two can define it.

It's Jacobian.

It's like all the time people talk about Jacobian.

I think it's important we actually define okay, what what's that and why is it important and why do we talk about that so much?

Paul, you wanna take a

I try to explain it to my students.

Usually in school they had to do integration, and I hated integration in school, so nowadays I have two ways to do integration.

is from whatever, like LLMs, or Ravié, Ravié loves integration, so whenever there's a difficult integral I'll it to Ravié, Ravié will solve it be happy about it, and I'm happy

because I don't have to do it.

But anyway, so then probably people learn integration by substitution.

when we substituted something, I don't know, we had x and then x was difficult, the integral was difficult to solve after x, then we just transformed into y, y is equal to f

of x, but we cannot just plug in f of x and y, we cannot just change it, but we need to adjust the integral m and we need to adjust the boundaries m for the integral to remain

valid.

That was always very awkward for me to get right.

And essentially, this adjustment terms that we not integrate over

x but we're integrating over y and they are related in some kind of nonlinear manner.

This adjustment term is kind of the Jacobian adjustment.

Of course, in densities we need to make sure that they continue to integrate to one and if we just like plugged in f of x for y or the other way around, then the density would no

longer integrate to one so it wouldn't be a proper density anymore.

So essentially we have like just messed up our math and that's why we need the adjustment term which is Jacobian adjustment in multivariate terms.

In multivariate cases it becomes kind of...

the determinant of a matrix, it's kind of bit more messy.

Right.

Beautiful.

Yeah.

Thanks, Paul.

I think uh I'm gonna make a YouTube short so that part it's gonna go viral in the Bayesian world.

But I think it's great.

So Jacobian, another term you hear all the time is Cholesky decomposition.

Like for sure I'm just thinking about that right now.

and also mass matrix adaptation.

And that so episode one sixty three, thanks to Adrian Seyboldt, actually I understood what mass matrix adaptation is and and why it's useful.

So

Give that a listen if that's something you've been curious about.

yeah, Steve you wanna you wanna add something?

I don't know about anything we just mentioned now.

I worry that if I say something I'm not an expert in this place.

I think if I say something, I'm probably gonna fudge it up a little bit.

There's probably people who have been on the podcast who are gonna be much better at talking about.

But I think it's also interesting if you have questions about that actually.

because I mean if you as a Stan core developer still sees stuff like that that you're like you you can't really understand, I think it's that means very probably a lot of

people have these questions.

And we have a lot of people in the room that probably can answer that question.

So like yeah, feel free to also ask those.

Do you?

Yeah.

I'll think for a minute.

Not right now.

Cool.

Something

no yeah, before asking that question, I definitely wanna talk about what you're up to for the coming month, Paul.

Because I know uh I know you do a lot of stuff.

So yeah, I'm I'm always very curious to hear what you're uh what you're working on right now and what you're paying attention to.

So the hardest part at the moment is trying to find motivation to write more grant proposals.

Hehehe.

It's like I had grant proposals and then you know you can write follow-up ones and I should do it and I promise to do it and then it's hard and so I need to finish these.

But research-wise it's kind of the diagnostics that I talked about how to diagnose whether out of distribution data will actually cause any issues for the posterior accuracy of ABI

without knowing the ground truth and posterior not having MCMC reference.

And also something we have observed is that sometimes when we have some of these

statistical models and we learning these amortized posteriors with neural networks.

And let's say we have five parameters for the purpose of the argument and four parameters are just very well learned and the joint posterior is very well learned.

There's one parameter where we just learn the prior.

So we don't really learn how to condition on data properly.

So we just recover the prior there.

So we're learning something as opposed to nothing, but we're not really learning what we want.

And of course, this could mean that the parameter in that statistical model is truly not identifiable, but usually we know it is.

And if we run MCMC, if we

could, then we would see a good recovery of the parameter.

But we don't.

And we figured out that there are some issues with the neural networks, the implementations, but generally how they stick together and how basically the gradient

flows through them, which then leads some of the neural networks to learn to ignore data for certain kinds of parameters.

So we have figured out some architectural changes and additional loss functions that we can employ to help the gradients move in the right direction so we can more reliably

learn.

joint posterior of essentially all parameters and not ignore certain ones.

That's something that will hopefully be done soon.

I currently have four people doing four case studies.

em Let's see how that works, if this is going to speed things up or slow things down.

uh

I think that's one of some of the main things.

And then something that I used to work on, which we have a paper coming out soon, is about leave one group out cross validation with Aki and Anna, both being in the room, and

Svenja, one of my PhD students, who, and basically where we try to make this kind of.

almost as efficient as LOO-CV with importance sampling, but in a hierarchical model, if we're leaving out individual groups, and there's a bunch of approaches we can do, and I

think we have found good trade-off between accuracy and speed there.

So this is almost done, but just a little bit advertisement.

okay, yeah, super cool.

So that'd be to make sure everybody understand, leave one group out.

So that'd be LOGO.

Yes.

that's nice.

Yeah.

That's nice.

Better than logo.

Cool acronym.

uh So that means I guess it's for hierarchical models, right?

So that'd be you have

by definition groups in your hierarchical models.

And what you're doing is that you're holding out one of the groups per se and then evaluating how the model does predicting that group out of sample.

Yes, exactly.

Because if we had a hierarchical structure and we would do LOO-CV, then we would predict only single observation within an existing group.

And that usually would do very well because we have all the observations of the same group still in the data set.

So it often makes sense to just, if the task is to predict a new group, then to leave also the groups out.

But there's just like a lot of difference between the full posterior and the local posterior without that group.

So it's very hard for importance sampling to bridge that gap, or usually impossible.

So we have to do some smart integration.

em of parameters to make it work.

yeah, okay.

Yeah, because basically you can with hierarchical models you can predict a new individual in an existing group.

So existing group here means like the model has seen in seen it during training, but you can also try and predict a new individual in a new group.

So like completely new group that wasn't seen at at training.

Um and that's obviously harder.

and so you want here that be a way to understand before having to do that kind of prediction how the model you're working on is

Like how good the model is at that task.

Yes, and in a way that ideally of course we can implement this in BRMS or something so it's just one click of a button and there's a little bit of more work involved to make it

work like this but I hope you'll have algorithms that can be done.

Yeah, yeah, That's also where so that's a funny thing where um sum to normal can be kind of a like a a solution that then brings a problem because if you do sum to zero sum to

zero, sorry.

Uh if you do sum to zero on the existing groups but then you want to predict a new group, that means you're breaking the sum to zero, I think.

Um looking at Adrian

depends on the level.

if you correct within Stan for the sum to zero constraint, so what comes out of Stan is as if you didn't have the sum to zero constraint in the first place, which is kind of what we

do, then we do the post-processing in R, Python or wherever, then we don't have to bother.

But if we kind of just didn't do the correction in the middle, I don't know, things would become very messy and I would prefer not to think about it.

Yeah.

I've I've thought about that a lot because I use that a lot for hierarchical models.

especially in sports analytics that happens a lot because you have new players coming in and but they are part of a group.

And so yeah, like it's it's definitely it's definitely hard.

The best solution I found is either you're lucky enough that you're over parameterized model, so without the sum to zero parameters

You know it's over parameterized, but it still samples fine because you have enough data and so HMC is is fine.

If you're not lucky enough to do that, then you need to use the sum to zero constraint, but then the out of sample predictions, what I would do is that I would kind of write a

new model that I would but I would not sample from it.

I would just do posterior predictive sampling on that model, but using the trace of the previous model.

So it's like you have a new parameter, but that's informed by the previous run.

It's bit artisanal, I think.

I'm not sure it's the best mathematical uh solution, but that does work.

Yeah.

yeah, I still have a lot of questions, of course, and I have the last two questions for you guys uh at the end, but we can already open the floor to the questions actually.

So are there already some questions?

Yes.

So I think I have a mic over here.

yes, perfect.

Thank you, Sarah.

Here.

So you will be officially in the podcast, thank you so much.

And uh I love reinforcement learning, so you will get a sticker at the end.

Thanks.

Uh my question's for Paul.

Uh in simulation based inference, like neural networks generally we don't have a very easy way of like gauging their certainty in their predictions for uh given parameters.

So like is the posterior in simulation based inference does that widen or contract depending on how much training data you give your neural network?

So I'm just rephrasing the question, tell me whether I understand correctly.

So of course, when we do simulation-based inference, way I do it, we still target what I call the analytic posterior that also MCMC would target, would we have MCMC available.

Then of course, the question is, given infinite amount of training and infinite expressible architectures, we will converge to the analytic posterior.

That doesn't help us practically, but theoretically, we will converge to the right thing.

Then of course the question is what happens, let's say our architecture is flexible enough but we have, our training is not sufficient so we haven't converged yet.

And then it depends on the loss function and the kind of architectures, whether the approximate posterior that is kind of...

incompletely trained is more like over underestimating the uncertainty.

So since we are usually training with forward KL loss, we can do it because in simulation-based inference, we need to just simulate from the model, which is something we can do with

a little bit of tricks.

So we're using forward KL, so it's more like mass covering.

So it will probably

rather overestimate the uncertainty or in the cases I just earlier mentioned like just estimating the prior which is kind of an overestimation right often but if we use

different kind of loss function let's say we would use for the purpose of the argument reverse KL or so we would probably not be mass covering but we were like like miss modes

that we would be underestimating so it depends a bit on the loss functions what theoretically happened in practice I have seen all kinds of things going really wrong but

mostly it's just that certain parts of the of the analytical

posterior is just not learned parameters and like not learning from data at all.

So it's mostly that we are, our posteriors are too wide, which could be the conservative place.

But of course, if we were to just learn the prior or like only learn from half of the data, that's too wide, but not really a satisfactory or really conservative solution, if

that was going in the direction of the question.

Yeah, okay.

Thanks.

Beautiful.

Thank you.

Hearing us.

Uh any other questions already?

Okay, perfect.

So I get to ask more questions.

Thank you guys.

m

Yes.

Yes, something I wanted to ask you, uh, because I get that question a lot also, and obviously it's changed uh the way we work a lot, is how do you use AI tools in your own in

your own work right now?

And Paul, especially I know you you teach, you have students, how do you handle that like and how do you what's your advice to your student on how to use these tools?

And then Steve, I'm definitely also super curious.

Uh AI tools, like in general.

AI.

AI tools.

Yeah.

Sorry.

Would you like to start?

yeah, totally.

Uh so s I use AI stuff all the time.

I learned recently I was like the second largest token user at the Flatiron Institute.

yeah.

I love uh I will not disclose their name.

Um I'm fine saying, yeah, I I don't I like AI, it's cool, it's

Yeah, no, I know.

I d but my question is not do you like it or not?

It's like how do you use it best?

You know, because it it's a tool so

It's nice.

So sometimes like if I'm like working on um how do you say, I will use it for like the initial review pass or like to help me with a review.

When someone submits code, I will usually have it do a first pass look to say, well, what seems weird here?

Um I will often be the person who like so I am often sketching out the top level API.

Um and I will kind of treat like a vibe coding session where like but by vibe coding I mean literally like I am coding and then I go, Hey, like this is what I want this to look

like.

Just fix all the syntax errors.

And then it goes through and it fixes all my syntax errors for me, which is

Super nice.

uh Codex in particular is like weirdly good at the Stan Math library.

um And yeah, it's just it's really nice.

Like if I generally need to write a new function with a gradient, I will I have a little skill that I'm gonna be putting up a pull request for soon, which literally like gives it

access to things like SymPy to like be able to like uh find the gradient calculation uh and like do the prim and reverse mode.

Or like the regular

regular function and then the reverse mode function.

and so like I think I use it quite a lot for like how do you say just speeding up myself.

Like it's not like it can do all of my job for me, but it can do it can give me like a twenty percent boost in my productivity.

ah So I think that's those are the main ways that I'm using it right now.

Yeah, awesome.

I think that that will be very helpful um to listeners.

Paul?

I also often use it for example for vacation planning, it's very useful.

Different goals and then it writes uh some nice summary and it tells you of course you made a great choice.

But also um sometimes I just forget some stats, machine learning, whatever concepts and then I just ask you can you remind me of the math or like having this kind of rough idea

of how things should relate to each other, can you just

give me a thought and then know can give me some math or some papers or whatever and of course I have to verify sometimes it's garbage, sometimes it overlooks something but often

it's kind of you know going in the right direction so I don't have to look things up and then I can only look things up that I'm unsure about so that's very useful but of course

you know we have been here growing up kind of without AI so we had to learn a lot of stuff.

ourselves so now hopefully we are skilled enough to verify what AI gives us so it's a speed improvement but doesn't make us stupid uh or replaces us.

But what you asked about students, so I recently had a course about a Bayesian data analysis which essentially Stan and BRMS and I'm giving another course in the other

semester about simulation based inference so I'm always ending up with a project where people have like a data analysis project essentially with the methods we teach and then

they are giving a presentation graded and a report which is

And you need to pass both presentations first and then report a second.

um And I have seen like lately of course that the reports are getting less and less valuable to the degree that I skip over most parts of the reports.

Nobody cares about the introduction.

I can write this so I'm just checking very small things.

But what I do is like basically, you know, check the presentation in a way that you know, you kind of have to make an oral exam with them because the rest AI can, on the level of

student projects, can do very well.

So what happens is a lot of students, they don't really show up to the course, they don't really show up to the question session for the project.

they just come, give an okay presentation, which is clearly mostly AI-generated.

Their code is clearly AI-generated.

Even the way they talk is okay, it's not AI-generated, but sometimes they're reading from an AI-generated text, they're like, can you please give me your phone, you know, I would like

to give you a presentation.

But then when you ask them...

basic questions, which if they had done it themselves, like just failed once or twice themselves, wouldn't trivially know.

They have absolutely no clue.

Let's say ask what kind of neural architecture does he use for your ABI model, which he wouldn't know if you've written the code.

And it's fine that AI writes it for them, but they just.

didn't learn anything, so I let them fail.

But it's really sad to see because they're replacing themselves with AI, so they're making themselves useless, which I'm not sure is the right strategy, because if you're only as

good as the AI tool that you're using, why would any company or professor or whoever hire you?

So I'm trying to teach people, you need to understand it themselves and then just use it as an assistant rather than the assistant taking over your thinking.

But the other perspective on this is I'm getting old and I'm getting conservative, so this is like new and strange and scary and perhaps I'm just not quick enough to catch up on the

reality.

Nice.

Like several asterisk wear old.

So take it the U I will say like I for kind of like I another way that I think about it is that like we had

Compilers like ten years ago, like you could not sometimes trust the things they would do.

like a compiler, internal compiler error ten years ago is not a crazy thing to see.

and we're kind of in this weird phase where it's well like you can just kind of now dictate what's happens and the AI is kind of a you kinda think about like a compiler.

Um we're not really at a point where like we can like totally trust the compiler, right?

Um and it becomes also another weird thing where it's like, well, like are these skills that we're gonna need?

Like I don't I don't know.

I don't have a strong opinion about it either way, but I do find

Overall nice.

Yeah.

Yeah.

No.

yeah, I resonate with a lot of what you just uh mentioned here.

We don't have a lot of time left, so I won't I won't dive into that.

But last opportunity for questions, Sean?

Here you go.

I think you already have a sticker, right?

Yeah.

uh

I don't think it's okay.

Um so actually this this theme about you know when you have an AI or a software, right, more generally that that gets it done for you.

Like I I feel like we've been having this conversation about

Stan and BRMS and other products around it for a long time, right?

So the fact that you know Stan abstracts the inference, like should we be concerned that some of the users don't understand the algorithms that underlie Stan, whether it's MCMC or

variational inference.

And same with BRMS, is the abstraction maybe preventing people from understanding uh the the model.

Well enough.

How can we think about the balance that needs to be struck for different users?

Do we really always want to make it as easy as possible for the users, or is it sometimes useful to to challenge them and and and force them to think harder?

I think personally kind of the way I think about it is like, well, like we were having uh when we were having lunch the other day, we were talking about like someone who was

teaching students and it's like, well, like one of them just used BRMS and they were like, This is fantastic.

Like I kind of only teach them like a very basic machinations of like how the low-level stuff works.

I don't really make them look at Stan itself.

Um and there was someone else who was like, Well, I really want them to understand what happens, right?

Like I think it's just like vitally important.

I kind of again kind of going back to like uh computer stuff, I can r I used re you know, twenty years ago you'd really know how to write assembly.

Like it was like very important if you know how like read and write assembly.

And that's just way less now.

Like now there's like things like Godbolt that lets you just read your assembly from your compiler, but like you don't have to like actually write it yourself.

And I think that like that's kind of analog that's kind of like an analogy to like what I how I kind of feel now, where it's like, well, like it's good to be able to read this

stuff.

Do you always need to be able to know how to write all of it?

Maybe not.

um Yeah, you are definitely younger than us in that perspective.

No, I mean, I always, I don't have a good answer to this, but other than thinking, of course, you know, it's always a question of what, if it's kind of the AI stuff that we're

seeing now, is it, definitely something qualitatively different, right?

But does it fundamentally change the game of putting abstractions over abstractions over abstractions?

Like, is it just like another step quantitatively or is it something qualitatively?

I don't know, right?

also cannot know because of course it's kind of something that came up when I was like already over 30.

So it's...

I am not entirely sure to what degree students need to understand, but of course I'm in the stats department now, so there's a certain expectation that they would need to

understand certain lower level stuff.

If I were to teach psychology students, I would just need them to understand what you put in, roughly what the model does, what you put out, can you trust it, can you not trust it,

how to interpret it right.

I never teach an MCMC sampler, like just roughly how it roughly works, perhaps Metropolis.

So I think it depends essentially on the audience.

But then perhaps also my expectations are just strict because the people we are now having as students, they will go to industry, they will build some databases or some analysis

pipelines.

They will probably never ever touch Bayes ever again, right?

So um I'm not really sure if I'm just getting too conservative and too...

uh

I don't know, strict with what I expect people to know, or if I have a valid point.

Amazing.

Well, um last question for you two.

I usually ask two questions to end the show, but I don't wanna put everybody on hold.

So just one for each of you.

Um quick answer.

Steve, if you could have dinner with any great scientific mind, dead, alive or fictional, who would it be?

Uh it would be Jim Simons, who started the Flatiron Institute.

Uh I got to meet him twice and he was one of the most charming people I've ever met and he was just a very I'd be very happy to see him again I guess.

Okay, great.

Short answer.

I appreciate that.

And I think you're the first one to answer that, so well done.

Paul, you already answered the two questions when you you came on the show a few years ago.

So since you like to travel, I'm gonna ask you what would be your ideal destination for next StanCon?

Like you have carte blanche.

um I think I would like to have an island near the equator where you have beautiful beach but also still intact coral reefs or restructured coral reefs and we could just go have a

beautiful conference and then all go to the beach and enjoy nature while it lasts.

think that...

Yes.

That would make me very happy.

Yeah, okay, awesome.

Yeah.

So sounds like we're going to the Galapagos, guys, next time.

So thank you so much.

Um thank you so much everyone for uh being here, everybody who is listening right now.

Obviously, thank you so much Paul and Steve for taking the time and being on this show.

And of course please big round of applause for

the organization committee, especially uh Sara Hamis, Måns Magnusson, sorry for the pronunciation, and obviously also Aki Vehtari uh who all invited the podcast here and did

a tremendous job.

Thank you so much.

This has been another episode of Learning Bayesian Statistics.

Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbayesstats.com for more resources about today's topics, as well as access to more

episodes to help you reach true Bayesian state of mind.

That's learnbayesstats.com.

Our theme music is Good Bayesian by Baba Brinkman.

feat. MC Lars and Mega Ran.

Check out his awesome work at bababrinkman.com.

I'm your host.

Alex Andorra.

You can follow me on Twitter at Alex underscore Andorra like the country.

You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbayesstats.

Thank you so much for listening and for your support.

You're truly a

Good basie and change your predictions after taking information And if you're thinking I'll be less than amazing Let's adjust those expectations Let me show you how to be a good

daisie Change calculations after taking fresh data Those predictions that your brain is made Let's get on a solid fountain

Shit.