Episode

My guests today are Bill Engels and Jesse Grabowski.

And if those names sound familiar, it's because Bill wrote PyMC's original Gaussian process submodule during a Google Summer of Code years ago.

And Jesse is a returning guest who came on to talk state-space models and Kalman filters back in episode 124.

This time.

They are both here to talk about Gaussian processes and specifically about PTGP, a brand new opinionated GP library built on top of PyTensor and PyMC, with a lot of help from

Jesse's work on PyTensor's rewrites system.

We get into what a kernel actually is, what length scale and amplitude really mean, why GPs are so expensive to fit exactly, and how PyTensor's rewrites

Let PTGP stay fast without hiding the math from you.

We also talk about HSGP, inducing points, and when to reach for which approximation.

And since both of these guys build tools that people now use side by side with AI coding agents, we spent real time on that too.

How to write a library and documentation for a world where an agent might be the one reading your code first.

This is Learning Bayesian Statistics, episode 166, recorded July 9th.

Twenty twenty-six.

Let me show you how to be a good base.

Change your predictions after taking information.

And if you think it now be less than amazing, let's adjust those expectations.

What's a Bayesian is someone who cares about evidence.

Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible.

I'm your host, Alex Andorra.

You can follow me on Twitter at Alex underscore andorra, like the country, for any info about the show.

learnbayesstats.com is Laplace to be.

Show notes, becoming a corporate sponsor, unlocking Bayesian merch, supporting the show on Patreon.

Everything is in there.

That's learnbayesstats.com.

If you're interested in one-on-one mentorship, online courses, or statistical consulting, feel free to reach out and book a call at topmate.io slash alex underscore andorra.

See you around.

Folks, and best Bayesian wishes to you all.

Bill Engels, Jesse Grabowski, welcome to Learning Bayesian Statistics.

Hello?

Hey.

Yeah.

And Jesse, welcome back actually.

Um I think you've been on the show.

I improved my background.

There were complaints about last time I had like a dirty sock.

I was stuck in Shanghai.

My internet connection was terrible.

Uh so here I Yeah, yeah, that's true.

Shanghai will

It was like some behind the scenes that my connection was so terrible.

I think it took us three hours to record that, and then your editor guy was just like, What are you doing?

Yeah, yeah, yeah.

I remember now.

Yeah, yeah, yeah.

I mean, but state space models are worth it, so you know.

Um, so it's fine.

You know.

Um things the juice is worth a squeeze.

Yeah, yeah, exactly.

but thanks again for joining live from Chicago.

It's uh starting to get late for you, so uh thanks a lot.

We we appreciate it with with Bill.

Uh Bill, you're live from Portland, Oregon.

I remember the background.

Uh from the time we we used to work together.

Uh so let's let's talk about you actually be as it's your first time on the show, but finally it's it's a long overdue episode.

I managed to lure you in, of course, with uh both GP and Jesse.

So Yeah, I begged and begged and begged and you finally let me come on.

So thank you.

Thank you.

So Bill is making me look uh good here because I was the one begging.

So but thanks uh Bill for for uh saving my feelings.

yeah, actually can you uh let's do your your origin story, you know?

Um how did you come to what you're doing today?

You know, you now you do a lot of Bayes stats and you do a lot of Gaussian processes.

Um you're like one of the

You know, like the GP guy from the PyMC team.

But how did that happen?

Like where where did you go wrong?

Where did I go wrong?

Man, yeah.

So that's actually a great leadoff, 'cause uh I ran into this stuff for the first time a long, long time ago when I was an undergrad at the University of Oregon and it wasn't in a

stats class or anything, but I was I was working on a physics research project.

And it was with the LIGO experiment.

and the task was detecting gravitational waves from core collapse supernova.

And they used Bayesian methods there.

So um that's how I got introduced to what Bayesian stuff was.

I remember at that time I thought statistics was just like you have dots and then you draw a line through them.

And it was kind of really boring.

You just

Why don't you just draw the line?

So that was very eye-opening.

Yeah, so that was a a Bayesian inference problem kind of at the core.

That was my first introduction to Bayesian stuff, and then working on that is also what led me to Gaussian processes actually, kind of early before a lot of other stats stuff,

which is which is interesting, I guess.

A weird way to find them.

Yeah.

Yeah, so you started working first on on Gaussian processes?

Kind of early on I was just I I just really liked working on this project and got kind of stuck on it and it was more fun than physics homework.

And uh you know, eventually the problem was was really that you had to make a a surrogate model for these simulations of core collapse supernova and um

GPs are used for surrogate models, but I you know it was like learning about, oh, what's maybe machine learning?

What kind of statistics?

It's gotta be some kind of regression thing and yeah, and then kind of just stumbled across Gaussian processes like that.

And then um yeah, that's that's how I was introduced to them.

So before I learned much about statistics or machine learning.

Yeah, okay.

Wow.

Okay, so how was it?

But is it's not the easiest

No.

Entry door to the world of stats.

It was very difficult.

But yeah, I spent a lot of time.

It was really interesting.

I spent a lot of time learning about it and I just eventually thought this was a lot more interesting, well, than the physics stuff, physics homework at the time, you know, and and

kind of decided to go that direction instead of physics.

And so um took a few stats classes, took a machine learning class and um

Went uh graduated and then went to uh went uh get a job and doing data stuff after that.

Yeah.

Wow, damn.

But I don't think I had it all figured out then.

I just barely kinda understood GP's Right.

Yeah, yeah, yeah.

I mean, But it was interesting to learn about other stats after seeing those because, you know, everybody talks about the special case of GP is regression, a special case of GPs

are neural networks, and a special case of

you know, time series, AR models, whatever.

And so they they have connections all over the place.

So it was uh it's kind of fun to learn statistics from working backwards like that.

Yeah.

And then also learning more about GPs at the same time.

So they've always sort of been the little touchstone for for something I can come back to or a a way for me to understand something, thinking about kernels.

Yeah.

Yeah.

I mean and thankfully you did that because then I learned a lot of GP stuff thanks to you.

basically thanks to your production on the on the PyMC side, you know.

Hopefully accurate.

All the tutorials, you know, of like all the things I I read all of that, uh the code for the the sub package.

Because at the time you when you still had to read and write code.

Uh yeah.

incredible.

Back in the dark ages.

Yeah, it was another world.

and uh and yeah, like I mean

I learned so much from all the PyMC GP sub package that you wrote.

So thanks Lord for doing that.

Uh and I think I too thank you.

I'm glad I'm glad people seem to like it.

So Yeah.

It was really fun to work on.

So I appreciate that.

Yeah, I'm sure I'm sure a lot of people will be happy to hear about you today because I've had a lot of uh of people asking me to have uh to have you on the show to talk about GPs,

so um you know, I'm sure.

A lot of people would be very happy in the in the community.

Um Hey, cool.

I'll do my best.

And how did actually so you wrote the sub package for GP for PyMC during Google Summer of Code, is that correct?

That's right.

Yeah.

Yeah.

Um so Yeah, Google Summer of Code was great, by the way.

I

It was really nice to have a period of time to get paid a little bit over the summer.

I this was during my I was doing a master's in statistics when I when I did that.

And it was also really nice to uh work on some code, like a code project, which I really missed while doing my masters, because it's all like math on paper and I missed coding

things that, you know, did something and were used by people.

So it was it was a great time to do that.

But um but yeah, the

The reason I got interested in PyMC before that was that same research project because I needed a software that you could do it's really specific.

You could auto-diff through a Fourier transform, you could set priors and do stuff like that, and it was really easy to kind of extend.

And PyMC fits the bill there and and Theano, right?

Theano was the back end of PyMC at the time and

It had Fourier transforms in it and I don't think other stuff, at least that I could find at the time, did.

So um and doing gradients through a Fourier transform and then a GP was uh ab above my skill level, let's just say.

So yeah, it was like the perfect fit to to look at that.

So that's kind of how I got interested in PyMC and it was easy to experiment a lot and then

was doing summer of code, it was like, well maybe some of the stuff I could um, you know, include it in PyMC.

And so it looked like I got a lot of work done really quickly, but a lot of it was kind of floating around on my computer already.

And it was just uh a lot of cleaning it up and making it nice and making a good API for it.

Um but yeah.

So that's that's how I kind of submitted that to PyMC.

I mean, you had to write the code anyways before.

So you know it's still a

Yeah, it's the right Yeah, yeah but I mean the having the mentors and working with like Chris Fonnesbeck and the other folks that helped me out at the time was super, super

helpful to because that was the first time I'd, you know, written code that had to be like good, that was gonna be used by more than a couple people and was like visible in public.

And so yeah, it was great to learn about how how to do that.

Yeah.

Yeah, no, I mean I

I'm guessing it was nice and uh and yeah, Chris is is such a great mentor and teacher that it's just yeah, my each time I get to work with him I'm I'm super happy and excited,

honestly.

Yeah, yeah, just learned a ton, super encouraging and yeah, it was great.

Yeah, honestly.

and so that's that's how you you started the the GP route on PyMC

Uh and then like did you do you have any let's say, you know, stories from the trenches in how you used GPs for clients for models in in real life, uh and and maybe advice you can

give to to listeners who are curious about them but haven't tried them out yet.

Yeah.

Um

Yeah, I've snuck them into a few models over the years.

Um working at so I worked at a kind of a series of I guess just straightforward data science jobs.

They weren't, you know, doing Bayesian stuff in particular or or whatever.

And yeah, there were always opportunities to use to use Bayesian statistics.

Um and then and then GPs can fit into that.

Because one of the reasons I liked PyMC was that

You know, there's kind of the split.

There's sort of like the GP model as like a machine learning algorithm.

And you can also put sort of embed GPs inside of a you know, like a hierarchical model, like something you would make in PyMC.

And um and that sort of second thing I thought was a really nice thing to do that was kind of underused because there's it's really nice for that time when you have, you know, just

some function.

And you need to sort of fit it and it's within this larger model.

And maybe you don't really care about what the result of it is.

Maybe you do, maybe you don't.

And you need to sort of get rid of that variance or separate that variance from whatever other thing you're modeling.

And GPs are great for that because otherwise, you maybe you're using polynomials or you're using splines or something like that.

And you know, you can't predict outside of the window that.

those are based over outside the training data and then you have to futz around with the order of it and how to set the priors and it's just a bit of a rabbit hole.

And GPs make that really slick where you'll just have a scale on it and then a length scale parameter and two knobs to turn and set priors on and all that model selection stuff

just kind of falls out automatically.

And so that's sort of why I liked, you know, using GPs in PyMC and

It was also nice to have that functionality built in so that, you know, what happens a lot when you're, you know, working on a Bayesian model is that you have to sort of build it up

stepwise, right?

Like the Bayesian workflow ideas and you sort of need to quickly be able to try stuff out and like, well, maybe this thing's a bit of a function.

I could just throw a GP at it and see how it does.

And just making that cycle faster for dealing with little nonlinear function in your model is just a

really handy little thing to have in your toolkit and something that's quick to use.

So you know, there were there's definitely been times at at work where you're just working on kind of a regression model and you wanna use Bayesian stuff because you don't want to

have to explain P values to anybody and the model itself is something not quite off the shelf.

And so you can just whip up a little Bayesian model and sample it and, you know, read the results off in a pretty straightforward way and

Yeah, help people out.

Yeah.

Yeah, and I mean I resonate a lot with that very practical view of GPs.

Um I think one of the best ways I understood them is yeah, like if you have some kind of function um that you don't really know what the form, the mathematical form is.

Um that's a good use case for a GP.

Uh Yeah, yeah.

And that's a very good, usually very good first approximation.

And then sure, if you care about the actual formula, then yeah, for sure.

It's gonna it's probably gonna be better, uh if not if not from a prediction perspective, at least from a computation perspective.

huh.

GP's tend to be uh tend to be computationally expensive, at least the vanilla ones.

but yeah.

Like basically, okay, I have some form of nonlinear um relationship here.

Um let's try a GP, you know.

and and that's usually a super, super helpful uh super helpful approximation of what's really going on.

Yeah, because that's kinda one way to build them up is they're a prior over unknown functions.

And you pick a kernel and the kernel's kinda like what

What variety of unknown functions are we talking about here?

And yeah.

And um you go from there, right?

And um there's a few ways to build GPs up GPs up from you know, things that make sense.

I like that version.

And then I also another one that kind of comes in handy is that a GP is the same thing as a hierarchical model where maybe you have some continuous covariate, right?

Like uh one of the examples that

People have in PyMC a lot is the it's like batting averages.

It's one of those top line PyMC examples.

And you know, you have a bunch of baseball players.

Yeah, exactly.

There you go.

And say say you want to consider age as a covariate.

So all the players now aren't exchangeable.

They all have an age.

So, okay, what do you do with that?

Well, that's just a hierarchical model, except

Instead of all the players just being kind of independent players, they now have this age relationship.

And that's you can model that with a GP with a length scale.

And so it's just a slight step up from a hierarchical model too.

So there's a lot of ways you can sort of find yourself wanting to use a GP.

Yeah.

Yeah.

And you can do you can do hierarchical GPs.

Um so

That's definitely super fun.

there are some tutorials.

I've actually we've been working on that together ah with Bill where there is a two part HSGP tutorial where it culminates in in an example where we show you how to uh build a

hierarchical HSGP in PyMC.

So that's exactly these kind of um these kind of of cases where you would have for instance

hierarchical higher level GP on age and and then each baseball player in your example would be um would be their own GP but that's rand I mean that's sampled from the

population of GPs that we have on the population of players, which is the first GP the higher level one.

So Yeah, yeah, right.

You can

Can go GPs GP's all the way down.

Yeah, yeah, yeah.

This is super cool.

And and now is HSGP, so Hilbert Space Gaussian processes.

I'll link also to a few episodes I have on that in the webinars.

I think when I think yeah, Juan Orduz came on the show to explain the math really behind that.

But basically, in a nutshell, you wanna talk a bit about HSGP, Bill, because I feel like it's been really a game changer for

practitioners honestly.

Yeah, yeah.

No, this was um it was from a paper Solin and Särkkä and then Yeah.

Ru Riutort-Mayol.

I'll probably just butcher the names horribly.

But yeah.

And that that second paper though was like basically like, here's this thing from this other paper.

And it would be really handy to embed in a probabilistic programming language because you

It's it's just such a good fit because so there's lots of GP approximations out there.

And this particular one works it works well for cases that happen a lot when you're working in something like PyMC and you want to throw a GP at some unknown function.

So it works really well for low dimensional things that have like a low dimensional input.

1D, 2D, maybe 3D, but don't go higher than that.

And when you're kind of using the common stationary kernels like exponentiated quadratic and the Matern ones.

And those are yeah, super common.

And you can kind of build these like, you know, generalized additive model type models with them.

And they work really well in there because the approximation is it turns it into a linear model.

So it looks a lot like a polynomial model or a spline model where you have this set of basis functions.

And then the weights for each basis function are parameterized by the length scale parameter.

And you end up writing something that looks like a linear model and it samples really nicely.

Um it's it's an approximation that works well with C C.

And so having that sort of ready to go and easy to use in in PyMC was super, super handy.

Because yeah, GPs are slow.

That is their big problem.

Yeah.

Yeah yeah yeah for sure.

Uh so I'll link to a bunch of these tutorials and episodes in the show notes, folks.

Uh feel free to look into that.

I know we're throwing a lot of technical terms here.

Um kernel, length scale, amplitude.

We haven't defined any of them.

Yep, sorry.

I mean it it that's fine.

we'll we'll probably do it.

Uh we'll see f how how much in the details will go.

Uh depends on what you wanna

Talk about an and show for a PTGP.

But first, before we do that, because we're twenty minutes into the recording and I haven't had asked any question to Jesse, which is uh very rude from me.

Uh even even for a French guy that's rude outside.

Uh so um yeah, Jesse.

So you've been on the show episode one twenty four, it's in the show notes, folks in the related episodes.

I really recommend it because if you wanna hear about Jesse's very interesting background

Uh very diverse and uh very interesting that's gonna be on that one.

Today's gonna be all technical from uh Mr.

Grabowski.

But yeah, like what's your experience with GPs, Jesse, actually?

like do you have worked a lot on on them or are you here just to learn?

I'm I'm the um I'm here for moral support.

Actually there's so Bill

Started with this funny quip that everything's a special case of a GP.

There's a nice coincidence that a Kalman filter is a sequential GP solver.

So in a special case where your covariance matrix is sort of blocked diagonal because you have these temporal relationships, instead of trying to invert the thing all in one shot,

like you're forced to do in a most GP land, you say, well, I'll just take it one block at a time.

I'll do my one little bit of solution and then I'll sort of march down the diagonal, solving the thing.

But the the complete solution is what we in time series land call a Kalman smoother that sort of takes this two sided look at the time series and gives you back a smooth uh

function based on your model.

So direct GP, I'm maybe much less experienced, maybe much less experienced than Bill.

Um, but there's this very nice way in which the things that I work on and the things that Bill work on tie off.

The other nice coincidence is that a lot of the math, well, not a coincidence, as a result of the fact that these are doing the same fundamental linear algebra operations, a lot of

the optimizations and w work I do in the background on PyTensor ended up really benefiting Bill's work um as well as the stuff that I was doing in StateSpace.

So if we have time, maybe we'll talk a little bit about some of the

linear algebra assumptions, some of the new number back end things, things you can do in PyTensor with number now that you can't even do in number with number, which is really

exciting.

Yeah, I think it's a good time now to to dive a bit more into PTGP, so because that's why I invited you guys on the show.

You have this new package out.

PT is for PyTensor, or at least I think.

And GP, I'm pretty sure, is for Gaussian processes.

Yes.

So yeah.

Damn.

That's some good inference on my end.

Yeah.

Creative name.

Creative name.

Yeah, yeah.

but so yeah, well what's up about that?

Like what's the idea?

What's the uh elevator pitch?

Why is that even helpful and interesting to people, Jesse?

And I think it's linked to what you were exactly talking about.

Yeah.

So GP's on PyTensor.

uh Wait, that's Bill.

So for at least nearly.

Sorry, don't Jesse.

Sorry.

Absolutely.

I mean, it's it's good, it's good for Bill to answer this.

because PTGP is his baby.

Let me just say that if you go out into the world and you say, I want to fit a GP, as you know, coming as a more of a novice GP person, it is not at all clear what you're supposed

to do.

Right.

I mean, there's PyMC and I PyMC, so you can do PyMC.

But PyMC is really locked into this full exact GP framework until the HSGP stuff came along.

But if you want to do four, five, six D inputs, HSGP's off the table.

Exact inference becomes off the table very quickly.

So there's GPJax, there's GPflow, there's GPyTorch, and they all kind of have like little idiosyncrasies.

And I think what Bill accomplished with PTGP and what I've been hashtag blessed to be a part of.

is to build something that from first principles says like we're going to be somewhat opinionated and we're going to be useful.

That this is like the practitioner's package.

Hmm.

Yeah.

Yeah, I love that.

Um I feel like it defines you also quite well, Jesse.

Opinionated and useful.

Um I think it's fine.

Yeah, there's that is the nicest thing anyone's ever said to me.

That cannot be true.

Um

Bill.

So yeah, like uh tell us uh tell us more about PTGP and well I I interrupted you, so go ahead.

I'm glad because that was a much better thing than I was gonna ramble about, so now I'll start rambling.

But yeah, I think it's true.

GPs are hard to use and this is my I guess I'll say my kind of semi controversial thing is that like if

If if GPs were really, really fast, they would be everywhere.

People would use them all the time.

And a lot of other models would just be kind of useless, you know?

Yeah.

Which is a funny thing to say.

In the age of LLMs and deep learning.

Especially.

But uh just because there are every other model is special cases of them and they're interpretable and they kind of sit right in this Venn diagram between statistics and

machine learning.

Like if you were gonna color color the middle in, it'd be GPs.

And you know, we talked about having GPs in PyMC where you sort of embed them within a larger probabilistic model.

There's also probably more often the case where you just sort of have GP on its own, the GP model.

Like I have some data set and I need to make a predictive model for it, and I really care about uncertainty and the predictions.

And I wanna have some level of interpretability of like why this model is doing what it's doing, um, then GPs are a great answer in that situation too.

And that's a that's a pretty common thing, especially when you need a model that's accurate and uncertainty matters a lot.

Yeah.

Yeah, I completely agree with that.

I think um like there is like a bad reason for not using GPs in the sense that it's just saying that it's a limitation of the universe.

is computational limitations.

And I think a be a good reason to not use GPs is if you have the actual mathematical formula for your phenomenon and you can derive it and you actually want it and that

actually helps solve your problem better and faster.

But it's like a very small use case, I would say.

Um I wanna can I can I jump in and say I feel like we've been a little mean to GPs up to now.

To say that they're merely functional approximators or like they're black box in the way that a spline model is?

Like I mean a spline model.

That's my black box example.

I'm such a boomer.

Like a neural network or something.

Yeah.

Have you ever went for this you know like easy to grab?

Have you ever seen that meme shapes?

Yeah.

Have you ever seen that meme from um it's called like

I forget the name of the it's a it's a superhero comic and the the guy's talking to his kid and there's like a fighter jet flying in the background and he's Look at what they

have to do to imitate even a fraction of our power.

And I feel like this is GPs and it's like splines.

People do splines, like you get basis functions, or people do like um these has support vector machines with like radial basis functions, or people do nearest neighbors.

And it's all boiling down to like I have this notion that things that are

Close in the input space or close in the output space.

And that's like fundamentally, this is what GPs give you.

And it gives you this really rich toolbox to define a notion of closeness and to say that in this case, close means this.

And in that case, close means that.

And you can, it's not merely right, like putting down, lying down basis functions and then saying like, well, whatever the data says is what the data says.

Mm-hmm.

And I think maybe this is what Bill means when he says they live at this very beautiful intersection point between machine learning, which is like let the data speak, and

traditional statistics, which is let's sit down and think really carefully about a problem and write down a structural model of how to solve it.

And plus they have this Bayesian like je ne sais qua, which really, you know, just pushes it over the top.

Mm-hmm.

Yeah.

Yeah, yeah, yeah.

Um completely.

Actually they are a fun black box in a way.

I don't think so I think describing the stem describing them as black boxes is not very Yeah, it's not accurate as you were saying because they are just I would say it's like ah

let's um maybe there is a good analogy.

It's like they they are like um Shredding Schrodinger's models.

You know, they are black box until you open the box.

then and then it's like perfectly transparent because once you once you

um observe data, they just become multivariate normals, so it's like a perfectly known object, and they also are extremely interpretable, which is kind of like the opposite of

being a black box.

because the kernel function is exactly what you just explained, Jesse, where it's like, okay, what's my notion of what does closeness means for my problem?

That's gonna give you the answer to that question is gonna give you hints

uh as to which kernel functions to look at and select.

And then you had the amplitude and you had the length scale.

So these ones also are meaningly interpretable.

I think, Bill, you can you can give that interpretation for us here.

Yeah, yeah, right.

Sure.

I mean there's your there's your definition of a kernel, right?

Is you want to go, okay, I got this data point and take my X values and I got this other data point and take my X values.

And then the kernel function says it's a function of those two data points.

And it tells you, okay, well, how similar should the y's be here?

Which is a really interpretable thing.

And I think it's something that a lot of people kind of naturally, when they think about modeling, that's kind of how they think about it.

It's in a lot of ways, it's easier to think about closeness in the input space and how that relates to the output than it is to think about like, well, if I increase this

variable.

Does that increase or decrease the y variable and by how much?

You know?

Because then you have to s separate the effect of everything.

And sometimes that's really nice, and you want to separate the effect of everything.

And sometimes there's a lot of interactions that you want to sort of have in the soup with you there.

And GPs are when, you know, kernels are really great for sort of thinking about like that.

The it's almost like a spectrum.

It's like you have these additive models with the terms and that separates everything out.

And that's the easiest additive model to make.

And then you can add interaction terms as you make your regression model better and better.

And then GPs kind of work from the other direction where you have this kernel over all the inputs and everything's interacting.

But then if you want to make your model more complicated, you split it out and you start looking for individual effects.

And so both of these things kind of work at a spectrum and then meet at a middle.

And then

You know, GPs you can have the additive terms in the mean function, is what they call it the mean function when they're working that way.

But it's really just, you know, all these things live on a spectrum of you have an additive regression model with errors that aren't correlated, and then maybe some of the

errors start getting correlated with interaction terms, and then a GP or over on the other side where it's like everything's correlated and and you have everything interacting, and

you can kind of you know, it's it's nice to be able to work

anywhere within that and start from one end or the other.

So and kernels let you do that.

And so that's kind of um what a what a kernel function is.

And they're all composable.

You can add them together.

You can multiply them.

there's like a little kernel algebra.

They have to kind of follow a set of basic rules.

and they're just functions that dis define, like I said, just to restate how similar given if these two X points for two different data points

You know, take these two, how similar are they?

Well then how similar should the the Y's be?

And that's kind of what the kernel's telling you very roughly.

Mm-hmm.

Yeah.

And length scale?

Yeah, yeah.

So length scale is a terrible word for me.

Like as a French yeah person, like in in French it would be a length scale.

So what is a length scale, Bill?

Yeah, so a lot of kernels that we use a lot have a parameter in it called a length scale.

Um there's all kinds of kernels that

Don't have length scale parameters, they have other parameters.

There's useful kernels that don't have any parameters.

You just say they're similar and you go from there.

Um, there's kernels over graphs, there's kernels over trees, there's kernels over all kinds of weird data things, but working in just sort of regular data where if x is five

and another x is 10, the length scale parameter says.

Okay, is five and ten, they're five apart.

Is are we gonna think of that as similar or not?

You know?

And the length scale, what it basically does is scale the inputs.

That's the simplest way to think about it.

And you can do a little do a little algebra on a kernel function like the exponential quadratic, and you can see that the length scale just defines just divides each x by a

scale.

And you know, you can divide x equals five and x equals ten, you know, by

Divide them both by five, and now they're one and two.

So now they're only one unit apart.

And then you go, okay, well that's pretty similar if the length scale is five.

But if your length scale is like a hundred, then five and ten are hardly different, right?

That's basically the same number.

Um and if your length scale is like 0.001, then five and ten are really, really far apart.

And those are hardly the same thing at all.

And so you're why if you have x six five and another x

is 10, then your y values there shouldn't be related to each other at all.

They have really nothing to do with each other.

And so that's that's what a length scale is doing in in a lot of the common kernels like the ones from the Matern family and the exponentiated quadratic that you'll that you'll

see if you look at the PyMC examples or or most examples with GPs.

Yeah.

Mm-hmm.

Yeah, and actually so you'll tell me how wrong that is, but usually when I teach GPs, a way I help

Students understand about length scale and amplitudes.

I use the first I think the most probably the most useful function of the whole PyMC package is the GP plot function that I think you authored during your GSoC, right?

This plot that you get with the envelopes and like first the colors are amazing with GPs because they are so, you know, like uniform and continuous.

It's just always extremely beautiful and pleasing.

They make pretty plots.

That's that's honestly another reason.

They make amazing plots.

Yes.

And you can use and you know, you can use the best color palettes from Matplotlib, like you the viridis and cividis and Magma and they work.

Whereas usually if it's like if you've got categorical data as I have all the time, it's such

boring palettes, you know, it's just like, you know, it's just one discrete color after the other.

It's like, oh my God.

Um yeah, GPs are great because continuous.

So color palettes.

That function is amazing because you can see really the GP if it's 2D of course.

And then so thanks to that, once you have that plot, basically the X axis is the length scale.

And it's gonna like the length scale is basically gonna tell you how far

Do the relation go on the X axis?

Yes.

Um it's like kind of a memory process.

Uh the memory of the process, because like if it's a long memory, then the length scale is gonna be long.

If it's a short memory, short length scale.

And also sometimes the length scale can be um the memory can be spiky.

It can be like if it's periodic, well actually every quarter you remember about something, you know, it's like um or like every year.

You know, oh shoot, I have to do my taxes.

You know, so it's like that's like the the annual parity taxes.

Parity kernel, you know.

Um, so yeah, that's the length scale.

And then y axis gives you the amplitude.

The amplitude is basically a scanning factor, and as you were saying, if you look at the most of the math of the kernels, the amplitude is just like, you know, that small factor

just on the left of the kernel formula, and that basically tells you how far.

Can the functions go on the y axis?

If it's a high amplitude, then well, it's gonna it can go the GP can be very, very capricious, let's say.

If it's a small amplitude, it's like it's gonna be a very calm, very calm GP, you know, like it's not gonna be bothered too much.

Yes.

Yeah, no, that's that's exactly right.

It's the same thing because when you think of those one D ones, you can think about a time series and you could say, okay

And and it's like a memory, like you're saying, right?

So X is five hours and X is ten hours now, right?

And if your length scale is really short, then it's like, well it doesn't know anything.

Like ten hours is nothing like five hours.

So much has changed in that amount of time.

So we don't know anything.

Like what happens at five hours has nothing to do at ten hours.

And so the same thing with X if the length scale is a hundred, then

If you're talking on scales of hundreds of hours and five and ten are basically the same time.

And so then you're gonna have something smooth if you're if you're thinking about things in that scale.

So it's it's the same that it's really funny you mentioned that function because just to go down the cause I was I just kind of hid that in there to help make plots for the

documentation, make pretty plots.

But it has been useful and I use it all the time.

Even just to plot one D stuff because you it's like a nice shaded thing, but it

It's funny, it has a horrible syntax and I just stuck it in there to be handy for plotting because I didn't want to have to like have you heard of have you heard of Hyrum's law?

No.

No, what's that?

It's a law that that any any piece of your API, if you're a programmer, if you have anything, no matter how hidden it is, no matter how private it is, if it's out there for

long enough, somebody will take it and make it an essential part of what they do, and you must maintain it.

Yeah.

Yeah.

I see.

That the that function needs some help.

It has the silliest API that's I don't know, because I mean that that you know, that filters users, you know.

Because if you make it easier to use, then more users are gonna use it.

And then you need to maintain it actually.

And then your plots.

Whereas if it's an Easter egg, you know, like Yeah.

If it's like, you know, animal style at In-N-Out Burger, you know, it's like just just the people who know knows it's there.

You know, it's like it's fine, like you don't need too much maintenance.

It's like you know, like the

The people who use it, that easter egg, they you know, they ignore their stuff.

So you could you could use that, you know.

Just hidden deep in GP utils.

Exactly, yeah.

See GP utils.

Yeah, but they use it all the time for like, you know, even for regressions and and stuff like that.

It's amazing.

Yeah.

Yeah.

No, it's uh I've tried to make it nice a couple of times and then it gets too complicated and I get stuck because I'm like, well, this option or that option and now I have to

actually make it good and then

Yeah.

And then I get a little overwhelmed and have to do things that are whatever, do immediately, you know?

Mm-hmm.

Yeah.

Jesse, you wanted to add something?

No, I wanted to add that it's incredible that you've been on the West Coast for like two, three months and you're already preaching the gospel of In-N-Out.

Like you're really you're assimilating.

I do, I do.

Yeah.

So d I mean, so the fun story is that I I already knew about In-N-Out because I was in Stanford in twenty twelve.

And so that's where I discovered, you know, the truth.

Um so yeah, that was a fun that was a fun three month like I discovered in the same trip, In-N-Out, Jack in the Box, which is amazing when you're a student and you don't have

money and you're hungry at 2 AM.

and and I discovered Netflix.

That was the very beginning of Netflix streaming on your computer.

And in France at the time it was the middle ages.

We had like one big actor.

Who of course didn't want that because they were selling forty euros a month uh serv premium service so that you would get that only on your TV and not even on demand.

So so then when I arrived here, I was like, my god and I binge watched all the seasons of twenty four, which was amazing.

Nice.

I guess I guess the Minitel couldn't do streaming at that time at stuff.

No, no, no, no.

Uh but it just you know, not not uh it was the law, you know.

I d we like that in France, so but in fact I happen to live one mile away from the last Blockbuster.

Oh yeah?

wow.

Yeah.

Damn.

I still haven't been there sometime.

I have not been in because I do not have a VHS or DVD player.

I will admit, but I need to.

Because I I mean

Yeah.

Streaming services are terrible now.

I honestly would think it'd be nice to just go rent a movie.

Yeah.

Not like which serve, it doesn't have it.

Yeah.

I have to unsubscribe from this one and resubscribe to that one.

It's dumb.

Yeah, and anyway.

Night's gone.

The golden age is gone.

Yeah, yeah, yeah.

Now it's like now you actually need to pay like about one hundred bucks for your each month's book.

It's dumb.

Terrible.

I'd rather go just rent the one movie or series that you're gonna watch.

Walk over and

Get a DVD and be a lot easier.

Just the one.

Turn it when you're done.

Actually, who should uh like you know in in the in the biopic they'll do about you and Gaussian processes?

Um what b who should play your role, Bill?

Haha.

I didn't prep for that question.

I'll give you a few minutes, then we'll we'll we'll come back to that.

Yeah.

I always think about that.

That's a I see Ryan Ryan Reynolds, you know, you're like a lovable, like lovable loser type, right?

And like you find that came out mean.

I didn't mean for that to be mean.

I meant Well, I just heard lovable.

That's all I heard.

Yeah.

Yeah, me too.

Yeah, yeah.

I've gotten I've gotten uh Walmart Woody Harrelson one time.

And that could be mean too, but that could also be positive because

Hear me out though.

Hear me out though.

It's Ryan Reynolds and the name of the movie is the Notebook.

The Jupyter Notebook.

The Jupyter Notebook.

The Jupyter Notebook.

Nice.

No, that's great.

I I I buy I'd I'd go watch that, yeah, for sure.

That'd be good.

That'd be good.

Invite us to the to the premiere, please.

I'll I'll prompt the AI to just generate that for the whole world.

It'll be done in an hour.

There you go.

So actually, so to get back to get back to business,

Why why did you use PyTensor for this new this new package?

You know, like what what is that adding what's the difference with the vanilla and HSGP sub module from PyMC?

What would users even uh be interested in in that?

Because PyTensor is awesome.

The end.

No, but like it is though.

It's awesome.

And

One of the things so Pi PyTensor is like a you know, it's like a it's a very boutique hipster deep learning framework with autodiff, you know, nobody knows about it.

It's not one of the big mainstream TensorFlow, JAX, whatever, Torch, PyTorch, one of those ones.

But it's like way better, actually.

Because okay, Jesse should answer this one, but I'm gonna I'm gonna

bumble over a couple reasons why I think it's awesome is that you can if say, okay, say all those big companies decide to never support the big autodiff stuff again.

And there's this new cool one that everybody starts using.

And then everybody goes, we gotta rewrite all of our code and PyTensor, you can just wrap it and now it's a back end.

And PyTensor's still alive, right?

So PyTensor is what Theano has kind of grown into over the years.

And Theano is super old.

Like how old is it?

Thirty years?

Twenty years?

Thirty?

Does he smell like two thousand two, two thousand three.

Like the early commits aren't even on Git it predates GitHub, so the early commits are lost to time.

Amazing.

Wow.

Damn.

Yeah.

So that's it's and it's still around.

It's still kicking because of this kind of uh design choice where they had a C backend, but you could run other stuff.

And so

As long as you can kind of hook into it and run it from Python, you can run anything.

So it's gonna be around.

Um and it also has this cool so okay, Jesse should take over at any moment I start saying things that are wrong.

But like it makes a graph of your compute and it represents it in Python.

So you're like, I want to do one plus A plus B.

It says, there's A and there's a plus and there's a B.

And then and if you do like A, I want to do A plus B divided by B.

PyTensor can look at that and go, why you just want to do A plus one?

That's what that is.

We don't have to actually divide B by B.

So we'll scratch that out.

And your actual program is just A plus one.

So it does these optimizations for you.

So it kind of has this little compiler step.

And this graph it makes, you can work with it.

So it's in Python.

And it's not some deep C level complicated thing that you don't really have access to.

It's just in Python.

And that opens up a lot of things you can do with it too.

So those are the two main things.

And both of them are really handy for making uh GPs.

But so that would be on the back end, right?

Yeah.

Um users would not see that.

So Well.

How unless they want to Yeah.

They might.

Yeah, yeah, no, for sure.

But then what do they

get like do you have actually examples already of of PTGP that you can show here or that we can put in in the show notes?

Yeah, for sure.

Yeah.

We've got some stuff we could show.

Um I'd be great.

So while you get you want me to while you will you get set up.

you have that already, Jesse?

Yeah yeah yeah I'm all I'm all ready to go.

I'm all ready to go.

Take it away and actually I'd I'd love to hear you Jesse about uh the new stuff you were

uh hinting about on PyTensor that made you asked what a what a coincidence.

Yes, there's a lot more cool stuff that PyTensor can do.

So uh let me share my screen.

Can see this?

Awesome.

Yeah yeah yeah.

Yeah yeah so we have a Jupyter notebook not yet uh Bill's by biopic but almost that's right that's right.

And if you're listening I I recommend switching to switching to YouTube.

And like and subscribe on YouTube.

That helps the show.

True.

The So I need to correct the record before I get into the Jupyter notebook, because I made a I made a joke about who would play Bill and I got the actor wrong, right?

Is Ryan Gosling is the hard throb with the abs who is in Jupyter who is in the notebook.

Right.

Ryan Reynolds played Reynolds was not in the notebook.

And who likes to play lovable losers, who gets the girl in the end, which is Bill.

I think it was a very good joke, but I got the actor wrong and I want the show to reflect that.

Thank you.

This is a show about uh about accuracy, so thank you so much.

Yeah.

Yeah.

The abs are then the abs are spot on.

So that part's not good.

Yeah.

I want to jump to the I think you can zoom in a bit actually, Jesse.

I agree.

Yes, yes, yes.

If I can how does one work?

Let's see.

Command shift.

Command plus.

Yeah.

That's right.

But you gotta get a shift in there.

That was what I was mucking up.

So I'm going to jump to the the good stuff here, which is kind of how does this all work with a GP?

So we've talked about GPs are math heavy, they're hard to work with em because they're expensive.

Why are they expensive?

So I have on the screen kind of the two main equations for a GP.

One defines what this kernel matrix is, and the kernel matrix is just a function.

of an input matrix X and it's pushed through some function.

Okay.

The function has to have special properties.

Essentially it ensures that no matter what you give it, what comes out hey Bruno.

No matter what you give it, what comes out will be a valid covariance matrix.

And then you're going to take this valid covariance matrix and you're going to pass it into the log P of a multivariate normal.

And the log P of a multivariate normal has a really

unfortunate property that you need to invert the covariance matrix.

So this first term here for the listeners is y transpose covariance inverse y.

So this inverse step is actually like O n cubed, a little bit less than on cubed, like O N 2.7 or whatever.

But it's really bad.

And it's also quadratic blow up in memory.

So if you have

10,000 data points, that means you need to store a 10,000 by 10,000 matrix in your computer.

That's not so bad.

But if you have a million data points, like you're kind of lost.

So what can be done?

This is

I'll say one more thing that the way that I've written this here with this actual k to the minus one is kind of the math way of thinking about it, where you say, I'm gonna actually

invert this thing and then I'm gonna do standard matrix multiplications.

Um I say that evocatively because in a moment I'm going to make a pitch for why PyTensor's rewrite system is actually nice, even if it's on the back end.

So what can be done here?

If you're if you're sort of hip and you know numerical computation, you kind of know that k inverse y is a solve operation and that you shouldn't do inverses, you should change

this to be a solve.

Right.

So you can write this in code as solve ky.

And then if you're really, really hip, you kind of know that, well, k is positive definite.

And so I could do a Cholesky decomposition on K.

I could store this.

decomposed matrix.

And then I could use a triangular solve or two triangular solves to get this y transpose k inverse y.

And that's a little bit cheaper.

And then all the way for free, we haven't talked about this log determinant of K.

But once you have a Cholesky decomposed matrix, the Cholesky matrix is always lower triangular.

And the determinant of a lower triangular matrix, as everybody of course knows, is just the product of the diagonal.

So now you're not doing any kind of fancy

Linear algebra, you're never calling out to LAPACK, because you already took the time to make this K, this um Cholesky factor of K, you could just read off the diagonal and do

standard math, standard arithmetic.

So GP libraries, what they'll concern themselves with is sitting down and thinking really hard about how do I write this thing out in the most computationally efficient way

possible, and then stuffing it deep into the bowels of their library.

And hoping that nobody ever looks at it because it's not going to be obvious at all.

If you just came from a machine learning textbook that showed you this equation, how it's actually going to be written once you go to the code.

What's nice about PyTensor is what I'm going to do is I'm just going to write it exactly as it appears in the textbook.

So I'm going to say that this quadratic term is just y pt.linalg inverse covariance y.

And this is like absolutely verboten because now you're directly doing this O n cubed inverse.

You fool.

You've left money, you left cash on the table.

And then log det.

I'll use the slogdet function which computes the log determinant of a matrix.

And then I'll calculate um exactly the formula that was shown above.

And I'm eliding details about the exp quad thing here and and and whatnot.

And you could do that.

Okay, now PyTensor has this suite of rewrites.

Anytime you give it a program, it's going to look at the program and say, what do I know about the relationships between the objects in this program such that I can make it a

little bit better, a little bit more stable, a little bit faster.

In this case, uh I have this counts linalg ops function, and it's going to tell us how many linear algebra operations are in here and what they are.

And in this case, we end up with one slogdet function and one solve.

So we didn't write a solve, but we got a solve.

So already we've won.

That's nice.

Now, what's new is that we have this assume function in PyTensor.

And what assume does is it lets you tell PyTensor that certain objects in your program have certain algebraic properties that aren't that are not necessarily inferrable from

just looking at the operations.

In this case, we can tell it look, K is a covariance matrix, a covariance matrix is positive definite, and a covariance matrix is symmetric.

It's going to remember those things and it's also going to reason logically about this over a lattice, like a logical lattice.

And it's going to say, well, if I have a positive definite matrix and I take a dot product with another matrix, well, what do I know about that other matrix?

Well, it's positive definite two.

So that's going to stay.

Is that true?

Are you shaking your head?

Did I get that wrong, Bill?

Is the dot of two positive definite?

It might be wrong.

But the point is.

I was You're good.

Okay, okay.

Yeah.

the point is is.

If it knows something about another matrix, it'll propagate these properties forward.

Here's a more simple example.

I am a diagonal matrix, and I tell PyTensor you are diagonal, and I multiply it by some scalar.

You know, I take i times four.

Like while the zeros are all going to stay zero, the i is going to become a diagonal of fours.

The result of that is just uh a diagonal matrix again.

And actually PyTensor knows that, if you're diagonal, I don't even need to look at the off-diagonal elements.

I'm just going to take that diagonal, which could just be

10 numbers instead of 100 by 100, I'll do 10 multiplications instead of 100.

I'll give you that 90% savings.

So if I tell it that k is positive definite and then I run the exact same uh compile, then what I come out with is one Cholesky decomposition and one Cholesky solve.

So you see the determinant is completely gone.

That's become just a summation, because we're doing the log of a product.

So I just add the diagonal, and then we're doing one Cholesky solve to

um actually do the um the solve got further specialized is what I want to say.

So more numerically stable, a little bit faster, really great.

And everything like numerically is completely equivalent.

This notebook will be in the show notes.

We're going to get it live on PyTensor um examples gallery.

So there's a bit more and it shows you kind of how you can extend it.

And Bill actually did more specialized things.

So he actually specialized the inverse function.

We in PyTensor we specialized solve, but we didn't specialize inverse.

So if you took the gradients of this, you would be un and unfortunately there would still be a matrix inverse left over, and that's kind of a bummer.

Bill said, like, well, let's just change that into another Cholesky solve.

And we can reuse the Cholesky factor we already have laying around.

So you don't even need to decompose it again.

And then it makes it even better.

You end up with four Cholesky solves and one decomposition.

So very hackable, very extendable.

That's kind of the power of PyTensor.

and in this case, it gives us three things.

It gives us numerical stability, it gives us readability and kind of maintainability.

Um, and then the third, and I guess I could just have mixed it into the second, but three is a wonky number, it's approachable, right?

So again, if you're someone who's coming from a textbook and you want to understand how your code works, you go into the library, if you go into PTGP, the point is is that you're

going to see the closest thing to how it's written in the papers.

And then we're going to lean really heavily on PyTensor to give us that same speed up that the get that the fellas in GPJax and GPflow and get uh from handcrafting these these

functions really carefully.

Yeah, and that opens up a lot of stuff, right?

Because a lot of these, like a Cholesky, there can be certain structure in the matrices and you want different solves and and all of that's now on the table.

But I want to just kind of highlight.

Jesse, you went over something really quickly that I think is really cool.

Is that you wrote a a rewrite that replaces the inverse of A with the Cholesky solve when A is known positive definite, and it was eight lines of code, and you did not have to add

it to PyTensor.

It's just sitting in your notebook.

And then now it got applied to the rewrites that PyTensor is doing.

So and this calculation that you ran is that using.

There's a JAX backend, there's a Numba backend, there's a C backend.

Isn't there a what's the Mac thing?

MLX?

Right.

Someone's finishing that up.

Yep, there's an MLX.

And then there's also a PyTorch.

So we've tons.

Nice.

And I mean Yep.

So you could run that in all of those just by changing a flag.

You know, if whatever a certain a certain tensor library is not maintained anymore and there's a cool new one, you can use that one with PyTensor.

And so if you're working on some library and you there's like, there's a rewrite that's just specific to this thing I'm doing, you can just have it in your library without adding

it to PyTensor, or you can add it to PyTensor, one of the two.

It doesn't whatever makes the most sense.

And you know, you can just keep going with what you're doing without.

PRing it in and do that on the side and it's just it's a great library to work with.

And so thanks a lot, Jesse.

Yeah, that's I think it's great to see it actually on the screen.

Um is there already a Python a doc uh website for PTGP that we can link to?

Awesome.

Yeah.

So we'll do that.

Also link to to the examples.

To make this even more concrete for listeners, if you if they wanna use PTGP, so they install it with Conda or uh pip, does that depend on PyMC or is that just PyTensor?

I'm so glad you asked.

Let's quickly look at what a PTGP model actually looks like.

Yeah, perfect.

Yeah.

I think it's great if we show basically yeah what what that looks like, how they would even write a a PTGP model.

So the answer to your question is yes.

PTGP depends on both or builds on top of both PyMC and PyTensor.

And the decision was made very early on, like kind of right away, to say, what what is PTGP not?

It is not a probabilistic programming language.

And maybe Bill, you can talk about kind of your thinking there because it's, I think, interesting and important.

But

The point is that when you need a prior, when you need to simulate, when you need to sample, we're going to reach for a PyMC.

When you need to do something with GPs, some specific GP logic, we're going to have functionality in PTGP that does what you need to do.

So if you look here, I have um a model, a very simple model.

we're fitting on this motorcycle, the Silverman motorcycle data set, which is recording like the head velocity of a crash test dummy when a it impacts.

Yeah.

Mm-hmm.

So this is kind of the case that was being discussed in the introduction where there could be a physical model here.

There's some Newtonian model of physics and mutt that describes how the evolution works, but we're gonna say we're not so interested in that.

We're interested in understanding maybe we're interested in like interpolating or we're interested in um extrapolating or we're just interested in something else and we want to

kind of take out the physics piece of it and then study some other effect that comes in on top.

And so the GP's gonna suck that up and then we can do whatever we

So the actual model definition is just nine lines of code.

We're going to define three priors for the length scale, for the amplitude, and for the observation noise.

And those will just be normal PyMC priors.

And then we're going to take the PyMC prior and we're going to multiply it by a P a PTGP Matern52 kernel.

Uh and you can see kind of the the watchmaker's hand behind the design here.

Because the PTGP kernels look very much like the PyMC kernels.

So if you're coming from PyMC this should all feel extremely familiar.

And then And like other libraries kernels like GPy or GPflow.

I mean they all kinda use the same design for good reason.

And yeah.

Absolutely.

And then the the magic of PTGP is kind of this line here where you're going to specify the approximation that you want to use to estimate your GP.

And I use this term approximation loosely because in this case our approximation is no approximation.

We're going to fit the exact full GP.

But w in the introduction I used the term opinionated and I used I described PTGP as the the kind of blue collar, you know, yeoman's

practitioners uh GP library.

Yeah, I dress for the occasion.

I I put on a shirt just for that joke and I waited an hour for it.

Well done.

That's that's dedication.

The consequence of that is that what is provided are battle tested algorithms that scale up with good guidance on when you should use out approximation one versus approximation

two versus exact.

In this case we have a very small data set.

Everything's copacetic, so we can just use the exact thing and we can get an answer.

The way you get an answer is you just call ptgp dot fit.

Bada bing bada boom.

You're done.

Okay, and so that that fit stuff is what does it do, is that um L E think or is that C C?

Bill, you want talk about the philosophy of fitting?

Yeah, yeah, sure.

Um like I kind of said earlier, like there's the GPs that kind of make sense within a probabilistic programming language.

So there you're going to use MCMC.

And then there's GPs that are sort of like a GP model.

And this is models of GP.

And that's kind of what's going on here.

And in this case, we're doing MAP estimation, right?

Because we've put priors on the length scale and the amplitude eta.

And the noise sigma.

Like we've got those priors thanks to PyMC.

That's already done.

And so we're gonna run probably I think L L B F G S L F B whatever, that one on uh and fit is like a nice quick wrapper over the sort of default basic, just optimize this thing um

using SciPy's minimize.

And yeah, it's uh you can kind of

It's set up so that you can kinda get in there and tweak the optimizer because you can run into weird edge cases.

Like Alex said, fitting GPs is hard and you gotta make it easy, but this is the easy code to just try it out and run it.

Run the minimizer and go, you know, when you're in a a pretty straightforward situation.

Nice.

And although and although it is MAP, so for the kind of good Bayesians in the audience, you will know that MAP is sort of poo-pooed because it doesn't give you posteriors over

the hyper parameters of the model.

And what that often ends up hap what is what ends up happening is that you don't get your posterior predictive distribution, you're just point estimates all the way down.

And so you took this beautiful probabilistic model that you could very well have fit in perfect Bayesian form and just reduced it to

Dare I say a frequentist result?

I mean I'm gonna have to a edit that part out, but just beep it.

Yeah.

Yeah, I'll beep it.

We dropped the F bomb, yeah, absolutely.

Yeah, it'd be like it'd be like an editorial choice.

What's what's cool about GPs and what they have in common with state space models is that they're kind of Bayesian from first principles.

And so even if you only have point estimates for your parameters, you still get distributions out for your estimates.

And so you can see here that we have plus or minus two standard deviations, everything's normal, so that works out to your ninety-six point whatever uh percent HDIs.

Um and yeah, it's it's it's nice.

You get a good result, it's fast, it's simple.

We love And so is that Bill because I remember you explaining that to me.

uh a few months ago or years ago where you were like, no, you'll you're still gonna get uncertainty estimation with MAP.

I was like, What?

What what are you smoking?

Um and you're like, No no, you get that because if I remember correctly, it's because it's it's an MvNormal, a multivariate normal and so thanks to that basically you get that

uncertainty estimation.

If the am I remembering Right.

Yes, that's right.

Um

Because you're it's a Gaussian process, right?

So you've got this kernel.

The things you don't know are like the length scale and the amplitude.

So those make your kernel.

So that's a matrix.

And then that matrix, it's it's in a big multivariate normal.

So then you're drawing samples, and then you have the uncertainty coming from that.

And yeah, you are you are underestimating the uncertainty a little bit because there is actually uncertainty in those hyperparameters.

But if you

If you were to actually do it that way and use MCMC, you slow way down because you need to make a new matrix every time you change the hyperparameter and make a whole new matrix and

then do a whole new Cholesky and a whole new whatever to get one sample and repeat that every time.

So by using MAP estimation, you don't have to do you can do the Cholesky once on your MAP estimate and then draw lots of samples from the multivariate normal instead of doing the

Cholesky every time to just get one sample.

Mm-hmm.

Yeah.

Yeah, so it's like it's slightly I mean it's underestimating uncertainty for sure.

In some cases it it will be slight, in some cases it will be high, so it will depend on the use case.

and and yeah.

That's that's a choice.

But what's the like is there a default way of sampling in PTGP?

Is that is that the fit method or like people can just choose whatever they want?

So And baby Bill, you can

disagree with me here, but essentially what is recommended is that you look at kind of the size of your data.

And if you have very small data sets, we're talking hundreds of data points, then you can just go for the unapproximated.

There should be no issue if you have a modern computer, you can easily solve it.

There's no reason to cheat yourself of the full thing.

As you scale up to something in the order of thousands to like 5,000, 6,000, there's this very nice variational free energy approximation.

Which is an induce it's a part of this inducing point family, and we're going to need to maybe talk a little bit about inducing points.

I hope we have time.

But this is kind of the most exact inducing point formula.

Bill, if you agree with that, please.

Yeah.

It's inducing points when you have larger data sets, larger, not largest, and you also have a normal likelihood.

You need that too.

So there's a lot of these.

That's why GPs are hard.

There's a lot of these like if this, if this, if this, then this is the best.

And and it's it's tricky to navigate that.

And if you have really huge data, just from say five to six thousand to infinity, then you're going to reach for an algorithm called uh SSGP, SVGP, I'm sorry.

Stochastic variational GPs.

because that can do mini batching and you can just let it rock with an Adam optimizer.

tuned to whatever batch size your computer can handle.

so it sort of scales infinitely.

Um at least that's the promise.

Ish, hopefully.

Yeah.

But no, it it scales up for sure.

But yeah, yeah.

Yeah.

And there's there's more exotic and exciting tools coming.

Things like you can take infinite data points and sort of a sphere somehow.

And then that turns it into an O one problem.

So no matter how big your data is, there's these exciting new approximation methods where the inducing points are picked in such a clever way that it scales constantly with your

data and then everything's really, really fast.

Yeah, it's kind of trippy.

The inducing points don't have to live in the space of your data.

They can actually live in some other space.

And in that other space, maybe it's more efficient to do linear algebra.

So Yeah.

Yeah.

I mean that makes sense.

Like based in inducing points, the idea is not to try to summarize your your data set, right?

So it's like I'm guessing does it have some familiarity with uh PCA or stuff like that?

Kinda.

It's easier I kinda think of them as like spline knots.

Sorry, Jesse, go ahead.

Oh right.

I was just I I was gonna push back on PCA just because PCA summarizes the column the columnar dimension.

So you have like a thousand features and I just wanna find so I wanna find ten features that summarizes it really well, but I still have as many rows as I have, right?

Mm-hmm.

The important thing about the the the most important thing about inducing points is that you say, I have too many rows.

Like I don't need all this rows.

I can I can get the same answer with many fewer rows.

I just wanna carefully pick the synthetic rows I'm going to use to solve my problem.

So it's this isn't like sampling.

This isn't like polling.

A little.

You have a full population, then you're trying to find the smallest representative sample unbiased of your population to learn the actual uh election result.

It's exactly like that, yes.

And the inducing points don't have to even be draws from that population.

They can be c some kind of thing in that space or another space.

Yes.

But they summarize it, yeah.

They reduce the rows.

Yeah.

They could be they could be like two voters or three voters like summarized and like

And you you want to keep it, you don't want to throw it out because you don't really know where it's dense, but you maybe have some sense that it's a bit too dense, you know.

And it's like this function that Jesse's got showing on the screen is like the uh

Data's denser than what you would need to play connect the dots through the curve.

Yeah, yeah.

I have a kind of more extreme example as well.

This is the daily the daily Mauna Loa data set.

It's quite famous data set.

And this is daily data going from the present all the way back to 1970.

So we're talking about 15,000 data points.

But if you see, if you zoom in, I mean the data is very highly structured.

Right.

And like you simply don't need all of this.

Now I picked 600 inducing points.

So if you look at the bottom of the graph, and and for the viewers I have

The Mauna Loa data set which goes up into the right and has periodicity.

It sort of wiggles up and down.

And on the bottom, I've plotted the the places where I've put down inducing points.

And I put down a ton of inducing points.

It looks extremely dense, but it's still like less than one tenth of the actual data.

So even with this kind of extreme example, like I didn't bother to do anything smart, right?

I just said, I've got this time series data.

I'm just going to lay down a bunch of dots on the time series, kind of evenly spaced.

And then you get the same answer you would have gotten if you solved the whole thing.

Because there's just too much there's too much re uh repeating information in the data set.

Yeah, that data is amazing.

I I'd love to get that data.

I've never fitted I mean fit or sample the GP on on that that data.

It's incredible.

Yeah, it's a good example.

Classic example.

Yeah.

Yeah.

And so all the all the methods to infer the GP parameters here that we've talked about, this is all already available in PTGP, right?

So fit, sample I'm guessing with uh with classic nuts.

Um and SVGP things like that.

Yeah, except for nuts because they're sort of we're still thinking about where to kind of split the line.

Because that's that's a that's gonna be you're using that within PyMC.

And PTGP sits on PyMC.

It's a it's a fuzzy line.

But yeah, there's gonna be no there's definitely not going to be like a different implementation of nuts that PTGP would use.

You'd use

nutpie or the sampler built into PyMC and maybe you'd use some PTGP code to construct the the GP itself or or not and then and then you'd use PyMC for the sampling for that one.

But yeah the the optimization stuff though is going to be in PTGP because one of the you know one of the goals I have is to make it easy to use, right?

Cause um it's

There's a lot of great GP libraries out there, and they kind of occupy different niches.

And a lot of them implement, you know, the method from this paper or that paper, but there's not a lot of guidance on past the basics on when to use what and also how to fix

issues when they come up.

So often you can fit something and run into problems, or the fit will do this and that.

is a sign of this pathology and this is what's wrong with your model and you need to fit it.

And it's kind of like how how nuts is where you look at the trace and it says something about identifiability or something like that in your model.

and it's a little simpler though because you're mostly doing optimization stuff.

But I just want to make we want to make sure that how to fix that, how to kind of cycle through models quickly because we all work under time constraints and we got to get stuff

done.

Um, so how all of that is uh how how you can it's made with that in mind, so that that's easy to do, you know.

So you're not in that situation where you're like, well, that looks like a cool paper.

Maybe I could try it.

It'll take me a really long time to implement and then it doesn't work, and then is it because I did something wrong, or is it because this method is wrong and what do I do

next?

And um so sort of have the batteries included and the the help kind of included.

Yeah, yeah.

Yeah, I love that.

and um Jesse, you were were you done with the demonstration or did you have anything else to show?

Well s we can do one more.

I didn't want to um Yeah, yeah.

Let's let's think it's great to so that people can see basically what they can do with that.

100%, yeah.

So let me just point out

So we're looking at the VFE approximation.

This is the one that is for a medium amount of data points.

I think I I pushed it pretty hard with this Mauna Loa data set.

It's 15, 15,000 data points.

So it took about nine minutes on my my MacBook to fit, which is a bit longer than one would hope, but it works.

I I wanted to point out one thing Bill mentioned, that it's batteries included.

So we saw this fit function, but everything is also.

Totally decomposable.

So there's this compile SciPy objective function, and it gives you back all of the bits that you need to then go ahead and call minimize yourself.

So if you wish, you can uh customize this as much as you want.

If you love genetic algorithms, you know, you can go into the differential evolution thing in SciPy and you can give it your function and you can let it rock.

Right.

So

We do L-BFGS-B because this is a really good algorithm that's hard to beat, but we also expose all of the components.

So if you're a user who's running into problems and you think that you can do a bit of tuning, that that's all available for you.

Nothing ultimately is hidden from you.

Right.

And oftentimes, even in this situation, if it's if you're having something that can happen, kind of a failure mode, I guess, of this this VFE approximation is that

You'll have problems optimizing both the inducing point locations, the noise, and the hyperparameters at the same time.

Sometimes, um, and you'll you'll just see training, do something weird.

I forget, either sigma goes high or sigma goes low, because it'll be like there's no noise, or there's all noise, and then but the fix there can be to first.

optimize the inducing point locations and alternate between that and the hyperparameters.

And sometimes you want to hold Sigma fixed for a while, optimize those other two things, and then let Sigma start optimizing so it doesn't run off to one of those one of those bad

spots.

And so it's that having this all broken up lets you do that.

And then there's there's like skill files that we're making and the skill files for VFE are in there.

So that when you're running this and you're using an LLM like we all do now.

And it sees this and there's a problem with optimization, it can look through the skill file that comes with it and be like, well, this happened.

This is how you fix it.

And it just knows how to do it immediately.

And so, because there's a few common modes, right?

And so you can either, you know, you can start with dot fit, pg.fit, and just do it, or break it down and be able to hold different parameters fixed or not.

And yeah, just being able to do that quickly and then also being able to hop from VFE to some other approximation and change your priors kind of quickly.

Just so that the workflow of actually using these things goes smoothly for you.

Mm-hmm.

That's of what we're going for.

Yeah, yeah.

Yeah, I I love that.

And since you've opened the kind of worms of agents in AI, that's great.

So I'm gonna I'm gonna play us out uh with that.

Well done, Bill.

Um but so first thanks a lot, Jesse, for this Jupyter notebook and and you've proven once again.

That you're such a boomer because you're still using Jupyter notebooks.

Yes.

Like not it's it's not even Jupyter Lab.

It's like Jupyter notebooks, like twenty tens vibe.

Um in the age of AI agents, that's uh well well well done, uh with your MCP server for Jupyter Nut.

I did.

I wrote my own I wrote my own Jupyter classic MPCP MCP server.

So if anybody else wants to boom around with me, Jesse Grabowski slash uh NB NB MCP bridge.

it installs the server and it installs the bridge and you can collaborate with with an agent from your cloud code terminal and everything lands right in your Jupyter classic

session.

Hmm, nice.

How how is your experience with uh Jupyter notebooks and and AI agents?

So Bill is the one who turned me on, so that came out weird.

You want to believe something?

Believe that.

Yeah, I but I mean we're still talking about Jupyter notebooks.

Bill is the one who turned me on to AI and Jupyter notebooks and there was a blog post from I think the the author of NB Convert, like I got an email like a day later and he was

saying there's a stack you can use and you can plug there's a there's an MCP bridge for doing like collaboration, like multiplayer notebooks, and then you're gonna plug in your

agent and then the agent can work in the notebook.

And I tried it out and it was just miserable and it was just terrible.

I hate Jupyter Lab.

I don't know how anybody can use it.

Like you run a cell and it jumps all the way to the bottom, and then you gotta scroll back up to see what the output is.

Like by default, it doesn't have like compli closing parentheses automatically.

You can't do the like question mark to get the like little pop-up at the bottom that shows you the health.

Like maybe, maybe I'm absolutely like Stockholm syndrome, but it just seems like so much worse than like the old school like bare bones HTML thing.

So I said, you know what, I've got a I've got an agent, I've got a Claude Max account.

Give me uh give me an MCP server for this.

And it like one shot it and it was really good.

And I've been sort of tinkering with it and improving it ever since.

Um I haven't had any problems with dropping, I haven't had any problems with like clobbering each other's stuff.

There's other problems, right?

Like it's janky.

It's the same as everybody's hobbyist AI but it's like my janky software, so I'm willing to overlook the issues.

I think if somebody else does it, they'll say like this is crap.

Yeah.

It's the IKEA effect.

It's like you you built it, so it's got much more value to your eyes than than to others.

Um but so d I'm curious, do you do do you do a lot of plot uh a lot of plots?

Plots?

I think you say plots, right?

That is hard for me.

plots.

Do you do a lot of plots in in Jupyter that you do with the agent that the agent is supposed to do?

And how does that work?

Like what's your experience?

Um It it does a good job.

I I what I ended up doing was I strip all of the

the images when I send the sort of blob to the agent.

So by default it'll just like do its best and then give a thing.

But then there it has a tool call where it can specifically say like give me back the PNG and then it'll look at it and it can make uh improvements on it.

So um I have softened like I got an obsidian vault so I'm not like completely stuck in 2002.

And you can just like rapid fire like plots into an obsidian vault with like some markdown and sort of like DIY

notebook.

and that's really that's really great.

the reason why I like it?

Because I'm like Yeah, the plus great.

I every plot I showed you just now was written by Claude.

You just have to have you know you have to have you have to have taste.

You tell it, you know, I'm French and therefore I need something aesthetic.

And if you show me something bad, I'm turning you off.

And then it gives you what you want.

I mean so because to me, in my experience, even if I give it like a reference notebook

It will come back with plots which are absolutely a monstrosity.

And like but stupid stuff.

Like the title will be phone size one hundred and forty eight.

And then it will come back and be like, Yep, all done.

Uh I followed your example.

The notebook is ready for you to review.

And I'm like, No.

If you manage to

Make that better, I need to hear about the solution.

For sure.

And I mean, part of the answer, like and part of the reason why notebooks were good then and they're good now is because the code's right there and the output's right there.

Right.

So you really I when I just want to like fire off a bunch of prototypes into a folder, like that's fine.

But if I really care about something, I wanna be close to the code and a notebook means that the agent will put the plotting code right there.

It'll make the thing, it'll have font size 170, and then instead of typing I'd like a

A dissertation to the thing about how how dare you have put the title so large I need you to do it again.

Yeah, you just do it.

I just go into the code and I say make it 18.

Yeah.

Yeah.

Yeah.

Yeah.

So that's where like stuff like if you do that in VS Code, for instance, that's the stuff where you would do like the line edit or things like that.

Yeah, yeah.

Okay.

So but Tiki but that means you take back the steering wheel.

Because that's what I end up doing.

Basically I just use the agent for the first version

I know that for most of the code's gonna be fine, let's say eighty percent of the way there.

So, you know, like for technicals that stuff.

But then for plots, if it gets to fifty percent of the way I'm happy.

And and then I take the wheel back because it's just like basically it seems like at least for now, plots with agents is like trying to drive your Tesla on

Sinuous mountain roads.

You know, it's like it starts to do weird stuff that can get you killed, and so you're like, wait, wait, wait, give me back the steering wheel.

it is definitely not, you know, like the the one on one where just you do full self driving mode, you're good.

Yeah.

Okay.

No, it's not like I'm not completely stupid.

I'm not having the worst experience with agents and and plots.

That's reassuring for tonight.

No, I mean What about you, Bill?

Yeah.

Because you told me before the show that actually PTGP, one of the motivations to do that was uh interactions with AI agents.

So can you elaborate on that?

Yeah, I mean I mean for for one, I wouldn't be w even doing this.

I mean, this has been something that's been kicking around in my head for years, where I'm like, oh, if I if someone would just pay me a salary for like two years or longer to just

do this thing for fun, that would be awesome.

But that's not gonna happen ever, you know?

And so now that this exists and it's kind of good enough, this is it's it's good enough to build these things.

And and I, you know, I guess I do this for work, but I kind of like I kind of hate programming.

I hate fixing bugs.

And I hate like getting stuck in little details that I just want the stupid thing to work because I want to think about

whatever problem I'm trying to solve, you know?

And it's great for that.

So, you know, that it the fact that that stuff exists now is kind of why I could work on this project.

But it also kind of got me thinking, it was like, well, if you can just have the robot generate whatever code you want, why do you even need to make a library?

Why don't you just make it do the thing you want it to do?

And then so I guess it was like, well, unless it can kind of just one shot

different GPs and all these back end I mean it's I think there's still a there's obviously still a place where libraries can you you still need it of course but but the fact is that

most people are going to be act interacting with it kind of with an agent in the middle, you know?

and so I was thinking how how should a new library be made in that context?

And um I think

I this is what I'm thinking now on and, you know, subject to change 'cause it's all new, but you know, the the documentation should be you know, the thing can read all the code.

You can just feed it all the code and it can read it.

And the doc strings are just there to to sort of talk about intent and what it's for.

And humans probably aren't gonna read it too much.

And then but the documentation should be really nice to look at and good for people to read and

Conscious of people's attention spans because we have much we have much shorter context windows than uh LLMs do, and there's many other fun things to do that give us dopamine

besides reading code all day, at least me.

And yeah.

I remember what you did, but okay.

Yeah, and it's and so like, you know, that's what I was trying to so having these skill files, right?

Instead of having extensive documentation, I'm like, here's a notebook showing this problem, here's a notebook showing this problem and how to fix it.

It can be in skill files.

And then when you run across it, um the thing, you know, you point it to the skill files that come with that ship with PTGP and it knows what to do.

Um and another thing that is kind of interesting is you know, when you're using it to write.

These GP approximations, I mean all they've all been written in other places because I'm not putting anything new in here.

So how to cite stuff kind of prominently so that other people's work is actually in there.

And I'm not just like, you know, vulturing around and cobbling together a bunch of other people's hard work that they had to do real math and think really hard about how to make

it.

So that's so that's kind of important, I think, for this.

Um but

But yeah, so it's I think skill files are a really nice way to to sort of debug and then and then I also think you can kind of interact with the I also think people aren't gonna s

at least me, I I w I don't sit and just like read documentation, like top to bottom, beginning to end.

But you can kinda have the LM read it and then it's kind of like choose your own adventure where you can ask a question and then it gives you the explanation and the explanation

comes from reading it and and you know, you still look at it, but

Yeah, so I just yeah, how to make a a library that's going to be, you know, has a a purpose before before the LMs can just one shot anything that pops into your head, you

know.

Yeah.

whenever that happens, if ever.

Um and so it's just like stuff that's tested and it works and stuff that's trickier to just sort of one shot and, you know, have have a few kind of like

likelihood functions in there, but the stuff that's easy to extend let the L like have PTGP be kind of be ready for people to want to extend it um to and make it sort of open to that

and sort of say where you could extend it and what it has and so I don't know.

It's you can tell I'm still still thinking it through.

But um but yeah I think it's interesting though.

I would

I should hope so.

It's not like the technology set in stone.

So right.

Nobody knows, right?

Yeah, yeah.

Um but that's so that's super interesting.

You you have a skill file that ships with a package when you pip install it.

And look at that.

That's like it even Riverside is like super super happy with that.

Exciting.

Um so you have that and the skill file.

So basically that's you going into the skill file and you wrote, okay

Like basically debugging steps where where it's well, you're gonna work on models that that looks like that look like that.

Often a problem gonna it's gonna be that.

So you're gonna solve it like that way.

And then iteratively like that, and that's what the skill file is about.

Right.

So there's one in there now.

Let's see, where was it?

But it's uh it's um for uh the VFE approximation.

And kind of all the issues that that you can hit just with that one approximation.

Um yeah, it's in docs, agents, ptgp, VFE.

There's a folder for that.

And there's Amazing.

Yeah.

There's a skill file for that.

And then there's scripts for like making plots to see um like preset plots to sort of help you debug stuff.

So, you know, here's uh here's a bunch of pitfalls with like links.

To like each section in the skill file, right?

And it'll be like Sigma collapse.

Sigma drops to zero.

ELBO appears to improve.

It fits degenerate.

Sigma grows during training while ELBO plateaus.

And the condition number of your approximated whatever because of how you chose the induction inducing points is too high.

And so there's a cutoff, or you have really small eigenvalues there.

Because this is all the stuff that happens when you're fitting GP models and you go, and I think the I think.

Part of the reason people don't use GPs is they hit one of these things and they go, GPs suck.

The test error is high.

Screw this, I'm back to XGBoost or something, right?

And but really there was just like a weird little thing that happened and it's something that you could fix.

Um so there's lots of lots of these sort of issues and um I you know, I like to work on I like to work on problems while working on this so that I hit these things and can fix them

and keep track of them.

And so it's uh yeah, it's like it's a place f where you can kind of collect these things 'cause I think there's a lot of just like there's a lot of folk folk knowledge about how

to fix Bayesian models, like when you see, the you know, my my look at my chains and they're in two different places like this, that means I have, you know, multimodal

something's going on, but you know, I should fix the prior because there's an identifiability issue and um

You know, I think there's a lot of that with GPs out there too, that's you know, maybe more since they're more exp obscure, they're har it's harder to find that stuff.

Um so I wanna have that kind of come with it so that, you know, you don't have to be an expert in this stuff, but it can kind of guide you and so you so you do learn because I

you know, I still think you can't run this stuff without knowing what you're doing, right?

So it's like I mean you can run it, that's the property.

Yeah, that's the issue, exactly.

And then you do something really crazy.

Um but uh but yeah, so it's it's kind of that in between that can kinda help people learn and and do it and kind of that balance between, you know, knowing enough to be dangerous

and then also having the guardrails kind of built in and kind of help tells you why.

So Yeah.

Yeah.

Yeah.

Yeah, I I think that's that's amazing and that's gonna be super helpful to people.

So make sure to add all these links to the show notes.

Um Jesse, I do wanna ask you about, you know, your like the good practices you've seen um arise from your work and and people's work with interactions with AI agents and how they

fit into your work because I do I know you do so many things

And you know so many things and so many literatures and so on, so I'm really curious to hear about, you know, what you recommend people to do and what you recommend not to do.

Use Jupyter notebooks, that's right.

I guess I'll start I'll start with a literature literary reference since you've talked me up.

You've heard of this like top like um Adam Smith had this notion of like a double coincidence, like double miracle.

it was in the context of barter economies where they say like a barter economy could ever work because you need someone who needs exactly a sheep and who has exactly a goat, right?

And then we can we can exchange those things, right?

Or you're a haircutter and so you're willing to cut hair and you need exactly a pound of wheat, and I happen to be a wheat farmer, so we can do it.

Yeah, yeah.

So it's basically ketan.

Like you you know the the the game.

Exactly.

Yeah, everything's like real life.

Yeah.

Exactly, exactly.

You think about think about like

developers, like statistical open source developers, right?

Like there's not a double miracle.

There's like a hex tuple, like octuple miracle.

Because you have to be interested in statistics and like using models to solve a problem.

But that's not enough.

You have to be interested in the the methodology itself.

Like you can't be someone who just wants to like pip install something and run it and get an answer and move on with your life.

Like you're you're as interested in the tooling itself as you are as interested in solving some difficult problem.

Then in addition to that, you need to be interested in programming per se, because now you're getting into like Git and workflows and collaborating with other people and writing

code.

And then in addition to that, you need to be a communicator because now you need to write documentation and you have to make sure that you have examples.

And then in addition to that, you need to be a public speaker because now you're going to PyData and you're talking about your stuff and you're promoting yourself and you're

putting yourself out there.

And like the the the number of people that sit in the middle of that Venn diagram has to be measure zero.

And I think one of the

Powers of LLMs, one of the superpowers of LLMs is it lets you pick your spot and say, I'm someone who's really interested in algorithms.

And I'm gonna really focus in on algorithms, and I don't care so much about writing examples.

Like Claude, write me an example.

First try, no mistakes.

And you get a nice example, and you can really dial in on um focusing on like the algorithms that really motivate you to work.

Or you're someone who's really, really interested in studying some.

specific applied literature and you come to a Bayesian model and you say, I want to give this a try and you run it and it doesn't fit and you get an error message that says, your

model's not fitting, you should re-parameterize it.

And you think, what are you talking about?

Re-parameterize it?

I have no earthly idea.

And so now you can go to Claude and you can say, Claude, you it says I need to reparameterize it, what do I do?

And it tells you about the difference between a centered and a non-centered parameterization.

Or it tells you about

the pitfall of having a grand mean and local means and a random effects model.

And like, oh, silly me, I put a I put a location parameter in all of my random effects.

I need to just have one, right?

All of this folk wisdom, right, that kind of Bill is alluding to gets balled up into this big pile of weights.

And it lets you say, I want to be the world's foremost expert on schools of fish.

And I'm going to use a Bayesian I'm going to use a GP to estimate the locations of schools of fish at a certain time of year in the Pacific Ocean.

And I'm gonna be able to do a really, really good job on that because I have access to this side brain that can sort of take on the job of knowing the ins and outs of that

piece.

My perspective on LLMs fundamentally is that everything is an arbitrage between learning something and doing something now.

And if you go like full Gary Tan and you just LLM nonstop, everything is an optimization problem, like G stack, let's go, then you've

Sacrificed every opportunity you have to learn something new.

Um, on the other hand, if you put on the purity ring and you say, like, oh, I'm bad, AI is uh taking all the water, then you've completely left on the table the opportunity to build

incredible things and to move at pay at really fast speeds in those zones where you've decided for yourself, I don't want to be a domain expert in that, but I still want to be

able to make progress in my problem and

modern problems require that you have this like octuple miracle of skills and interests in order to make progress on them.

So um I'm I'm super bullish on technology.

I love it.

I also have felt myself getting dumber.

Like I've felt myself making the wrong choices on that split between learning something and doing something because your boss comes in and says, I need this yesterday, like fire

it off.

No mistakes, first try.

And I'm like, well I guess I'm

Putting it out with Claude because I do what I have to do.

And and I miss those days where I would sit down with a paper and I would get a pencil out and I would rederive the whole thing and really feel like I understood it deeply.

And now it's just like, you know, Claude, read this, give me a summary, like let's put it into PyMC.

So there's a trade off there.

You have to be cognizant of the trade off, but the same time, like your reach just grows enormously, tremendously, and it's incredible.

Yeah.

Yeah.

And

I feel like so you know what you should you should come back on the show to talk about that, you know.

Like how you use AI agents and and how how that goes into your workflow and the kind of models and so on and the gotchas.

I think the gotchas are the most important because at least in my experience from what I see, I feel like now more and more one of the main skills pun intended is uh is to to be

able to filter the

the LLM's output.

And I feel like, you know, LLMs are a bit like YouTube uh the YouTube algorithm where if you wanna learn something, it can sometimes give you some absolutely incredible gems and

it's gonna work and that's gonna be awesome.

But a lot of the time it's also gonna give you the bullshit, conspiracy theories, videos that look like they are credible and they are not at all.

And you actually need to be

able to make the to make the difference and that's why to tie to your point, Bill, being an expert in what you're building is much, much easier to do that.

Yeah, no, it's true.

I mean, it kinda comes down to some simple stuff at some point too, is they they will just do what you ask them to do, right?

And you have to think really carefully about what you're asking it to do and and this thing you're picturing in your head and then what you actually wrote.

There's usually a pretty big difference between those things, you know.

Yeah.

If you were to pass it to someone else and and then, you know, of course it's not always gonna do it faithfully either, but um yeah, it's it's uh it's an interesting problem.

It's it's not unlike working with someone else, except you you sort of have to get to know them and what they're good at and what they're not good at and how to work with them

effectively and and they're usually

You know, it just like working with people, they can be w far better than you at this thing or that thing and yeah.

Maybe you're a little better at this other thing for a little while and um and just it's like a how to collaborate with this weird little thing that works very quickly and can

make a lot of mistakes quickly and make a big pile of crap very quickly, or you can do something interesting, right?

And you kinda have to not be lazy and say, Yes, yes, yes, yes, yes, yes, yes.

Mm-hmm.

Which is hard to do sometimes, you know.

No, for sure.

For sure.

For sure.

Especially under time pressure.

And this I think it's even harder than be thinking a lot about how you ask what you want.

Because then you can ask it exactly the way you want it, but then at least two things can happen.

It can come back and tell you if d it's done it, but you know, it just thinks it's done it and then it didn't.

It literally happened to me several times.

Um yeah.

And also it can happen that it cannot somewhere in the thinking chain, for some reason cannot answer the question you ask it.

But then since it wants to come back with an answer, it will switch what it's gonna answer.

But you know, like ever so slightly.

And it's gonna be a related question.

It can be a good one, but it's gonna be a related question to what you ask, and it's so close to what you asked that it's gonna take you a while

To understand that it's not exactly what you asked.

And and that's like this is very tricky.

Uh but I think Jesse you wanted to to add something for a while, so I'll I'll give you the floor.

I just wanted to you mentioned the YouTube algorithm and how YouTube has on one hand these incredible learning resources and on the other hand it has just slop.

I think there's a more pernicious failure mode that is relevant to LLMs, which is you take something like 3Blue1Brown, who's one of my favorite channels on YouTube, does

math videos, amazing stuff, and you sit on your sofa in your pajamas with a cup of ice cream.

And you watch a video about how to solve an integral with animations, you think, wow, that's incredible.

I've learned so much.

And then you think, I'm so smart, I'm so great.

I watched a video about integrals.

And then you close it and you play, you know, whatever.

And you have this illusion that you've learned something and that you're like a sophistiqué and everything is delightful.

When in reality, you're still a knuckle-dragging Cro-Magnon.

And

LLMs have the same failure mode where you can convince yourself, like, I made this.

You know, like I you shoot off a dashboard and it looks really nice, and you're like, I made a dashboard.

And you have to realize it, like, no, no, no, no, no, no, no, no, no, no.

You prompted an LLM, you got an interesting response out of it, but you haven't learned anything, you haven't built anything, you got something useful.

Right.

And you can't you can't lie to yourself and act as though you've improved yourself.

By having done that.

I think that's that's quite dangerous and it's something that you have to stay on alert for.

Yeah.

It's something that's happened and it's like a a thing in data science that's been around a long time is you can write a model, get a fit, boom, done, don't think about it, but

then well, actually the this feature that comes in, there's something wrong with it.

And so your model always predicts this thing when that feature takes a certain value because you didn't look closely enough at things.

Right.

And you didn't actually learn about this thing that you're modeling, right?

Mm-hmm.

And I think I think it's nice to think about models as not a tool for doing anything, but like a mechanism for learning about the thing that you're feeding into it, you know.

And it it especially with LMs able to just pump stuff out, like you still need to be super critical and care about getting things right, right?

And that's that's kind of difficult to do, you know.

Because you can, yes, just sit there and feel like, look at all the stuff I did and look at all I learned and the thing is the code is running and it doesn't error.

It's done, you know?

And you know, sometimes you can get great results and and that's the sign that there's a problem.

You know.

Yeah.

And all all this sort of stuff.

And I think I don't know if if this it's like this problem's always been around, but LMs just kind of pour gasoline on it, you know?

Yeah.

Yeah, no, exactly.

I think it's like d and that's a very good one.

I had never thought about that.

But yeah, it's a very related problem to what we had before.

But now it's like yeah, it gets even worse because it's just like bigger dimension.

So Yeah.

Yeah.

Yeah.

That's yeah.

But they'd be great.

I mean, I was working on a i because you sometimes you don't need to know a lot about something too, or and and you need to sort of make a note to maybe I don't know, ask

someone who knows or whatever, but like you know, I was working on an example for

PTGP and it was on like um, you know, water main breaks.

And there's water mains are made out of different pipe materials.

And I know absolutely nothing about that.

And I don't know anybody who knows anything about that.

But LLMs know about water main pipe materials, right?

And in a pretty straightforward way, like, what is this type of plastic?

You know, like I trust the answer that will come out of it for something like that, right?

That's pretty

Like I imagine that's pretty basic for people that know about water mains and how to do water stuff and people that know pipes, right?

So it's like it's like basic stuff from other fields.

Now you just have access to that, which is really, really nice.

And that kind of thing really can help you move on because um yeah, it's it's it's like I've seen people talk like, well, models that are really specific and just just trained on

this one field.

are are useful.

But also the fact that models are trained on everything is also really, really useful too because there's there's all these connections between different things and they can be in

there.

So Yeah.

No, that's true.

Yeah, I just think that basically it's like it it lowers the it lowers the the the floor to to entry, the barrier to entry and like and it also increases the ceiling of what you

can achieve.

But that also means the standard deviation can get way, way bigger on on what you can get.

So

yes, yes.

Um so I guess we'll talk about that again in the show, like almost every episode now.

because it's in our life.

Um but I need to to to get you to let you guys go, especially Jesse, it's very late for you, so uh thank you so much for for being on the show guys.

I think it was awesome.

And Bill, I have to ask you a lot of questions, of course.

So first one, if you had unlimited time and resources, which problem?

Would you try to solve?

Um so I don't know uh what the other answers are, but I'll I'll go with uh I'll take you very literally at unlimited time and unlimited resources and say uh would be really cool

to to figure out what the heck's going on in outer space.

Uh-huh.

Yeah.

Love that.

Because uh hey, we just get to look at it with telescopes and do science like that.

And it's just little bits of tiny, tiny little bits of photons that come off of a very big place.

That we we we can't know anything about that.

And if you had unlimited time and unlimited resources, you could probably learn something about what's going on.

So I think that'd be cool.

Love it.

And and world peace and and world hunger, of course.

No, no, no, you just get twin end

Second question, if you could have dinner with any great scientific mind, dead, alive, or fictional, who would it be?

Well, this one's probably a popular answer, but I'm gonna say Richard Feynman.

one of the one of the things that got me into science or whatever and switched my major in college when I was a aimless youth, I read um Feynman's Rainbow, which is a really nice

book about that his

one of his I think his first year or it he some he was a grad student of his worked with him, um, wrote I haven't read it in a long time, so probably got that it's something like

that.

But it's a really nice book and really inspiring because he's sort of a a person that um put a lot of passion into a lot of different areas and enjoyed learning about stuff.

And um yeah, so he's sort of a inspirational figure in that way and had a lot of kind of varied interests and uh, you know,

w life is short, we all have a limited amount of time here and uh you know, you gotta make the best use of it.

He's like a kind of a good, interesting example of of that.

And kind of appreciating the things that you get to learn about while you're here.

So yeah, I don't know.

It'd be cool to have dinner with them.

None of that.

Yeah.

Topical uh answers.

Like basically you stayed on the same topic for I guess I guess so.

Yeah.

Well done.

Well done.

Yeah.

Impressive.

Awesome.

Um well guys

I think uh it's time to to call it a show, but uh yeah, uh really awesome to have you here.

Uh Jesse, I guess you'll you'll come back on the show very soon.

Um we'll talk about all these all these stuff.

Be also will come back, you know, whenever you have uh you have some new stuff to share.

You're always welcome.

Sure, yeah.

I'll start begging, you know, maybe tomorrow.

Yeah, begging again.

Yeah.

This is this is this is Laplace T V, you know, so I'm always happy to I'm always happy to check.

Where would you go?

I mean apparently Bill has hobbies, but Jesse and I don't, so you know, we're always available.

Awesome.

Well guys, to to have you on the show.

Um listeners, take a look at the show notes, they're gonna be great for this one also.

And uh well, Bill, Jesse, thanks again for taking the time and being on the show.

Thank you.

Thanks for having us.

Thanks so much.

This has been another episode of Learning Bayesian Statistics.

Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbayesstats.com for more resources about today's topics, as well as access to more

episodes to help you reach true Bayesian state of mind.

That's learnbayesstats.com.

Our theme music is Good Bayesian by Baba Brinkman.

feat. MC Lars and Mega Ran.

Check out his awesome work at bababrinkman.com.

I'm your host.

Alex Andorra.

You can follow me on Twitter at Alex underscore Andorra like the country.

You can support the show and unlock exclusive benefits by visiting patreon.com/slash learnbayesstats.

Thank you so much for listening and for your support.

You're truly a good baby and change your predictions after taking information.

And if you're thinking I'll be less than amazing, let's adjust those expectations.

Let me show you how to be.

Good days you change calculations after taking fresh dating.

Those predictions that your brain is making.

Let's get them on a solid foundation.