The Ethics of Data-Intensive Applications - Designing Data-Intensive Applications by Martin Kleppman

Nathan Toups (00:00)

how much of our digital exhaust or

this stuff has been used to manipulate people's behaviors and their sentiments about things. think a big part of the argument of who controls TikTok is based on who gets to censor which data and why. And that should be concerning to anyone because again, it's not that in good times that we have these powers, it's in bad times the people who might abuse them.

Carter Morgan (00:15)

Right.

Nathan Toups (00:30)

are the ones that we should worry about.

Carter Morgan (00:39)

Hey there, you're listening to Book Overflow. This is the podcast for software engineers by software engineers where every week we read one of the best technical books in the world in an effort to improve our craft. I'm Carter Morgan and I'm joined here as always by my co-host Nathan Toops. How you doing, Nathan?

Nathan Toups (00:51)

Doing great. Hey, everybody.

Carter Morgan (00:53)

Well, thanks for listening everyone. Make sure to like, comment, subscribe wherever you're at. Leave a five star review, share the podcast with your friends and coworkers. It's been fun. You can join the discord and in the discord we have an introduce yourself channel. And I was like hearing how people discover the podcast and we've heard several people who said, yeah, I was talking to a buddy about it and they recommended it. So really does help the podcast grow. And if you're interested in booking time with Nathan or I on Leland for a one-on-one coaching for about your career or just.

any sort of software topic, you're more than welcome to do that. Links are all in the comments. ⁓ As far as what we've got going, this week we've got our finale. That's probably too grandiose a word. ⁓ For designing data intensive applications, this has been our first and only four-parter. Yeah, it's been intense. We're gonna do the author introduction, the book introduction, talk about what we...

thought of this last section of the book, but I think we should take a moment up top, Nathan, to appreciate we did it. We read Designing Data Intensive Applications.

Nathan Toups (01:56)

We did it.

We made it all the way through, and I'm excited to talk about it today.

Carter Morgan (02:04)

Yep, it was by far the most requested book of the podcast. And so we can say that we have fulfilled your wishes. ⁓ And we're ending the podcast now. That was the only purpose, was to work ourselves up to designing to add intensive applications. And this is going to become, ⁓ this is a Claude Bot podcast now. So every week we're just going to talk about Molt Bot and Molt Book. ⁓

Nathan Toups (02:25)

Yeah, you basically

were taking that LinkedIn strategy. We're just going to get real unhinged and be like, everything you know is wrong. Here's my 15 bullet point emoji list of, you know, everything I've learned in last three weeks using multbot.

Carter Morgan (02:28)

Yeah.

Leh.

my God.

It's Twitter is just like this constant back and forth of like AI will never replace software engineers. Again, it's so over. And I hate that phrase. this is your last chance to escape the permanent underclass. just like, I promise you whatever comes, that's not true. And yeah, we'll probably talk about some of that over the course of the episode. Who knows? ⁓ But let's, let's do this book justice. Let's, let's talk about designing data and intensive applications by Martin Kleppman.

the author introduction and the book introduction. For the author, have Martin Kleppman is an associate professor at the University of Cambridge, where he works in distributed systems and local first collaboration software. Before academia, he was in the trenches. He co-founded Reportive, which was acquired by LinkedIn in 2012, where he worked on large scale data infrastructure. He's also one of the few people behind, sorry, not one of the few, he's also one of the people behind AutoMerge, an open source library for building collaborative applications. And for the book and instruction, we have

Data is at the center of many challenges in system design today. Difficult issues need to be figured out, such as scalability, consistency, reliability, efficiency, and maintainability. In addition, we have an overwhelming variety of tools, including relational databases, NoSQL data stores, stream or batch processors, and message brokers. What are the right choices for your application? How do you make sense of all these buzzwords? In this practical and comprehensive guide, author Martin Kleppman helps you navigate this diverse landscape by examining the pros and cons of various technologies for processing and storing data. Software keeps changing, but the fundamental principles remain the same.

With this book, software engineers and architects will learn how to apply those ideas in practice and how to make full use of data in modern applications. So this is the final week. read chapters. Is it 12 and 13 or 11 and 12? I forget. It's 11 and 12. The final two chapters. Let me get the titles of those for you just so you got them.

Nathan Toups (04:22)

11 and 12, yeah.

Yes, so chapter

11 is stream processing. Remember, last week we left off on batch processing, which was the beginning of part three, stream processing. And then 12, I loved it. 12 is sort of, I think it's called the future of data systems. And he kind of, he gets to round up everything we talked about, pontificate about the future. And then also, he had a really great section on data privacy and...

Carter Morgan (04:36)

Yes.

That was great.

Yes.

Nathan Toups (04:57)

some other stuff in surveillance that I think is really worth talking about.

Carter Morgan (05:02)

And of course by the future of data systems, we mean the future as Martin Klempman saw it in 2016. But I think there is still a lot here that applies. But I really enjoyed the last section too. guess, yeah, Nathan, give me your overall thoughts. I know you just shared some, but if there's anything else you'd like to share.

Nathan Toups (05:19)

Yeah, yeah, so

I thought this book ended on a high note. I think it finishes off nice and clean. ⁓ We took everything that we learned and sort of, it gets us to this sort of thesis around streams and how streams are this sort of logical next step after batch processing. Brings up a lot of the interesting distributed database and data system problems that we kind of visited over the course of the book.

And I actually think even though this was, and then of course ends it out on yeah, a glimpse to the future from 2017 or 2016. And I think that his views on things have not changed. There's really some, I shouldn't say his views haven't changed. His views on the future are still very applicable now, almost 10 years later. And actually with the rise of,

AI and agentic AI, think the data privacy and surveillance section is probably more important than ever to think about. And I'm excited to dig in and talk about the rest of it. What about you?

Carter Morgan (06:23)

Yeah, I'm glad. I'm glad we read it. We did it. think I found this book ⁓ to be, I've been upfront about this, that this book, I just remember preparing for big tech interviews and I was like, I should buy and read this book because this is like the book everyone talks about, right? And then I ultimately didn't do it because I think like most people, I probably got a chapter in and then quit reading and.

That's why we have this podcast because it forces us to keep reading some of these longer books. ⁓ but it's, ⁓ it's not incredibly applicable to your like day to day job. This is much more like, I guess if your day to day job was like literally, you know, designing and building Kafka, then this would be helpful for you. ⁓ so this isn't so much an implementation guide as much as like discussing the underlying theory, which is really, really interesting. And I really enjoyed it because

We are exploring more and more of an event-driven model at work for some of the parts of our system. And this book is all about event-driven stuff. ⁓ Sorry, this last chapter, stream processing, talks a lot about kind of event-driven architecture and how that all fits in with the streaming model. So I really, really enjoyed it. ⁓ But it's all at a very, very deep level. And yeah, I think you just got to know what you're getting into as you read this book.

Nathan Toups (07:46)

Yeah,

I will say if you're in a career path that's in data engineering or platform engineering, ⁓ cloud architecture, sort of these areas, ⁓ I do think that typically those of us working in those areas try to abstract this stuff away so that the carters of the world can just focus on solving business problems, right? So if I'm doing my job well, you really shouldn't have to think about these sort of like...

Carter Morgan (08:03)

Right.

Nathan Toups (08:13)

weird things all day, really hopefully what you're thinking about is, I have this great library. ⁓ If I use data in this pattern, I can lean on the system that my SRE and platform engineering team helped us build. And I know that this transaction is going to go through with whatever certainty level that we have in place. ⁓ And so I think that that's where sufficiently advanced technology is indiscernible from magic, the Heinlein quote.

Carter Morgan (08:40)

Riot room.

Nathan Toups (08:41)

And I think that's true for good platforms. Good data intensive infrastructure has abstracted away ⁓ this cognitive load for users day to day. And so I do think that this book, if you're getting into the sort of juicy nitty gritty details, you want a good, and I've talked about this before, right, with the NP complete stuff. A lot of the graduate algorithms classes to teach you the class of problems you shouldn't try to solve. ⁓ This book is a great reminder of, hey,

These are juicy, hard problems. I probably shouldn't be trying to figure them out unless I'm a PhD researcher. What's the best in class tools that are available that meet the trade-offs that our team needs? And I think this book gives you vocabulary to have those conversations and go, okay, I've heard a lot about Kafka. This organization I'm interviewing with uses it heavily. I see because of their business model, why, right? You know, this book will give you enough education to not be completely ignorant. ⁓

Carter Morgan (09:36)

Right.

Nathan Toups (09:40)

of this trade-offs.

Carter Morgan (09:42)

Understanding those underlying patterns is really important because you talk about, know, ⁓ any sufficiently advanced technologies in this thing is full from magic. We have actually, this is only semi-related, but at work, we have been using GraphQL and we're exploring a pivot away from GraphQL to rest because there's a bit with GraphQL where like, it just works. You just write the query and you get back the information you need and like, it's all fantastic. But...

If you don't know what you're doing with GraphQL, it's really, really easy to write incredibly inefficient, incredibly non-performant GraphQL that just bogs the entire system down and makes everything so slow. so, you know, I think I say we, because I'm a member of the team and I take responsibility. All these decisions were made before I got here. But I think we were just kind of like, yeah, you know, let's use GraphQL and it's going to be great. ⁓

but not understanding how it operated kind of underneath the hood has really ⁓ been a troubling point for us. so, you know, do you need to know everything in this book to hook up an Amazon SNS that hooks into, you know, an SQS queue and just processes some events that flow through there? Probably not. But having an idea of all the different patterns in stream processing and kind of what's going on.

at a lower level, you know, it's never going to hurt you.

so I guess we can just get right into it with, ⁓ with stream processing, cause that's what chapter 11 is all about. It makes it distinct. It makes the distinction up top between, cause really we talked about batch processing last week, but batch processing, stream processing, and the future of data systems are all in the final section of the book. ⁓ batch processing, there's a bounded input and it processes a complete data set and it produces output. What's a good, like real life example of

Nathan Toups (11:48)

man, I'm so excited.

Carter Morgan (12:09)

batch processing. Like everyone always gives like the word count example. I'm trying to think, is there anything more applicable?

Nathan Toups (12:14)

So any

sort of data, like, here's a great example of a good batch process. I used to work at a fintech company. We would get a payload, and I'm not joking here, an FTP payload, because the fintech industry is just full of legacy stuff. And again, this is like six years ago. Hopefully we've gotten better API development. But we would get NASDAQ data and New York Stock Exchange data and a few other data vendors.

and they would deliver these daily payloads at night, two in the morning, three in the morning. We would have a pipeline that would have to run end to end and have the data available for internal systems by eight in the morning. So there was this very like cyclic thing. And so a bounded batch process was perfect for this, right? We didn't need some Kafka stream processor because the data wasn't coming in as a stream. We weren't processing live analytics data or something like this. ⁓

Carter Morgan (12:55)

Mm-hmm. Right.

Nathan Toups (13:10)

And batch processing is very easy to reason about. We just talked about in the last episode the UNIX philosophy. You can really think of this as pipes where you have this sort of immutable input data set, and that's how we treated it. We didn't want to modify NASDAQ data that came in, but we would want to do some enrichment and some normalization and all the stuff that you do in a data science team, where we are extracting the stuff that we want, we're loading it into systems, we're doing these other pieces. ⁓

And batch was really easy to reason about. If something broke, I could look at what step in my batch process that it broke in, right? And do all these things. ⁓ Of course, data sets have gotten so large, and the types of workloads have gotten so large that we now have unbounded inputs, right? ⁓ For instance, exactly. So, and you can imagine if we're trying to talk about financial markets, what about financial markets that never rest? What if we have

Carter Morgan (13:57)

Mm hmm. Which takes us to stream processing. Right.

Right.

Nathan Toups (14:07)

you know, what if we were having a data process that you can imagine we're getting real time data from the markets as they open and close around the world, right? Now batch processing doesn't make as much sense. Maybe I could batch process each market as it opened and closed. But maybe I'm doing something in the the brave new world like in crypto. Crypto doesn't have a sleep time for trading desks. You're having 24 hour feeds 24 7. A lot of them in that.

endorsing it, but know, things like Polymarket and other trading, ⁓ sort of other platforms that kind of try to trade all kinds of stuff. You have a continuous set of work that you want to do all the time and I guess for internal business processes, this might be an analytics pipeline, right? ⁓ You know, imagine ⁓ you're looking at behavior analytics of chat GPTs, chat interface 24 seven. ⁓

Carter Morgan (15:04)

Right.

Nathan Toups (15:05)

You have tremendous amounts of data coming in. I would imagine that their data science team would like to study behavioral analytics that come out of it. ⁓ That's an unbounded input. There's never-ending supply of the millions and millions of active users that are pulling in events.

Carter Morgan (15:24)

Yeah. ⁓

There's a lot of different ways you can kind of go about stream processing. ⁓ I guess the core of stream processing is this idea of an event, right? Which is that an event can be almost anything, right? Like an event is just a record that something has happened. It's an immutable record, right? So whereas like in the database, like a transaction, like for example, let's say an...

Like, let's say you are trying to book, like, I don't know. We do a lot of one-on-one booking sessions that might work, right? And so, you know, to book a session one-on-one with someone else, like you're scheduling a lunch, right? That session gets entered into the database, and that is a record in the database. But then let's say you want to update the time that lunch is happening. Well, you'd go in and you would modify the timestamp on the event, you know, the happening at timestamp. ⁓

And then boom, you're done. Events aren't like that. You'll never go back and modify an event and say, actually, I meant for this event to mean these things. Events are just going to be kind of constantly self, they're going to constantly emitting. ⁓ These are, I think, easiest to understand, like when you talked about Nathan in the sense of like user analytics, right? Which is like, I click on this, I view this picture, I buy this thing, right? And then all of that stuff gets emitted and kind of consumed by some sort of downstream analysis pipeline. ⁓

But ⁓ it can also happen for ⁓ social media notifications. This is how all kind of social media, or at least a lot of social media feeds work like this, where it's like when you log on and want to, you're expecting to have your timeline filled with 35 tweets. Twitter is not going and querying all the tweets that were made in the past 24 hours and saying, okay, which ones would be good for you? ⁓

As those tweets are created, they're being emitted as events and those are all being, those are being consumed by downstream consumers. And kind of as they come in, there are workers downstream that handle, okay, would this tweet be a good candidate for this user? And if it is a good candidate for that user, it kind of sticks them in your proverbial inbox and then uses those as candidates to populate your feed. ⁓ So I mean, that's kind like a high level overview of, ⁓

event processing and how event processing can work ⁓ with this streaming model.

Nathan Toups (17:54)

Yeah,

exactly. ⁓ so as soon as we start talking about events, we inevitably start talking about messages, right? And so the idea of messages, I love the way that they set this up, because especially any of us who've been working through tech for a little bit have kind of experienced some of this. Again, everything's a trade-off. We all have this really strong framework of thinking about the trade-offs in systems at this point. We're in chapter 11.

Carter Morgan (18:03)

Right.

Nathan Toups (18:23)

really thinking about stuff. And I love that, you he talks about like, well, what type of system do you have? Do you have a voiceover IP phone system? well, that's probably using UDP. And, you know, if you drop some packets, you really just want it to skip or interpolate or try to figure something else out and just move on. There's no point in replaying a part of your conversation from five seconds ago. You just will just drop that part of the conversation when things come back.

Carter Morgan (18:33)

Mm hmm. Right.

Nathan Toups (18:52)

we're back, right? ⁓ But he also brings us into things like zero MQ, web hooks. These are what he would call direct messages. These are messages that kind of go through a system. But as soon as you want to start having sophistication, like guaranteeing that something's handled in your system, this is where you get into systems that you've probably heard of like Rabbit MQ, Active MQ, what we would also call message brokers. ⁓ And these, again, they have some sort of durability guarantee.

⁓ You might even re-deliver. think when we were talking about earlier, where you're like, just SNS and SQS, right? Which are two AWS services for the uninitiated. ⁓ But simple queuing service, it can be set up where it just drops messages after a certain time, or you can set it up the much more ⁓ dignified way using a dead letter queue, right? Or if I fail to deliver or fail to process a message for some reason, I can pull it out in my mail sorter.

Carter Morgan (19:38)

Right.

Nathan Toups (19:50)

and shove it into a queue that just sits there to be analyzed later, right? There's all these like kind of flow control pieces that you can have. ⁓ Where I think, and I think this is where things get exciting for this chapter, because there's again, there's a good nod to these. What I loved and I think what people love about this and people have talked about this book, DDIA, really kind of introducing the ideas of Kafka to a lot of people. And these are the ideas are called log based message brokers. ⁓

Carter Morgan (20:15)

Mm-hmm.

Nathan Toups (20:19)

I would say if there's any excitement in the industry about stream processing and about these things, log-based message brokers are just like a complete shift in your thinking. ⁓ And it's one of those moments where you're like, when you realize how cool Kafka is or in what it represents, you're like, ⁓ man, there's just whole categories of things I couldn't do before ⁓ because of this. Have you had much experience with log-based message brokers?

Carter Morgan (20:45)

Well, so no, I

mean, I've we've used Kafka in previous jobs, but I've never touched it too much. mean, I guess I hadn't realized that there's like, there's a big difference between like a log based, like a Kafka approach as opposed to like just SQS or something like that.

Nathan Toups (21:04)

Wow, it's amazing. I'm gonna get excited about this for a second. So log-based message brokers, and especially there's this whole thing called event sourcing that kind of has come up out of this. It's not directly related, but highly, highly related. These are append-only logs, meaning that you use this system and it has these nice partition pieces and we can talk about those. And again, it's not just Kafka.

Carter Morgan (21:06)

Okay, let's do it.

Yeah.

Nathan Toups (21:33)

⁓ Amazon has a proprietary service called Amazon Kinesis. ⁓ I think Twitter has some tech that they wrote a few years ago called like distributed log or something like this. the idea is you just announce an event on this event stream, right? So you say that something happened. It gets written to this append-only log. And then you have these message ⁓ consumers that will follow certain topics or certain shards and they keep track of how much they've processed. So it's this sort of...

other end of it. But the cool thing is I can come up with a new experiment or a new service idea or these other things and rewind time and go back to, you know, all the way back to let's say that our Kafka, ⁓ our Kafka goes seven days in history or something. I can go back and run, make a new consumer that starts with seven days ago and processes all the events in the event log. Yeah. And so

Carter Morgan (22:27)

Very cool.

Nathan Toups (22:29)

not only can I experiment with new ideas without having to talk to my DevOps team, ⁓ I can also fix bugs. Let's say we realized yesterday our consumer had a bug in it and we've actually processed the data incorrectly. ⁓ I could actually create a new thing that ⁓ goes back and goes one day back in the history, reprocesses these things with whatever fix and amends this data. So it's just a super powerful pattern if your data is structured this way. ⁓

Carter Morgan (22:42)

Right.

Nathan Toups (22:59)

so that you can process data in this sort of like, oh, well, if you have all the certainty around it, have this pin-only log, I know that there's these like strictly ordered set of events per shard, meaning, and there's all these like things around it. He gets into a decent amount of it in the book that there's like guaranteed message ordering per partition. And partitions can be this whole other thing you can talk about, but for instance, you can do things like, you could, let's say you have, you know,

50 partitions, let's just pretend, pick a number. You can do a ⁓ balancing across these partitions by let's say user ID. And so maybe we always need to have any event tied to a user ID in strict order. always want all user ID events to never go. So what I would do is I would just hash the user ID so that it always gets put in the same partition. It's a consistent partitioning scheme. ⁓

And so it gets balanced amongst the 50 nodes. And so everything related to that one user maybe sits on some set of the partition table, knowing that anything related to that one user ID will strictly be processed in order of when it saw it. So you get this sort of deterministic replayability. ⁓ Again, in some systems you need. In some systems it's really, really important that if I go back and hook up a new consumer,

And I want to also see what happened in the world. ⁓ I can rely on this append-only log, right?

Carter Morgan (24:31)

Right, right. So I guess as opposed to something like SQS, I want this processed. It's done. The event functionally no longer exists. Right. That's interesting. Now I gotta think about that. I was just reading on Twitter, someone's like, Kafka's either the backbone of your business or meme software for tech interviews.

Nathan Toups (24:46)

Well, yeah.

No,

it's kind of crazy. There's also some interesting companies that I've seen have used Kafka in some pretty amazing ways. I remember one of the first ones that I really paid attention was ⁓ there's ⁓ an observability platform called CoreLogix. We are not sponsored by CoreLogix. I'm just a fan boy. ⁓ But CoreLogix processes observability data, and they do it much more cost-effectively than Datadog. And if you like the way that CoreLogix does stuff, it's really efficient.

One of the reasons that they're able to do this is that everything on their platform is Kafka Streams. So they're able to have this reproducible hot window of doing observability stuff. And there's a bunch of other secret sauce, but that's their big thing is that they have this like a pin only log of all the stuff that happens and place markers of things. And that they can, you can kind of like rewind and fast forward in time and do a bunch of really neat stuff. ⁓

Carter Morgan (25:28)

Interesting.

Nathan Toups (25:51)

I think another thing that this book brings up really well, and I think actually is probably one of the most important thesis ideas is what's called CDC, not the Center for Disease Control. This has changed data capture. ⁓ And this is really cool, right? This is one of those things where you inevitably start thinking about this stuff and you're like, wouldn't it be cool if any time I did something on a database, I could stream that this thing happened.

Carter Morgan (26:00)

Yeah.

Yeah.

Nathan Toups (26:20)

⁓ in some a pin only log so that other things could like pick up on this stuff happening, right? ⁓ Turns out this is like a really cool pattern. You don't have to use Kafka for this obviously. There's a bunch of really interesting technologies like ⁓ I think it is it Debezium? Debezium? I'm gonna mispronounce it. Roast us in the comments. there's a bunch of tools that I think Postgres does this out of the box. There's like a bunch of other things that they kind of outline, but you know.

Carter Morgan (26:27)

Yeah.

I don't know how it's pronounced. Yeah.

Nathan Toups (26:49)

Do I want, every time something happens, do I want to admit that a row was written or some data was mutated or something was there? And if I could just fit that into a pin-only log, ⁓ I can then go back and react to certain things happening in the database without having to burden the database with a bunch of SQL queries pulling on something, right?

Carter Morgan (26:55)

Right, right.

Right.

We're, ⁓ we're exploring this a bit at work. are exploring a migration of our data from MongoDB to Postgres. We're just, the more we look at the shape of our data, more we realize that it's just kind of fundamentally relational and Mongo is not a great fit for it. And it's causing all of this. I think a lot of our problems could be solved by just like, if we were Mongo whizzes and understood our system better and understood exactly how to, like, I don't think Mongo is fundamentally the limit here, but I think there is something to be said for like,

It's tough to bring on an engineer and you tell them to use Mongo and then say, but also don't use any of these foot guns, right? Like, and that's all our code base is. It's just foot gun after foot gun after foot gun that we were all writing until we realized like crap, this is how Mongo works. ⁓ and so we're just naturally thinking in a relational way. So we're like, you know what, maybe it makes sense to do Postgres. I'm like, my gosh, a Dynamigration. That's going to be like a giant nightmare, ⁓ just to get that all going. And granted all migrations are a giant nightmare, but I was pleasantly surprised that

We use the hosted version of MongoDB, Mongo Atlas, and that has change data capture support built into it. And it's got a built-in plugin with this AWS tool called, think, DMS. It's like data migration service. And yeah, it works like in a CDC way. so I just rigged up, you write like this little mapping file and yeah, I gave it one of our collections in Mongo and it formatted all of it. First it like batch processed the whole thing, right? And yeah, it just formats it.

translate it into Postgres, put it all on a new table there. And then as new events come in, whether that's new records being created or records being modified, yeah, it replicates to our ⁓ RDS Postgres table in like half a second. So I was really, really impressed by this. ⁓ And again, it's just a proof of concept. And I'm sure if we move forward with this and do it in production, we're gonna see all the reasons why this is a very bad idea. If you know why it's a bad idea, tell me in the comments.

But I thought it was a cool application of CDC.

Nathan Toups (29:14)

Yeah, and

this gets into another really cool topic, one that actually came up a lot when I was in that blockchain company. There's this idea of event sourcing. event sourcing is a neat idea. ⁓ And the idea is that in an event sourcing system, you have a store of events, which are again, these are like facts about the things that happened. It's not just the current state. And so the idea is if you had your

Carter Morgan (29:23)

Okay.

Nathan Toups (29:41)

event sourcing data all the way back to epoch or whenever you start doing this thing, if I replay it in order, I should be able to reproduce the current state, right? the very like, it sounds like I'm like talking in circles, but the idea is I can actually take a snapshot and then replay the set of events and reproduce the data in its current state as it's represented in some, you know, source of truth data structure. Well, it turns out blockchains are this, right?

If I replay the state transitions that are expressed in every block in a chain, all of us can prove that the declared state of the ledger is the way it is. ⁓ Turns out that's much, it's very useful way outside of blockchain stuff. ⁓ he talks about this like calculus analogy that I thought was really kind of cool, which is that basically like ⁓ states are, you integrate, ⁓

events over time, the state. So like if I take all of these events, I can derive state from this at some point in time. ⁓ But that if I take the derivative of state, I get my events back out of the other side, right? So ⁓ there's this kind of interesting idea here that if I structure my events the correct way, it gives me this like arrow of time state transition that I can.

go back, replay as long as I'm using the same logic to do that. ⁓ yeah, this is where ⁓ you start playing with these ideas of immutable events. You start playing with this idea that ⁓ it's almost like an accounting ledger where once these things have been declared, you don't change them. What you do is if you need to reinterpret or verify that some set of things happen, you have this log of all these changes. And so, ⁓

I don't know, it's one of those things where you obviously, and this is where the diagrams in the book I will say really help. ⁓ So even if you're listening to audiobook, I would highly recommend going back and looking at some of this stuff. ⁓ You need to kind of spend some time with these thoughts and think about it. ⁓ Event sourcing is this like really powerful idea. ⁓ Some people I think have understandable, you have like criticism that this can get a little too pedantic in ivory tower. ⁓ But there are times in which

Carter Morgan (31:45)

Bye-bye.

Nathan Toups (32:06)

it's really, really important that you don't modify things. For instance, I think he talks about it here. ⁓ One of my favorite non-blockchain uses of immutable logs, verifiably provable immutable logs is ⁓ the certificate transparency project. Are you familiar with this? ⁓ so this is like my little soapbox pitch. After the Edward Snowden leaks happened.

Carter Morgan (32:24)

I am not familiar with that now.

you

Nathan Toups (32:32)

we realized that there are some trust model issues where some of the certificate authorities were actually issuing ⁓ valid, quote unquote, certificates for state actors. And in this case, it looked like we were trying to catch bad actors in like Iran. ⁓ so like legitimate Facebook and other certificates were issued targeting folks in Iran where there was a man in the middle attack. And it literally gave you the green padlock so that the browser accepted it.

⁓ Yeah, so the tech companies come together and they're like, man, ⁓ nobody asked us if they could do that. This isn't cool. And what if this happened through other state actors in other countries, right? ⁓ And so they came up with this idea of the certificate transparency project, which was this idea that every time I, as a legitimate issuer, issue a new certificate, I would ⁓ write the pub key to the certificate transparency database.

Carter Morgan (33:02)

Interesting.

Nathan Toups (33:31)

and it's an append-only log that actually to this day, anyone can go, you can go download the entire certificate transparency projects log and go audit it. It exists right now. And basically a bunch of the tech companies, Amazon does this, anything that you actually, if you issued a certificate off of the, for like your elastic load balancer, you actually are participating in the certificate transparency project. If you've used Let's Encrypt,

Carter Morgan (33:56)

interesting.

Nathan Toups (33:59)

you're using the certificate transparency project. It's completely invisible to most people, but it's to identify, and it makes it way harder for a state actor to get away with this. Some browsers even verify the certificates versus certificate transparency. And so what's neat about this is that, you don't have to trust each other. All you're doing is attesting that this certificate was issued by the certificate authority on this date. ⁓ And then you can go back and audit the

Carter Morgan (34:25)

Mm-hmm.

Nathan Toups (34:28)

the append-only log and go, yeah, I didn't actually do that. Or, hmm, something's broken in the chain of trust. And so there's some really kind of cool public good stuff that comes from this that, and again, doesn't require Bitcoin and proof of work and stuff.

Carter Morgan (34:43)

Right. There was another concept I want to talk about. I don't remember exactly what it was called, but it's this idea that like, okay, so you have lots and lots of events coming in, right? Way too many events, but you might have some people that are interested in just a subsection of the events. I think the example he gave might've been like apartments going up for sale in New York City, right? Like on a Zillow type application. so I've set my filters and it's like, okay, I want to know which...

If an apartment with two bedrooms that's over 1,000 square feet comes on sale, it's under this price. ⁓ So basically, rather than doing ⁓ a query across all of those different events, and apartments is kind of like a funny example because theoretically, there's not that many apartments in the grand scheme of data. But when I was at a previous company, we were doing more user analytics stuff, which had tons and tons of data coming in.

Nathan Toups (35:36)

Mm-hmm.

Carter Morgan (35:39)

But rather than just trying to query that and being like, okay, like every four hours, I guess, we'll just query all the data in the world and see what comes back. ⁓ The idea is that as the events come in, they are run past the queries as they come in. And so an event will come in and it'll kind of, might have all these different users who have set all their different kind of preferences. And then an event will come in and be like, okay, do I match this preference? Do I match this preference? Do I match this preference? And then if it does, it gets sent into a, you know.

or duplicated into that user's, I don't know, what's the proper term for that? some sort of storage system. Like I keep using the term inbox, but I don't know if that's correct. ⁓ An inbox, Yeah, yeah. But I thought that was really neat because I think when you think of like querying data, like everyone always thinks like the data arrives and then when I want to do something with it, I query it, right? And this is a pattern that had to be developed because like

Nathan Toups (36:19)

Some systems call it that, like mailbox or inbox, yeah, exactly.

Carter Morgan (36:37)

eventually you start working with quantities of data that are so large that you can't do this. And so ⁓ it's one of those concepts that when you hear it just makes so much sense. like, of course. Of course, as data comes in, that's how you would handle it. ⁓ But I think it's very unintuitive from how we typically think about data.

Nathan Toups (36:57)

If you want to look at a really good ⁓ open source implementation of this pattern, actually if you look at Activity Pub, this is what powers things like ⁓ Mastodon and some of these other sort of alternative to the sort of decentralized social media, they have a really strong inbox pattern, right? Like how do you build a decentralized messaging system so you can see who's followed and done things and ⁓ it's a really powerful, I mean it can be used for much more than social media.

but ⁓ it's a really kind cool way of showing and implementation of something like this. ⁓ it's, there's so much cool stuff. I will say that like 11 and 12 was a pretty casual read. I think it wasn't as many pages as we've had to read. Like some weeks was a lot. ⁓ And so we actually got to spend, spending a little extra time with streams was I think useful just because it's probably, you have to read the other chapters to get the most out of this chapter.

Carter Morgan (37:33)

What I'm trying to-

Right.

Nathan Toups (37:55)

But chapter 11 is probably the most important technical chapter in the book. I'll put it that way, right? Stream processing is probably the most valuable set of understandings that come into this. ⁓ I would say up there with transactions. So like really should understand transactions. Transactions are really, really important. But also stream processing ⁓ and the patterns around this is, again, even though this is nine-year-old book,

Carter Morgan (38:01)

Yes.

Nathan Toups (38:23)

people are still learning this stuff for the first time, you know, this year. ⁓ It's, yeah.

Carter Morgan (38:28)

Absolutely. Yeah,

and I'm with you. It's a when you think about like, okay, what technology is going to run into my career? Like we've had a whole section last week where we talked about like how time is an illusion and your clock is lying to you. And here's how to make sure the clocks all stay synchronized so that, you know, you can agree on the source of truth for a particular transaction or whatever. ⁓ And listen, you're probably never going to deal with that in your career. If you are dealing with that in your career, I hope you are being paid quite well. ⁓

But transactions and stream processing, those are two concepts that are gonna show up in your career. Transactions certainly, ⁓ but stream processing just in general. ⁓ There are a lot of companies, I'd be surprised, any company of any sort of decent size probably has some form of stream processing going on somewhere in the company, right?

Nathan Toups (39:19)

yeah, unquestionably. Well, and

the funny thing too is that actually, even when the MapReduce paper came out, right, that's the seminal work that really did all this ⁓ batch processing MapReduce work, Google kind of like low key through shade on it later and was like, yeah, we moved on from that almost immediately to like the stream processing patterns. And ⁓ so I think the world's still been catching up. Google does this weird thing where they're like in an alternate universe.

Carter Morgan (39:38)

You

Yeah.

Nathan Toups (39:49)

slightly in the future, right? Like they've just, know, they were using containers before it was Docker. They were using LXC. They were over-provisioning. Like a lot of these patterns that have become like kind of normal everyday things or that lots of companies have kind of, a lot of them take for granted now. And we're seeing this, right? A lot of this is sort of like the after burner, the after glow of some juicy problem Facebook had to solve.

Carter Morgan (39:49)

Right, Yeah.

Nathan Toups (40:17)

or LinkedIn had to solve, or Google had to solve. And then they turn it into an open source project. A lot of times, this is recruiting tool, I'll be honest, because they're like, well, if we can get the community to learn how to use this pattern, and then we can hire them because we're maintaining some really juicy version of this. This is probably why Kubernetes was open sourced.

Carter Morgan (40:17)

Right.

Right, right.

I also

know Google can be weird about that sometimes where it's like, they have like a lot of proprietary in-house technology, which you work on as a software engineer. And like, like that's any big tech company. Like I think like Facebook has its own like version control, right? Yeah.

Nathan Toups (40:51)

Yeah, they do. ⁓ So

Google does too. And then I knew a couple of folks who worked at Meta, know, slash Facebook. And one of the crazy things, and again, I'm gonna butcher this, but I know I've heard some stories about it. Apparently everything, and I'm not joking, like literally everything, infrastructure, ⁓ systems and everything is like SQL queryable. Like they basically have this like master SQL.

Carter Morgan (41:17)

wow.

Nathan Toups (41:19)

query layer that can interact with anything in their system. And it was just like the weirdest custom implementation thing that doesn't exist anywhere else. And Google's same thing. I remember I think they built Gmail before any of the modern JavaScript frameworks existed. And so they have all these weird own version of reactive type frameworks. ⁓ Back in the day when they called it Ajax,

Carter Morgan (41:20)

you

Right.

Nathan Toups (41:49)

where they built these JavaScript libraries that literally only exist within Google and do things slightly different than everybody else. And now it's this thing that they have to maintain, right? Just a different world.

Carter Morgan (41:56)

Interesting. But I also

know like Google is also bad about like building on top of Google cloud, like building their internal services with Google cloud. I've heard they're getting better, but like, for example, Amazon is actually really good at that. even like a lot of AWS is built on top of AWS, right? And I mean, so much of amazon.com is built on top of AWS, right? And so I don't have anything to say about that, but I think it's,

I think it speaks to the offerings of the two products. I've used Google Cloud, I've used AWS. I am an AWS guy. I'm actually quite a fan of the product for all of its warts and blemishes. But I think you can tell it's a little more battle tested than some of the Google Cloud stuff. now roast me in the comments if you like Google Cloud. I will say the Google Cloud project structure, the way they like manage projects, like is so much easier than AWS, like and everything having a different account.

Nathan Toups (42:48)

Yeah.

AWS has gotten a lot better with like multi account org, but you can still tell it was kind of like tacked on later. ⁓ But I became a big fan. I was an early adopter of the multi account org structure. I've been using it in productions for probably since since twenty twenty or twenty twenty one. ⁓ But for the longest time, they were playing catch up as far as the fact that

Carter Morgan (42:55)

Right.

I know right there I go. I need this.

Thanks.

Nathan Toups (43:18)

Google projects made it so much easier to kind of like, what they call reducing the blast radius and these other things. ⁓ speaking of reducing blast radius, the last little kind of round out, think that it's we're kind of breezing over, reasoning about time and ⁓ maybe stream joins and some other kind of cool stuff. You should read it in yes. But one of the notable things, the last kind of section of this in chapter 11 was fault tolerance.

Carter Morgan (43:24)

Goodbye.

Time is an illusion. Just remember that. As long as you remember that, you'll be fine.

yeah.

Nathan Toups (43:48)

really thought about what guarantees and stuff have to be in place. And he goes over the differences between things like Spark and Flink. And of course, these are the sort of main actors at the time. There's newer technologies that are out now. how do you, so streams never finish, but interruptions can happen. And you have to, there's these trade-offs.

you have to have things like exactly what semantics you have to think about, like what is your batch size, latency versus overhead. You have to think about like, okay, so you have some failure state in your stream processor. What are you doing in the system to make sure that something that's failing actually gets processed and then the stream continues to move on? ⁓ I thought this was like a really, again, a more interesting thought experiments of what do you do?

What do you do when you have these failure states? ⁓ And there's no correct answer. It's like he talks about the different strategies, but I just never sat and thought about ⁓ what did it actually mean to kind of recover from these like that.

Carter Morgan (45:01)

Yeah, I've been thinking about this a lot just because like

where the system we have at work is running into some kind of fundamental limitations. And we're just at a crossroads. We're a small company, and so we've only got 10 engineers, and we're a series A and hoping to raise a series B sometime in the near to mid future. And we're just looking at the kind of current system as it is. And it's a little like, you know what? If we wanted to, we could make this work. And there's a lot of kind of tuning around the edges that would

get us a lot further than we are right now. But we're also looking at, it's one of those things like you don't, I kind of hate like, we can't do X because it doesn't scale, right? And then like for so many companies, it's like you don't actually ever need to scale. But we are looking at kind of our business model and what we want to be. And part of what we want to be is going to have to, like if we want this company to be what we want it to be.

then there's going to have to be some amount of scale involved. And so you kind of have to look and say, OK, we're at the point where we could make some changes and introduce some new components. I've been talking about event-driven architecture. And we think that would be really, really beneficial here. And we think that we might be kind of ahead of ourselves in a good way, getting ahead of the problem. But we're also trying not to be super starry-eyed about it. Like, and then when we do this new system, there's going to be no problems. We're going to have to fix everything. Of course, there's going be problems.

And so reading this kind of fault tolerance section, because yeah, you move it towards streaming and event processing. Yeah, there's a lot of new ways you can break the application. And those ways that you break it can be really hard to fix in a way that's not just like, just go in the database and rewrite the value if you have to. ⁓

Nathan Toups (46:52)

Yep.

So a couple of concepts here that I think, again, they're valuable whether you're doing stream processing or not. They're super valuable for REST. They're super valuable, honestly, for any distributed system. Itempotence, right? Itempotence is this idea that I can take some action, and if I take that action again, it gives me the same result. something that's inherently not itempotent is doing a n++.

counter increase. So every time I hit an endpoint and it adds one to the counter, that's not an idempotent action, right? But an idempotent action would be, I say, update username, and it has this update username event, and then if I rerun that same thing again, it should give me the same output. I can rerun this thing five times, and it will just ensure that

the name is updated to whatever the field is that I have in this item potent event. One of the things that I hadn't thought about with this is like, you know, there are some systems that have, especially like Kafka, that have this concept of exactly once semantics. ⁓ It was always interesting for me to think about what exactly once, I like the heat in the book that it's effectively once, it's actually like a better way of thinking about it, which is.

Carter Morgan (48:19)

Hmm.

Nathan Toups (48:20)

It's actually possible that an event gets processed multiple times, but the visible effect is once. And so the idea is that, look, we guarantee that this thing will be processed, ⁓ but that it actually requires that you either have an idempotent write or that you have a transactional output, meaning either one of these things I can guarantee that this thing was only going to change the state of the system one time. You can think of it that way.

Carter Morgan (48:26)

yeah.

Nathan Toups (48:50)

which I thought was a much more useful, like again, he's so good with like giving more exactly once kind of feels like a marketing term where effectively once is like the much better framing for somebody who's building a system like this.

Carter Morgan (49:03)

Right, right. That makes a lot of sense. Well, I think we can talk now about the future. Let's all transport ourselves back to 2016 and think about the future. And granted, again, there's a lot here that still applies. ⁓ Yeah, Chapter 12 is really, really interesting. mean, there's kind of like two sections to it, like where he talks about like, OK, here's what I think. Here's how I think data systems might evolve in the future.

Nathan Toups (49:09)

yeah, the future as it... Yeah.

Carter Morgan (49:31)

And this is where I wish I were more familiar with the current data systems landscape, because it'll kind of talk about these neat ideas. Like, oh, that sounds neat. I wonder, does anyone do that now? This might be a solved problem at this point. And then there's also a kind of final section about ethics and his concern about a system or a world that is increasingly gravitating towards maintaining and collecting large amounts of data from its user and what our responsibilities are and.

and what the risks and dangers of this are. So I guess we can talk a bit about that first half, like, OK, what does the future look like here and what might data systems do? I don't know, Nathan. Was there anything here that really stood out to you that you thought was neat or anything that you listened to and recognized like, we're actually doing more of that these days?

Nathan Toups (50:19)

Yeah, you know, he kind of talks about like, he, one of the reasons that I was, you know, we kind of spent a lot of time on the log based approach in chapter 11 is because he actually gives you little glimpse of what cool things you could do with the log based approach in these distributed systems in the future and sort of like the shortcomings that.

doing these data intensive things and understanding and reasoning about state and being able to do playback and proving that certain events happened in certain orders become a lot easier to reason about if you have these sort of like, some of the event sourcing and event playback and immutable log sort of patterns. he just gets to kind of explore these things. One of the things that stuck out to me was, I think it was called

Timeliness versus integrity section, ⁓ where you basically say, know, timeliness is that users see up-to-date versions of state versus integrity means guaranteeing that there's been no corruption to that and that there is a trade-off here. ⁓ that, you know, timeliness means that like, I, ⁓ timeliness with like an event-based system decouples these and so like,

it's asynchronous so it's not guaranteed that you're gonna have the timely view on the data. But that if you use things like exactly once and idempotence that you can also, you can guarantee the integrity even though you may not have the timeliness. And so there's this like, again, he kind of just explores like, what are these trade offs and like, what kind of systems would you wanna build with these? And I know, I found it very, again, it was very like mature way of thinking about these kinds of complex systems that we're building.

Carter Morgan (52:08)

Right, right. I really like the section about, where he talked about clients, stateful offline capable clients. And he kind talks about like when the web first came into being, like all your client can really do is like request data and then display that data, right? And really the most interaction you can have with web pages, like scrolling it. He says, he's talking about like single page applications.

uh, and react and saying like, clients are getting thicker now. And so there's the possibility that you could have like a stateful offline capable client where, you work offline using the local states and then either via web sockets or some sort of pulling mechanism, like you're sinking so that you're, pushing your version of the local state up and then also pulling the changes down. Um, which is, I think we are seeing a lot of this, like notion is a really funny example because notion

is like, like it's a document editor. Like that's neat. like, then for the longest time, they did not have an offline mode because Notion's whole thing is that like every document is a database and like, actually is getting an offline mode was a really complicated challenge to solve. But when you first hear about it, you're like, what do you mean? I can't edit my document without wifi. And so they're moving to something like this. But I also thought this was interesting because in some ways we actually are

pivoting away from this where Next.js and server-side rendering has gotten hugely popular too, right? To make clients more performant, which is the idea of like, don't have these huge thick clients that we need to send down to the user and then it makes it harder for SEO indexing and optimization, right? Because they have trouble exploring the single page applications. so, ⁓ yeah, I just thought that was interesting where like,

We're seeing some of what he talks about here, but in some ways we are kind of going even further back to like make clients much less offline capable and much less stateful and instead just pre-bake all this HTML for me and then send it down and make it quick.

Nathan Toups (54:19)

I

think a lot of this is like, is old is what would it all becomes new again, sort of patterns. Again, we understand that there's these trade offs and depending on the shape of the work that you're doing, it makes sense to do it server side or it makes sense to do it client side. Toot my own horn a little bit. We recently created a book of book over flow.io slash episodes page. And one of the things that we actually have a real Postgres database.

Carter Morgan (54:24)

Right, right.

Nathan Toups (54:45)

and it has all of our, it has highly relational data like the episodes, the books that we've read, who the hosts and guests were, a bunch of stuff that I got to play around with. Thing is, that.

Carter Morgan (54:56)

Nathan says, we,

just want everyone to know Nathan did this and I modified the DNS record when he was like, hey, our website is cool now. It's not a react template you bought three years ago. Can you modify the DNS record? I'm like, sure.

Nathan Toups (55:06)

But I've been using a lot

of Next.js and ⁓ it's been really kind of fun because everything was sort pre-rendered server side, it was really fast, but now I have a database that I'm querying, so I'm using React suspend, which does this thing where I can have a placeholder and yank in data. But then I also, our data doesn't change that often, and so while I want this ⁓ database to be the source of truth for our data, ⁓

there's actually these other kind cool caching layers where I'm using suspend, but as long as the data has been rendered in the next last 15 seconds, it's cached on the server. And I get to think through, honestly, all of these like designing, we're not data intensive, this is very small. But can I do a performance thing where, a performance thing where like within 15 seconds, the data is updated on the website, but that for most people, it should be hitting the cache, right? Most of the time,

Carter Morgan (55:52)

Yeah.

Nathan Toups (56:04)

our data doesn't change that much. But if I did change the data, would that update propagate pretty quickly? And it was actually, using this book in my mind and thinking about which layer should handle which thing made it a lot of fun to work on, ⁓ just thinking through these problems. And so even if you're doing little bitty things and you understand the technology that you're working on and you want to think about, you know, who owns what and...

How does caching work and how do we do integrity and what is the use case of the user at the end of the day? I like this pendulum swing that's happening, right? Because I started with, I didn't want to use a database. I just would do server-side caching. I'd make it as plain vanilla as possible. And now I want, I don't want to hammer our database with 100, we have 100 episodes now, yay for us. But I don't want to have a query that returns 100 rows every time.

So for instance, we paginate 15 rows at a time. But as you scroll down, it preempts that. And so there's like, I got to just goof around and hack on stuff where I'm actually preemptively loading the next 15 as we get close to your scroll being there and just magically shows up. So if you're actually scrolling, you get to see all 100 with no latency perceived. But in reality, if you're just looking at the page and abandon it, we hit 15 rows of data. And so thinking through these kind of problems and thinking about how I'm going to like,

query and serve and cache and these other things is, I don't know, this is like the joy of thinking about ⁓ building and presenting stuff like this.

Carter Morgan (57:41)

Right, right. Is that the time we talk about ethics?

Nathan Toups (57:46)

Yes, speaking of, have a really sober ⁓ thing to think about. ⁓

Carter Morgan (57:48)

We gotta... Yeah.

Nathan Toups (57:55)

Yeah, privacy and surveillance. think that is, ⁓ should we skip all the way there or is it, ⁓ actually bias and discrimination is another one I think, yeah.

Carter Morgan (58:02)

There's some interesting stuff, like, you know, it's like what? Yeah.

Yeah. Basically that like, ⁓ you know, and I think it depends on how you view this, right? There, there is a thought process that like, look, it's a cold hard machine. It's just analyzing, you know, like, ⁓ like crime data, right? For example, like, and I know there, there's been discussions about this where it's like, you know, it was controversial. Zillow was allowed to show you like if.

neighborhood has more crime because like just the way it works at least in the United States of America, which is where Nathan and I are both from, ⁓ neighborhoods with more, you know, Black, Brown, Hispanic people tend to have more crime. And so if Zillow shows you, hey, these neighborhoods have more crime, right, then that depresses home values in those neighborhoods, you know, less people want to buy them. And then, ⁓ you know, that obviously hurts people of those races who are, you know, maybe trying to make some, you know,

Obviously, they want their home values to go up because that's a significant source of pretty much every American's wealth who owns a home. There's an argument to be made that, hey, we're just exposing the data. We're just telling you the true state of the world. But there also is concern about bias, that it's affecting one group more than others. Yeah. ⁓

Nathan Toups (59:20)

Did

you take AI ethics at Georgia Tech? OK.

Carter Morgan (59:23)

I did not, think,

I think, wait, did I? No, no, I took digital marketing. There's kind of two classes known to be blow off classes, which, know, it's okay, it's a hard program. There's only like two of them. ⁓ You gotta be in graduate algorithms. You hang your hat on that, Nathan Tubes.

Nathan Toups (59:27)

Well, it was, okay.

Yep. I took both. That's what kind of slacker I am. I, so I will tell you.

No, so I will tell

you though, AI ethics was one of those classes where I took it because it was supposedly an easy class, but actually ended up having like an amazing ⁓ impact on me. So it was one of those that shows us how much we can trick ourselves with data. And I think that's what kind of ties in. The importance here is that in the book actually, and I think someone actually recommended one of the books for that class in our book discussion recommendations. ⁓ There is a...

Carter Morgan (59:55)

Interesting.

Nathan Toups (1:00:14)

there's this whole idea of garbage in, garbage out. If I'm collecting the data in some biased way and I train a model in that biased collection method, I'm going to reinforce the bias on the outside. It's a very simple thing. one of the examples they gave was ⁓ you wanted to understand ⁓ the prevalence of high school students smoking cigarettes. And so you decide you're going to go to the parking lot at the high school and go around and survey and see

how many people are smoking cigarettes. ⁓ And you realize that the number is pretty high. You look around and the people hanging out at the parking lot smoke cigarettes at a pretty high rate. But the thing is that that's not a proper representation of the behavior and activity of all high school students. You've picked a place that's like a hot spot for smoking cigarettes. And so therefore, you could come to a conclusion if you didn't scrutinize this and say, we have a cigarette smoking epidemic.

Carter Morgan (1:00:49)

Right, Rem?

Right, Ram.

Nathan Toups (1:01:12)

Like the number of students that we sampled is super, super high. And again, for someone who doesn't understand sampling and doesn't understand how to collect data, they could take this and be very alarmed and run and do crazy stuff with it. think that exactly, for instance, not factoring in race and socioeconomic factors would actually skew that data tremendously because we, you know, I'm from the South and there are affluent neighborhoods that have folks from

you know, every different creed and race. ⁓ But then there are also historically impoverished areas that are skewed one way or the other. And it's very complex, right? And it really would be ⁓ unfortunate to reduce something to some set of variables that are just misunderstood. And ⁓ so, I think this is he, the reason we're bringing all this up is that he actually brings up the idea of bias and discrimination.

Carter Morgan (1:01:44)

Right, right.

Nathan Toups (1:02:12)

in a really clean way, is that, ⁓ I think, it, ⁓ machine learning is like money laundering for bias, I think that was like the quote from the book. And it is this, it's like, it feels very scientific, like, the data tells us this thing, right? And we've seen this, all of us have experienced this. As soon as somebody has a worldview that you vehemently disagree with, but are backing it up with some study, your spidey sense goes off and you're like,

that can't be right. Like I know that you said that there's this study that did this and we've all found something that disagrees with our worldview. And of course, the tribalism of us is that we will reject that outright. We'll look for data that backs up my worldview. And then guess what? We're in a world with so many people in it. You can always find a study that's going to back up your worldview, like spoiler alert. And so you have to be really careful. Like my wife actually has a master's in ⁓ analytics from Georgia Tech.

Carter Morgan (1:02:43)

Goodbye.

Okay.

you

Nathan Toups (1:03:11)

she actually is, she's substantially better at reading these papers and looking at the rigor that they've put into measuring stuff, stuff that I'll miss all the time, right? And she'll be like, actually, I that sample size is like super small. Or, if you look at the standard deviation here, like it doesn't say anything, like this could be a random roll of the dice, right? Or something. And I'm like, yeah, that's actually interesting. And I think a lot of stuff isn't scrutinized properly, right? And so,

Carter Morgan (1:03:19)

Right.

Uh-huh.

Right.

Nathan Toups (1:03:41)

It starts with this thing of like, you can be making an honest mistake and reinforcing biases that are ⁓ really, really not helpful. ⁓ But then it gets, I don't know, a step further, which is like, well, who's responsible or accountable for these outcomes? I think we've seen this recently with AI tools, right? AI tools are these analysis machines on top of datasets. And we've seen this a lot. I mean, it's gonna get controversial, but like.

ICE has had a really bad track record with misidentifying people and yanking them off the streets because their name matched up or like a profile mismatched or they accidentally applied records. I think I heard a story recently where like there was a guy who was an actual US citizen but an immigrant who was like yanked off of a cruise ship because he was misidentified as some drug dealer. It was horrible. Like this guy was actually like a military vet and a whole bunch of like literally the worst.

Carter Morgan (1:04:15)

Crap.

Cheers.

Nathan Toups (1:04:39)

PR nightmare, but also like a hellish nightmare for this guy who's done nothing wrong. And you look at these kinds of things and you're like, okay, this is a high profile cut and dry case. How many folks maybe had like a few misdemeanors, but weren't actually felons, but because they got yanked up with their misdemeanors, they actually got, you know, all these other things going on. And it gets really fuzzy where you go, I can't.

Carter Morgan (1:04:46)

Right.

Nathan Toups (1:05:08)

We have a group of people acting with authority on data that obviously is flawed. Like there's a few examples of it, and maybe most of time it's right, but it's wrong enough that it's horrible, right? Absolutely horrible.

Carter Morgan (1:05:19)

Yeah, well, and I was thinking too about some of the, what I will politely characterize as overreaches of our federal government recently. ⁓ And he talks about like, yeah, like basically like the great kind of like bargain we have made in the 21st century is free services in exchange for our information, right? And I, on the whole,

Nathan Toups (1:05:43)

Yeah.

Carter Morgan (1:05:49)

with the current system think that this has been a good trade. The world as it stands today, I think we have gotten a lot of great services in exchange for what I consider to be something pretty, like I don't care if someone knows my search history, but that's today with the current actors of today. I don't think we live in some sort of dystopian hellscape, right? But he points out that like we have all willingly opted in to like,

carrying a tracking device with us everywhere we go. He says, you go back to like the 70s, the 60s, not even the most dystopian novels envisioned a future like that. We're all citizens could be, you know, tracked very easily. ⁓ And, you know, it's a little like the tools we've created. And even though I think in the aggregate, we are wielding these tools today somewhat responsibly. Again, we have not turned into a totalitarian state, right?

There's a lot of power out there. And he has this interesting rev- er, he compares it to the Industrial Revolution, where he says like, the Industrial Revolution, we invent all this new technology. And in a lot of ways, it's really great for humanity, right? And lifts a lot of people out of poverty and really moves us forward as a race. And pollution is just not even on the radar. Like no one's even thinking about that this could possibly be a concern. And then, surprise, it turns out, it's not great to just pump tons and tons of greenhouse gases into the air constantly, right? And so-

We've learned about that, and now we're of look back on our answers and go like, my gosh, how could you even do that? But we are now working to remedy the mistakes of the past and move towards cleaner energy sources. ⁓ He thinks that we are kind of in the industrial revolution of data right now, and that we have little concern for all the ways we're abusing the data and all of the

potentially powerful tools we've created and that future generations will look back on us and be like, oh my gosh, like they were just handing their data out willy-nilly. This is related, but not the exact same thing. But I kind of think that way with like, we are obviously living through kind of a period of great social upheaval in response to social media. And I've talked with friends like, you know, is it kind of all gonna, we're always gonna constantly be in an environment like this. And I remain somewhat optimistic in the sense that I'm like, there's,

I don't think there's going to be a revolution bigger in the world history than inventing the internet and putting it in everyone's pocket. Right? Like the, is a huge fundamental change from how the world operated in like 1989. Right? And so I think we're learning how to deal with that. And I'm optimistic we will continue to learn how to deal with that and come up with better patterns for us. But it's tough because you're also working in, you know, like I hope one day we'll all kind of regulate our screen time better and do less doom scrolling and

Nathan Toups (1:08:19)

All right.

Carter Morgan (1:08:39)

come to a healthier place as a society. But the social media companies have every incentive for that to not happen, right? I know. It's a lot. I think, I thought that was an interesting parallel that basically, like, we are living through another industrial revolution and just barely starting to grapple with some of the pollutants we're putting into the air. And hopefully, like the previous industrial revolution, we will learn how to deal with that.

Nathan Toups (1:08:46)

Right.

Yeah, no, and it does, it leaves in this like sober note and talks, kind of puts a challenge to you to say, you know, we should try to make our, not just our children proud, but like our grandchildren proud. Are we building a world in which people can opt out and people can actually understand what's gonna be done with your data? And we're always seeing this, right? Like I think it's even much worse than it was when ⁓ it's how much of our digital exhaust or

this stuff has been used to manipulate people's behaviors and their sentiments about things. think a big part of the argument of who controls TikTok is based on who gets to censor which data and why. And that should be concerning to anyone because again, it's not that in good times that we have these powers, it's in bad times the people who might abuse them.

Carter Morgan (1:09:47)

Right.

Nathan Toups (1:10:02)

are the ones that we should worry about.

is it, are we centralizing control in a way that if the wrong people, and wrong people defined by you, right, because we are a democracy, the wrong people defined by you, oh boy, yeah.

Carter Morgan (1:10:16)

Polish.

They get a hold of this. We're done.

Nathan Toups (1:10:22)

Yeah, so the best thing is like the is it's defined by you is that ⁓ I like to do this a lot with like playing Mad Libs, right? Play Mad Libs takes any topic in the news, switch the political parties, switch the leaders. Or would you still agree? Would you would you think that this was overreach? Or would you think that this was a good use or bad use of power was in the interest of the public or not interested in the public? And if the power itself makes you uncomfortable,

Carter Morgan (1:10:32)

Right.

Right, Yeah. ⁓

Nathan Toups (1:10:51)

when wielded by the party or the people that you don't like, it's probably power that the people you do like shouldn't have either, right? ⁓ And so it is, it's really, I really appreciated the book kind of rounds out on this, because we just spent all this time, the first 11 chapters, figuring out how to get more power and energy and extraction out of data at scale. And then he's like, by the way, we could do some really awful things with this if we're not careful.

Carter Morgan (1:10:59)

Right, right.

Hahaha!

Wow, it was fun. It was fun. I guess not the potential that we'll do really awful things. That's not fun. But we did it. We read Designing Data Intensive Applications. And if you are listening to this podcast, have listened, hopefully, to all four episodes of Designing Data Intensive Applications. So pat yourself on the back, too. ⁓ I guess we can talk a bit about some of our hot takes. ⁓

I think I kind of shared all of mine over the course of like the, guess I do have one hot take, is that with this, think most of our hot takes will be associated on this ethics section, right? Which is that like, I think you should strive to make the world the place you want to live in. And I have little patience for people who like work at Google and then profess to like hate Google. You actually have a lot of control over how you spend your time.

Right. And, you know, I just think like, if you think that like manufacturing weapons is bad, don't work at Lockheed Martin. Right. And like, I've just seen too many people do that and be like, well, I need a job. Don't I? I'm like, there are so many jobs out there. Right. Like I, I have had people reach out to me, you know, I'm religious myself and I actually had mind geek reach out to me, which is like, ⁓ they, they do like a porn hub.

Nathan Toups (1:12:29)

Right.

Carter Morgan (1:12:46)

They have like a bunch of pornography companies and I myself, I'm not, you know, a fan of pornography, right? I have some moral opposition to it. And ⁓ I just think like, I am sure that the engineers at these companies for the most part are doing very boring engineering tasks every single day, right? And you can probably go there and just work on those kinds of engineering tasks. And for the most part, it's fairly separated from the actual product you're pushing. Most of our jobs are like that, right?

Nathan Toups (1:12:49)

well.

Carter Morgan (1:13:14)

I still would not work at those companies because I think that is furthering a mission that I don't believe in. Right. And so I just, I see too much of this online with people who claim to like, hate, you know, hate some larger value, right? Like, I can't stand the internet surveillance. And then you look at it like they work for Facebook. I'm like, what are you talking about? There are so many companies you can work at. Don't be part of something that you think is evil and you have more agency than you think you have in this situation. So I don't know.

If you're like that, cut it out and apply your talents and your vigor to what you think will make the world a better place, or at the very least will not make the world a worse place. So I don't know, that's my take on that.

Nathan Toups (1:13:55)

Yeah, yeah, no,

I like that. I'm in a similar boat in that ⁓ I think that the ethics section of this book should be required reading. actually think that most computer science programs these days should actually have ⁓ ethics as part of the official curriculum, ⁓ just because we don't have a licensing board right now. And I think that there's probably a good reason for that.

Carter Morgan (1:14:20)

Right, right.

Nathan Toups (1:14:22)

But there are types of software engineering in which I kind of wish we did sometimes in that I wish that there was some sort of binding code of ethics in the same way that we don't want attorneys or doctors to misbehave in a way that was defined by their industry. I would love for this to be defined by software industry itself. ⁓ But I do think that there are some people in certain hot code paths using some certain technologies

that it would be nice if they had some sort of binding code of ethics that we could go, wow, you know what? ⁓ They're doing this, but they're also bound by that code of ethics. we know that ⁓ they're at least not trying to cause harm in X, and Z, right? ⁓

Carter Morgan (1:15:09)

of the rise of large language models makes the idea of a software engineering licensing board more or less likely. Cause on one hand, can kind of see like, yeah, it's the democratization of code. Anyone can produce code. On the other hand, I could see there being a lot of value in someone having a credential that says like, I know what I'm doing. you know, even though we are, the future is here and, and we are just at an age where large language models will probably generate most of the code written.

Nathan Toups (1:15:16)

Great.

Carter Morgan (1:15:37)

from now until the end of time. think the era of humans just churning out lots of code is over. But I wonder if there would be value in basically saying, I understand the code that this is outputting. ⁓ I don't know. It's just interesting because you see that in a lot of open source projects are saying, we are not going to let people open PRs anymore because you can just churn out so much code with large language models that we're getting all these garbage PRs. In the past, the

The natural gatekeeping for these open source projects was, did you take the time to write the code for the PR? And if you did, that's probably a sign you invested some sort of, you know, enough time into this. ⁓ and, we're just moving away from like that qualification of like, can you produce code? Doesn't signal nearly as much as it used to. So maybe we'll have to move on to other signals. What those are. I have no idea.

Nathan Toups (1:16:22)

Yeah.

I am a large language model user, specifically for certain types of projects, more heavily than others. I will say I love a good critique. I've been paying attention to Kelsey Hightower, who I have ton of respect for, has been giving these really great contrarian hot takes on why he likes to write code by hand for certain problems he's solving. And very principled approach. And again, I always appreciate a principled approach.

Carter Morgan (1:16:44)

Mm-hmm.

Nathan Toups (1:16:50)

Another person that I think is a shining beacon in this area with contrarian hot takes is Cory Doctorow. ⁓ He is very anti sort of foundation model consumer, you know, stuff. He has some really good critiques on like why the business model just doesn't make sense as a reason to give you pause. And of course he's involved with the EFF and whether you take everything he says face value or not.

or you think that maybe his outlooks are not exactly in alignment with yours, these are deep thinkers and they're really respected folks in our industry. And I highly recommend at least listening to their perspectives, because you're going to have a richer understanding of your worldview if you listen to folks who have critique amongst the echo chamber of the, know, all of our jobs are going away, large language models are the best thing ever audience as well.

Carter Morgan (1:17:43)

Right. Yeah, it's a, you know, I I'm inclined to believe that all the work we're doing here on Book Overflow, and if you're listening to Book Overflow, there's still value in understanding systems deeply and that the job of software engineer is not going away. Right. And I, you know, I know I'm wrestling with problems at work. I was wrestling with a problem for weeks. And I'm like, I wish large language models could just solve this for me because I'm going crazy trying to solve this. But

You know, it's a very, it's it's a turbulent time in the industry. Obviously there's lots of change right now. Um, but the cream always rises to the top. And I think if you put in the work and you work hard and, know, try your best to master this craft, like it's not going to go poorly for you. So, you know, a little note of encouragement here at the end of a, this, uh, seminal book, I guess we can say what we're to do differently in our career. Having read this book, what do you, do you got, Nathan?

Nathan Toups (1:18:39)

Yeah,

so I've used most of the Kafka that I've run into have been like mature deployments of Kafka where we're interacting with it. I've actually never set Kafka up from scratch or even used some of the like useful flow and I'm gonna spend some time with it and play with the patterns and tools and follow some blog posts. So I'm carving out some time to deal with setting this up from scratch

Carter Morgan (1:18:56)

Now, interesting.

Nathan Toups (1:19:09)

Maybe doing Kafka the hard way or something. Yeah.

Carter Morgan (1:19:13)

Nice, nice.

Yeah, I just want to implement anything event-driven. I'm with you. I've worked in event-driven architecture in the past, but never built it. This is kind of cheating because we're already planning on doing this at work. I have, you again, that whole fault tolerance section really made me think about, like, OK, how do we make sure that we do this in a sustainable way? As far as book recommendations, I'll go this one. Same as last week, sickos and maniacs, if you are just

absolutely chomping at the bit to learn more about this after having listened to this episode. Like do it because this is a fantastic book. But I think for most software engineers, ⁓ if you're just like, Hey, what's going to level me up in my career? The fastest, like I said it before, but like read fundamental software architecture, read a philosophy of software design, honestly read made to stick for communication. if you're a manager, read radical candor, the DevOps handbook. Like these are all the things that jump out.

to me is like, this is what I think would take like a junior to a mid-level or a mid-level to a senior the fastest. Does any of that have a sense of applications? Probably, I'd say the most textbook-like book we've ever read. So make of that what you will, but all books have their purpose and this book accomplishes its purpose perhaps better than any book we've read on the podcast. Nathan, what do you got?

Nathan Toups (1:20:33)

Yeah, I'm going full circle, right? Each week I've been progressively being like, wow, you have to really be committed, all these other things. I'm going full circle all the way back to my first week's recommendation, which is, this is a great book for software engineers who are deeply curious about systems architecture, who want to grow in their understanding of trade-offs. I said, as of part one, this is not a tutorial on how to build stuff. That is, as of all three parts, this is not a tutorial on how to build stuff. ⁓ It's a way to think.

deeply about distributed systems and how data flows. If your job has taken you into this sub-domain of work, right? And again, I would say that there's entire fruitful career paths in software engineering that doesn't make this your daily driver. But if you are interested in getting into software architecture or systems architecture, you're maybe DevOps and you want to get into platform engineering, ⁓ and you want to really kind of understand how data flows work, data engineering,

ML pipelines, these kind of things. I would say that ⁓ this book is absolutely essential. That yes, all the books that Carter recommended need to be there as a foundation. But at some point, you're going to want to, if you're in that position where you're like tuning the machinery ⁓ of how data flows through there, this becomes, this is a book where I regret that I didn't read it earlier in my career. I wish that, I wish this was a reread for book overflow and it wasn't. ⁓ So.

Carter Morgan (1:21:49)

Right. Right.

Right. And hopefully

we, I think we hope to reread this at some point with the second edition coming out. I'd love to do a more targeted reread and like, especially if it's just like, oh, just like chapters five, seven and 11 are radically different. And then we could just reread those, but we'll see. Yeah.

Nathan Toups (1:22:16)

Yeah, if

we're lucky, I mean, supposedly the second edition comes out in March. It's been pushed off a couple of times. I will say, I don't want to commit to it yet, but I think it would make sense for us to reroute our schedule a little bit, just because it would be timely and people are gonna be curious since the new book comes out. So yeah, if you'd like us to prioritize covering the differences, you know, we won't do a full reread. It's just way too many. We're gonna do an eight episodes.

Carter Morgan (1:22:21)

There we go.

Bye.

Yeah.

Nathan Toups (1:22:47)

in 2026. But if we can get a good breakdown of the diff, we'll read the diffs and then we'll do like a second edition update episode or a couple episodes maybe. ⁓ That sounds interesting to me. I'd want to know. So yeah.

Carter Morgan (1:23:01)

There we

go. Well, and we have ⁓ earned a much needed break. And so we are we're not going to take a break from producing an episode next week, but we are going to read an essay that is in part because we are doing ⁓ because we read Design that ⁓ Intensive Applications and it's a huge book, but also because I'm actually going out of town next week. I am taking what you're listening to this on Monday. And so Tuesday we are flying to Disneyland. I'm the kids to Disneyland, my wife and I. So that's going to be a lot of fun. ⁓

but they don't know about it yet. if you're listening to this, there's gonna be about a 36 hour window where you know the Morgan's are going to do that, but my kids don't. hey, relish that knowledge folks. Anyhow, but what are we reading? On Trusting Trust. Oh, sorry, you wanna say?

Nathan Toups (1:23:42)

Yeah.

Yeah, no, yeah, yeah,

yeah, exactly, yeah. We're gonna be reading Reflections on Trusting Trust, which is a famous Ken Thompson ⁓ paper and talk that he gave when he accepted a Turing Award. I think it was a Turing Award. yes, was it? Yes, it was a Turing Award lecture, which, know, seminal work. And then Coding Machines from Lawrence Castelute? Castelute? Castelute? And... ⁓

Carter Morgan (1:23:50)

Yes.

Nathan Toups (1:24:13)

This was actually recommended. big shout out to the Discord. We have a section on our Discord. It's called book suggestions. ⁓ You can suggest books that maybe we haven't covered, but you would love for us to make a case for why you do it. Everybody on the Discord can comment and talk about whether they also would like to hear that or not. This was a suggestion from somebody who was like, hey, I know that sometimes you do essays and blog posts. If you ever need a breather, this would be a really good one to put together. And we took that suggestion.

They have influenced the future of Book Overflow. If you want to be a part of that, come join our Discord. You can go to bookoverflow.io and there's a link to our Discord at the very, very top of the page.

Carter Morgan (1:24:52)

There we go. All right, well, so we're reading, is it two essays we're reading then? Yeah, great, great.

Nathan Toups (1:24:57)

Yeah, it's, yeah, two essays, they're pretty short.

Coding machines is like 30 pages, and then reflections on the trusting trust is three pages, so yeah.

Carter Morgan (1:25:07)

There we go. All

right. Well, it'll be fun. Thanks for listening everyone. Again, go to bookoverflow.io to see our new snazzy website. ⁓ And you can contact us at contact at bookoverflow.io. We're on Twitter at BookOverflowPod. I'm on Twitter at Carter Morgan. Nathan has worked with his consulting agency and his newsletter is rohoroboto.com slash newsletter. Thanks so much for tuning in everyone. Thanks for sticking with us for all the design and data intensive applications. And we're excited to move on to something new next week.

So we'll see you around.

Nathan Toups (1:25:38)

See you.