If I ever talk to software engineering teams, I always look for where the fear is. Like what's the part of the system that they're afraid to touch? And it inevitably ends up talking about stateful data persistence layer things.

Carter (00:20)

Hey there, and welcome to Book Overflows, the podcast for software engineers by software engineers, where every week we read one of the best technical books in the world in an effort to improve our craft. I am Carter Morgan and I'm joined here as always by my co host, Nathan Toops. How are you doing, Nathan?

Nathan Toups (00:32)

Doing great. Hey everybody.

Carter (00:34)

Well, we're excited for the episode we got for you today, another special episode. This is Premode Sadalage, who has been on the podcast before to discuss his book, Building Evolutionary Architectures. But he's partnered with Neil Ford again in one of our favorite books we ever read: Software Architecture, The Hard Parts. Pramode is a consultant at ThoughtWorks and has spent pretty much his whole career on the data side. And so he's a frequent partner with Neil Ford when they want to talk about kind of all things data.

Very interesting guy, tons of expertise. A very great interview. Nathan, you want to give him a sneak peek of what we're about to hear?

Nathan Toups (01:07)

Yeah, you know, it makes total sense that he worked on the hard parts because data is a hard part of these architectures, right? How do we evolve over time? How do you break things up, or how do you combine things back together? I loved this interview because while we did touch on some of the juicier parts of the hard parts, we got his odd opportunity to talk to him about emerging technologies that he's excited about, his thoughts on AI.

and just he's just a wealth of knowledge. I mean, I I came away from this, you know, pretty excited to work on my data project that I've been working on.

Carter (01:44)

Yeah, yeah. I love that about this podcast as our opportunity just to, you know, we just get to ask these folks questions about stuff we're interested in. And we hope, you know, that flows down to you all to gain some of that knowledge as well. Anyhow, stick around for the whole thing. This is promote Sadalage as he reflects on his book, Software Architecture, The Hard Parts.

Carter (02:05)

Well, thanks so much for joining us today, Promote. It's so great to have you back on the podcast again.

Pramod Sadalage (02:11)

Thank you for having me again Nathan and Conrad. It's fun having a chat about the stuff you wrote some years back. Yeah.

Carter (02:17)

Yeah. Well you join an illustrious group of two timers on the podcast. Who have we had? We we've had Mark Mar Mark Richards and Neil Ford, your co-authors. we've I I feel like we've had others, or is that it? There's no way no, we've John Osterho and Uncle Bob on twice.

Yeah, so we we've had a couple. But anyhow, so yeah, it's a it's a growing group and you are now part of the club. we love the book. Mark Richards, I know you didn't collaborate with Mark Richards and Neil Ford on their first book, Fundamentals of Software Architecture, but that actually won our we did like a tournament of all the books we've ever read, and that won. we we declared it the best book we ever recorded on the podcast. so we are very excited to read the

quote unquote sequel to it. And we're also very excited we saw your name on it because we really enjoyed your work with building evolutionary architectures with Neil Ford. so why don't you tell us a bit about kind of this book was written only a couple years ago. but tell us a bit about kind of the the landscape at the time and how you reconnected with Neil Ford and and this time Mark Richards and kind of why you decided to write this book.

Pramod Sadalage (03:23)

Yeah, so like in the industry or you can even say like working day to day at client sites for projects and things like that. You always end up in this situation that hey, here's an architecture right based on the context and based on all of that stuff. And when you show the diagram to someone else. I mean, the diagram does convey a little bit, but it doesn't convey the context. The the the.

the decisions that went into making the diagram and things like that, right? So there is always this notion of, the architecture is only good for the context, the problems you're trying to solve, the amount of money you're willing to spend, the type of people that are around and things like that. And those are all like trade-offs in the decision that went to make the architecture in the first place, right? So if I take that architecture and tell someone else, hey, implement this in your context,

The first thing that person should say, what was your context? Right? Not like, this good or bad? It should always be, what was your context? Because the content determines the architecture, right? So that was always our thought, like having been consulting for almost 30 years now is, hey, what is your context? Right? And many a times, like, it depends, is the standard consulting answer. And many people make fun of it.

But it really does depend, like, is your context, right? So if someone implemented an awesome design at some place and comes to a different place and said, let's implement that design, it should always behave what context that design worked for you in that other place, right? Because that other place may be online 24-7, Google-esque kind of shop, and you're coming here, and we are like a manufacturing place, open 9 to 5, don't care about other stuff.

Maybe that context will change your result. So I was always under this impression that that's what is the thing that that was one factor. The other factor is mostly around this notion of making a choice. Right. Again, goes back to the context. But for making a choice like the noise will distilled book I wrote probably to 2012, 2013 time frame with Martin Fowler was around this notion of how do you pick a storage technology? Right.

what are the mechanisms on picking like hey what do I need I need heavy writes I need heavy reads I need like consistency is more important than latency or whatever these axes on which are dimensions on which you make that choice applies to architecture also right like is residency more important or is other things more important and things like that so that was the second notion that was in my mind like hey this needs to be talked about in terms of architecture.

should not just at the data layer, but at higher level. And there's always this notion of software systems always end up interacting with the data layer at some point of time in their lifespan of these things. So it always is a decision that has to be taken together with the other step. And the third bit was mostly around this notion of, an architect make this decision for the company? And I have seen that many a times,

answer is no because what decision you make ends up spending money and generally money spent is better made by someone else than the architect because as an architect I may pick the best best bestest solution but the business may say I don't care about finance I just need two nights and that's a money decision right not architect

So as a consultant, have many times been in situations where we have ideas, where we have this notion of this is a way to solve it. But we never make that decision as consultants. The reason being, hey, this is a money spent decision, and I don't want to make this decision for the customer. So take it back to the stakeholder saying, here are possible three options. Here are the pros and cons of each one of the options. You pick this.

Nathan Toups (07:32)

Mm-hmm.

Pramod Sadalage (07:44)

Here are the tradeoffs you're going to live with. You pick this, here are the tradeoffs you're going to live with. So that notion is in the brain always as a consultant. So that made sense like, this also should be talked about, right? In the sense of when you are making decisions, all you are saying is I'm picking certain tradeoffs over others. But even underneath that is what are the tradeoffs in the first place? And let's expose them to the stakeholders.

So even they understand, what is the thing I'm playing with here? And what should I value more or others? And as architects, it's our job to kind of expose those trade-offs, expose those things back to the stakeholders. And sometimes I even think as architects, we should even expose that to developers and like junior architects and junior developers so that they learn in the process also. Like, hey, why did we pick this?

because somebody will just going to go read somewhere else and say hey like this looks dumb that design is better right and and the reason is hey like here are the context or here are the constraints under which we made this decision so you better know the constraints so that you know the decision that

So three driving factors to come to this like notion and Neil and Mark were actually writing the book. then they asked me, Hey, do you want to contribute on the data side? And I like, it's always fun working with Neil and Mark. So I raised my hand and say, sure, why not? And then I ended up talking about the data parts inside the, like the star rating system or the other stuff you see in the storage parts or the breaking apart. Most of that stuff is.

basically based on the life experience gained through talking to clients and understanding how this stuff works.

Carter (09:40)

How'd you get started in consulting?

Pramod Sadalage (09:43)

I would say it was more accidental. was the first job I had was in a consulting company. And then from there I took the second job, which was a big insurance company and probably worked there maybe a year, year and a half. And then from there I joined ThoughtWorks. I've been at ThoughtWorks for 28 years now, all my life almost doing consulting and

There are two things that I really love about consulting is this ability to see multiple problems across a bigger span of life or of the working life, if you want to call that. And that enables you to see patterns of like, under this context, I came up with this solution and it worked. Does the context match? That's what I look when I go to a new client. Does the context match with the other thing I did somewhere?

long time back or few years back. If the context matches, then I try to find, okay, what's the differences and then what are the solutions I used for similar context is. So then I try to bring those solutions, right? So that pattern recognition of problems is what gives me joy. Like, hey, there's a pattern here, right? And Martin taught me this like way back, like I think it was 99 when he was working with Fifth ThoughtWorks for the first time.

and he taught me this idea of hey if you step back a little bit everything's a pattern and can you categorize those patterns and if you can categorize those patterns then what are the commonalities what are they like if you apply this statistical way of looking at it hey what's the common what's in the like standard deviation middle like plus one here are they and what's the outliers and then can the outliers be like normalized off.

And then what sits in the middle is a nice little pattern, right? So like refactoring patterns, analysis patterns, architectural patterns are all those kinds of patterns that you can look at. And then once you recognize the pattern, then can you give it a name, right? If you can give it a name, then you can communicate in a much denser way. If Carter knows the pattern name, all I have to tell Carter is a pattern name, like extract method. Okay. The moment you say that,

If the pattern names well known in refactoring, example, extract method means, OK, take this stuff, extract as method, call the method, make sure everything works before and after. OK, that's a known thing. Then I just tell Carter, hey, let's extract this method. I don't need to tell a lot more details to Carter anymore because it's a well-known pattern, right? So yeah. So that pattern stuff, I really love that stuff. So that's what got me hooked onto this.

being in consulting is this notion of learning from many places.

Carter (12:38)

I'm I'm jealous of consultants for that exact same reason, right? I think there's something fun about working at the same company and kinda you have like your baby, right? And you get to see it grow over time. that's really fun. But sometimes I'm just when I you have to make those decisions about, okay, well, how you know, we're choosing a new architecture. We have to go down this path. I'm just like, I wish I had just I wish I had more knowledge. I wish I I could know, you know, all the different kind of pitfalls, which is why we're so grateful for books like.

Software architecture, the hard part. Because if you can't spend 28 years consulting, right, I I suppose the next best thing is reading these books, right? And getting the wisdom from the people who have who have done all this consulting.

Pramod Sadalage (13:19)

Right yeah yeah I guess similarly I would say the same thing like learning from other people because they have been through that journey so learning from others experience is a great thing yeah.

Nathan Toups (13:31)

So the book came out, I think, in 2021. So not that that long ago, but I noticed I was l doing a little background research and you went back and got your master's master's degree from Northwestern University and you were finishing up right around 2020. So were you working or and contributing to this book while you were finishing up your master's program? Or how did how did those overlap?

Pramod Sadalage (13:54)

Yeah, so a little bit of a back story to that. The first book I wrote in 2006 as well as the No School Distance book in 2012 is textbook in a lot of universities. So these universities would call me and I go give guest lectures and that kind of stuff. And one of the professors they say, why don't you teach and that kind of stuff? And it would be really nice if you at least got a master so we can tell our deans, hey, we have a person.

Nathan Toups (14:06)

Yeah.

Pramod Sadalage (14:23)

who has written books, has master's, but just with undergrad, it's really hard to justify. So it was in my mind to at some point of time get like a master's. And 2018 is when Rebecca Parsons kind of pushed me, hey, promote, where's your master's? So I said, OK, fine, let's go enroll. And then I was looking for like part-time programs in and around Chicago and University of Chicago and Northwestern being the two biggest ones around Chicago.

I said, let's go look at this stuff. So I finally enrolled in Northwestern, did a program in masters in IT. That's what they called it. And it was like 70 % like computer comp sci and 30 % was business, which was a good way for a consultant to understand financials as well as like the comp sci side of the shop here. And 2018 I started and the last semester or last quarter in Northwestern way of looking at it.

pandemic came so we are remote at that time. But yeah, you're right, like I was writing this book and then working full time as well as doing my masters, all three at the same time at that time. And the person that was most happiest when I finished the program was my wife, because then I will be available on Saturdays, not doing homework on Monday mornings and Tuesday mornings and evenings with my group and that kind of stuff. So it was really a good way of like finishing.

I think the big thing I learned like during the Masters was mostly around like how do you structure your thoughts in a better way. So there is this notion of like not just giving an answer but like structuring it with where and what are the reasons for the answer. How does the answer affect other people and that kind of stuff. So that structure in thought and that kind of stuff was really useful as part of the Master program. So overall it was fun experience.

Someone grading your work, like after 20 years of being in work culture, was really hard. The first quarter was really hard when people put like an A and like X or stuff on your paper. But yeah, overall really good experience.

Carter (16:26)

Mm.

Nathan Toups (16:38)

That's super cool.

Carter (16:38)

I know that feeling. Nathan and I actually met in our master's program. We did it was all online through Georgia Tech. but kind of the same thing. You know, we've been in the industry for a bit and then for separate reasons, we're like, you know what, a master's would be nice. And yeah, I I just know that feeling of like getting grades handed back. I'm like, what do you know? I'm a software engineer, right? Like I and also I and I can contest too, my wife was very happy when we when I wrapped up my master's degree.

Pramod Sadalage (16:53)

you

Yeah.

Carter (17:09)

What do you think

There's the evolving role of expertise in this field is something I think about a lot because large language models have obviously kind of accelerated velocity in a way where kind of just how fast you can physically type is no longer a bottleneck, right? And at the same time, it has opened up the ability to create full stack applications to non-engineers, right? You know, obviously we can talk about how scalable and extensible these applications are, but you know, that's a new thing that someone, you know, maybe in

enterprising founder type could be like, you know what, I I have this idea and I'm just gonna kind of vibe code it in a couple days. and so, you know, we're talking as as as three men who have a master's, and then you obviously have a ton of experience across consulting. and I know Thoughtworks publishes a lot of stuff about kind of LLM usage. I mean, what do you think the interplay these days is between AI enhanced development and you know, the

This expertise that we build up through either education or experience.

Pramod Sadalage (18:15)

Yeah, that's a good question. I think the expertise does matter over a long period of time, right? Like I could totally wipe out an app, sure. But is the wipe out an app going to be in production for the next 10 years? Are you going to be able to change the requirements on it as the industry or the ecosystem around you changes, the business changes and that kind of stuff? How comfortable are you going to be like?

Hey, let's change version one, version two, version three, and change that with the WipeCoded app as the users go. Are you going to be able to keep the data around for the users to see the data migrated over version one, version two, and that kind of stuff? Those are the kinds of questions you should be asking. Not just, can I WipeCoded app? The other related question to this is,

I love the acceleration. I use the LLMs myself, right? It's not that I am against ILLMs. What I love is the structure we have to put around something because the speed of typing code is not what is valuable in the longer. How you structure your programs, how do you like ensure the classes are well designed? How do you ensure losing coupled systems? How do you ensure a bunch of these quality gates, if you want to call it, right?

Those are still important no matter who writes the code, right? For the long term visibility or long term viability of the app, right? So in that scenario, if you think of it, how do I ensure that? Do I need a good harness? Do I need a good structure? Do I constrain the LLM to operate in a certain lane? Do I design the scenes of the application like if you want to use like

domain driven design concepts, do I want to constrain it certain lanes, do I want to structure a monolith versus like separate repos for each service and that kind of stuff. Those are decisions still humans need to make, right. We can't really like take a, like if you really go back like say 1995, you may have like a use case bundle of use cases. You can't just take all of that and feed it to LLM and expect it to spit out.

like a super doper well-designed, like modular monolith or maybe a microservice stuff back at you, right? It's just going to look at all of it and maybe it put all of that in one big file, right? Now, sure, that would work. But then version 1.1 of this is when you start making modifications, what happens? That's where I think is the key on how do you have this long-term sustainable.

Because even in normal, like human written code, if you want to think, we are always thinking about it's not version one that matters. It's version 1.1, 9.3, or 25.50 is where it matters. Because long term viability of this thing is what gives it sustainability. Because software, once written, it has its own life. You can't just like delete it and say, OK, let me come up with the whole new thing.

You could totally do that. Like I could delete all the code and say, I'm going to write fresh code. But then what happens with the data that the app captured? You still have to work with that, refactor it, maybe migrate it and things like that. So those are the kinds of questions we have to deal with. And certainly velocity increases. Amazing. We, I played that a lot. Sometimes for prototyping, sometimes for making this easier.

Sometimes for even for how do I, how to understand the right way to pass commands back to the LLM. So it does the right thing. Once it does the right thing, how does it test itself and things like that are amazing places to work right now for improved product.

Nathan Toups (22:22)

This this kicks off a lot of interesting things, partly with the human component, which is, you know, we we we can write code so so quickly. And I do feel like I spend a lot of my time putting context and boundaries around processes. but what we you and I were briefly talking before the the the podcast started about how long lived the decisions around data storage and schema and these things can easily be and that I think that

And so I'm curious, you know, when you have these systems that are so powerful, these these large language models, you know, we've seen this time and time again on Twitter and other places where somebody accidentally deletes a database because they sent the wrong prompt to their LLM.

I I can only imagine that, you know, these aren't new problems. You could always have deleted the database, but it's so easy to delete your database now. I I would love to see, you know, you've seen a lot of the changes that are happening in the industry. What new challenges are you running into when teams are thinking through the hard parts around data now that there's all this pressure to to ship with with large language models?

Pramod Sadalage (23:45)

Yeah, yeah,

that that makes a lot of sense, especially I think the one notion I have realized over all these years of working is no matter what application or what code you're writing.

Like when it is not running, it doesn't have any state in it. When it starts running, it has carry certain state because of either user input or it's pulling data from somewhere else or whatever. It has that state. And once that state is persisted somewhere, again, it doesn't have a state in itself. we generally call them stateless services because it has transferred the state to something else, right? But for the service to be useful,

The outside people are expecting you will rehydrate that state at some point of time if you ask it for that. A very classic example of this would be like in an e-commerce app, I take the order, the order is sitting in a service somewhere, and then it persists, and then the order object probably gets garbage collected or something like that. So there is no longer that order. But I ask for that order later on, like slash order slash 123.

It has to go back to the storage and bring it back, like rehydrate it back. So that stored persistence of the order is what carries. People fear that because, as I change code, can I get back that same order in the same way back to the user? Because they're expecting all the attributes of the order to be this. Probably not in the same structure. You could probably change the structure.

But they're expecting the order ID to be there. They're expecting the order date to be there. They're expecting like what items I order, what are the price I paid. Like you could go on your own Amazon.com right now, for example, and look at your order history from like 20 years ago when you ordered something. You'll probably see the same order like in the same way structure and all the information's there. But I bet the order object has gone to like lots of transformations in the time.

when you wrote it 20 years ago versus today, right? So that notion of carrying the order state from way back when to today is like you're doing micro migrations in some ways or things like that. People fear that a lot. What if I lose data and things like that, right? So that's why we need to be sure that as you change the structure of the object in the memory,

You want to make sure the persistent object can also be mapped to it. There is a way to map it and there is a way to migrate, right? So in relational databases, it's kind of easy because when you change the structure of the database, the database changes structure for all objects, right? It's not like you're changing it for order number one and not changing order two, because when you add a column, everybody gets a new column. Then as a developer, you need to be more aware of, OK, all the past data.

what does this column need to have? Right? So that's where like working with the business, making sure you're putting the right things in there. That's what keeps the order object valid for future use case. Similarly, you can do the same thing for other things also, like this order is has a history. Maybe there is new things that don't have a history. Maybe there are things that you acquire from other places. Like, let's say as a company, acquire another company.

and you try to retire their systems, then okay, how do I get data from their systems, bring it back into our systems? What is the representation of the order over there? What's the representation of the order over here? How do I map? And all these things become complicated and that's why people fear them, right? But I always go back to this notion of what does the order mean in the business? Am I represented in that order? As long as I think like a business, like the person who cares about that one order.

I can make good sense of what is going on and act accordingly. Right. So always thinking that that term is code is there in the service of data. Like some people don't like this sentence. The code is there in the service of data because the data is there long term. The code can be changed. Like your order object goes through a number of transformations, maybe wrote in Java. People are now writing it Golang or whatever, but it's still the same order data.

Carter (28:20)

never heard it phrased that way before, that code is in service of the data. but I I I totally agree. And I I had to develop more discipline around this myself because I am you know, I I I have great respect for people who are like, you know, before we cut once, we're gonna measure ten times. And I don't even like measuring the one time, right? Like I and so but obviously, you know, and I think there are

Areas of software engineering where that kind of like fast iteration and exploration can be very valuable. But data is not one of them, right? And so I have gained a lot of appreciation and have been trying to practice that skill a lot of, you know, like, okay, let's really think about our data model here, because there's a lot about this application that can change, but this piece right here, I mean, all data can change, right? And that you talked about that in building evolutionary architecture is about.

data migration and some strategies around that, but it is much, much harder. how do you think engineers can develop that kind of discipline and th those good analytical skills to lock in on the best or maybe we'll say least worst data shape from the beginning?

Pramod Sadalage (29:38)

Yeah, so there are two ways right. One is of course thinking a little bit about hey, what am I trying to process? What are like maybe two or three things? What am I trying to process is one. The other thing is hey, what are these common? You can see rules or best practices or conventions your own teams has set up or your own enterprise architecture has given you a guidance on and things like that. Am I implementing all of those?

And the third bit is, do I meet all the requirements right now? Right. Don't try to like, like the measure 10 times that you mentioned. I chuckled a little bit inside because measuring 10 times probably it's going to slow down everyone else. Right. And people don't like that either. Right. So I am more of a measure once or twice and then cut kind of a person. And the reason for that is if you go too deep down the path of, I want to make sure this is the perfect

data structure for storage, then maybe you'll spend a lot of time just trying to make it perfect, right. But at the same time, is this good enough for right now is the way I go after. But at the same time, you want to reduce the blast radius. That's what causes you problem, right. If there is this one table that 50 other things are directly hooked into and those 50 other things, maybe other services, maybe store procedures.

maybe views, maybe extracts or whatever stuff, then the blast radius of a change in that table causes you a lot of pain. And when that happens, you're going to be afraid of changing. Right? And then you start putting a lot of scaffolding around it. You see like design where people just make a foreign key and add other stuff into this other.

Right? Like order ID, this table also has an order ID and other stuff gets added here because I'm afraid of making changes here. Right? And at the end of it, you have like 20 order tables and you have to join all of them to get one order object. Right? So that happens because people are afraid of the blast radius. And that happens because we do not do good design of the surrounding things along with the table, like loosely coupled systems.

Nathan Toups (31:42)

To get me a

Pramod Sadalage (31:56)

not giving direct access to the database for everyone, like making sure, hey, when I have stored procedures, fine, you can write your procedure, not a problem, but don't have like repetitive stuff talking to the same thing again and again. Make sure you isolate access in one place and everything's using that one place and things like that. Like what do you do inside the code for like, DRY principle, like do not repeat yourself and a bunch of that kind of stuff applies to the data access also.

Right. So if you follow those principles, then what you have is a good structure in interaction also. Right. Because many years we have been like, hey, code, we think about all these design principles and stuff. Data, we think about all this stuff, but the interaction we totally forget. We do whatever we want. And that causes a lot of pain because of blast radius issues. Right. So you can do a good way of like, hey, I have this table.

that is being used by like n number of places inside the, let's say we have a monolith or whatever. But then don't create like 20 different finders or 20 different repository things that are talking directly to this. Create one repository class that is talking to your table. Then what do you need you put in that repository and everybody else is using that. Right. Now if something in the table changes, you change this one repository class.

you immediately find everybody else who is using it and then refactor appropriate. Right. So you have good dependency management system going on that way. The other way also of doing this thing is now, of course, when you break this up into microservices or whatever, fine, break it up, but don't give other services access to your database. If you need something from me, talk to me via a service. Don't talk to me directly. Right.

Because then we defeat the whole purpose of breaking it up in the first place. So that's why we, think in chapter five or six, we talk about this loose connection and things like that and a five step process on how you can break it apart as you go along. Because this blast radius is what causes fear and this like people seize up because they see so much dependency all over the place. They say, let's not like worry with this. Let's create a parallel structure on the side.

and two or three years later we have four or five parallel structures and then we are in bigger mess.

Nathan Toups (34:26)

Yeah.

It it it is it's always interesting. I've I've I've I went I was working in industry, I've also been doing consulting for smaller startups and you know you end up having this transition from the founder code into I've been lucky enough that, you know, these are series A, series B, series C funded startups. So they're they've maybe are in the process of getting product market fit. They kind of understand their customer at least a lot better than they did when they founded it. But a lot of times we're deal grappling with a lot of technic technical debt.

Pramod Sadalage (34:54)

Mm.

Nathan Toups (34:58)

that kind of emerged from some quick and dirt dirty decisions that came up. and so a lot of times it's the founder code that we run into. And I think this is that pattern is one I I I've seen a lot, especially from companies that came from like the 2010s era, where there was that explosion of microservices. There was this explosion of maybe experimenting with all the different new NoSQL technologies that were coming out.

And then you end up getting this like distributed monolith, right? Where they they kind of understood, but they didn't really understand. And so instead of having, you know, this service-oriented piece where there's data encapsulated behind the service, it's like they have a bunch of services talking to one database and you can't reason about the system. and so it yeah, it's it's been really interesting to he like as I read through these books, and hear you speaking about this, that it's not, you know.

Pramod Sadalage (35:27)

Yeah.

Nathan Toups (35:53)

We see this in the enterprise where folks are transitioning from maybe, you know, old systems into new things where they're breaking things up. But there's also the kind of like lots of bad decisions, or I should say I should use a Chesterton's fence approach, which is this the decisions that were made that made sense at the time don't make sense anymore. And the the business has moved on and they still have these, you know, decisions that aren't serving the company well.

Pramod Sadalage (36:18)

Yeah,

yeah.

Nathan Toups (36:19)

any

longer. I think that's maybe 'cause it's not always just bad. It's just not good now.

Pramod Sadalage (36:22)

Yeah, agreed. Yeah. And in

the defense of startups, speed is the most important thing for them in the beginning. Right. So they don't want to like, hey, I don't have time to write this finder, write this repository, blah, blah. There's a database. I'll just query the database and I'm done. And that's fine too. Right. But then we are paying, creating technical debt for the future based on today's needs.

And that's fine too, as long as we understand that's a trade off again, right? Going back to the trade off, that's a trade off we picked and that's fine too. The only notion that we then need to make sure is, we have to pay this debt later at some point of time. And then we need to allocate enough cycles for that, right? Because once the product market fit happens, the speed necessary increases. It doesn't decrease.

Nathan Toups (36:54)

Yeah.

Pramod Sadalage (37:17)

So that's when people don't allocate enough time to pay the debt. That's where the problem happens. I'm totally OK with creating the debt, but allocating enough to pay that debt down is not done with the product tax.

Nathan Toups (37:33)

Right.

Pramod Sadalage (37:34)

right and then we end up paying this debt forever and then at some point people think about let just rewrite this whole.

Nathan Toups (37:44)

Yeah.

Carter (37:45)

It's listeners of the podcast will be chuckling right now because that is exactly what we are doing at my company. We are a series A startup and the whole app was just kind of a big bundle of is you know, it's funny because it it was like I say it's the first spiritually vibe coded app because it was before large language models. But you know, it is a lot of just like tack on a feature here, attack on a feature there. you know, the the data model was and in some places was really good, but in some places was maybe less clear and

We just got to the point where the whole application just kind of fell over on itself. our problem was we were using MongoDB and not using it properly. And so we're doing like a lot of like round trips to the database and then doing like these joins in memory. and so we we kind of made the decision we're like, you know what? Like it it let's just rewrite lots of this.

Move it to what we think is the more appropriate pattern. And like you were saying, Nathan, like it's all about product market fit. You discover what your needs are, who your customers are. And so we're operating with a lot of knowledge that the earlier developers didn't have. but it it it reminds me too of something you mentioned. And I I remember reading about this in working effectively with legacy code by Michael Feathers, which is this idea that you get scared to touch.

Parts of the code base, right? And we're reading learning domain-driven design by Vlad Kononov this week. And I thought it was funny. He he mentioned he's talking about like, what's a core subdomain? What's like a non-core subdomain? He's like, this is not a perfect heuristic, but it's pretty good, which is like, what's the worst code? Right. Like that's probably your core subdomain. because that's the part people are most are are are most scared to change. And I kind of thought that that was a sign of like,

good meaty technical engineering. If I'm scared to touch this code, then that's gotta be really valuable code. And reading working effectively with legacy code, and what you're saying right now is like, not really. Like we shouldn't be scared to touch our code. And if you are scared to touch any code, like that's a warning, you know, that that's a big alarm bell blaring. Like, hey, you should take another look at this and you should revisit some of your assumptions about how you built this.

And so I I know that's a common feeling amongst engineers. It's like, I'm scared to touch this part, you know, we don't go into that part of the code base. Or if we do, it's just to make the the minimal change we can and then to get out. what would be your recommendation for engineers trying to, you know, courage is a silly word, right? But to develop like the the intuition and the courage to jump in and and to change the scary parts of the code base or the database. How would you help them navigate that?

Pramod Sadalage (40:25)

Yeah, that's a really good part of this life that I lead is how do you go back in and understand certain things, right? So many times when we are dropped into client sites, we end up seeing this code base or the database. Like I saw one database that had 287,000 tables, right? That was not fun, right? Like, how do you know what is actually used here, what is not used?

Carter (40:52)

wow.

Pramod Sadalage (40:54)

what is touched and this kind of stuff. So many times the first thing I tend to do is, hey, let's instrument and see what all is actually used. Right. So if you have like a standard test runs or you can even do this in UAT or whatever, try to build an instrumentation that tells you, okay, code came in here, went over here. I have called this method, called this method, wrote to this table, read from this table.

And you have this nice little graph of call graph. If you, I think that's what we call it in the code land that tells you, okay, what all tables were touched, how they were touched. And over a period of time, you get familiar with, okay, this is what generally happens if I go in here kind of stuff, right? One extra special thing that happened with databases is because especially legacy databases, people built this monolith and then built a bunch of code and that kind of stuff on it. The code itself is a spaghetti, but

Let's not worry about that. What happens is when you have this database, some other team sitting somewhere else says, hey, I need access to like these four things from the database and somebody just gives them access and they are now reading from your database. Right. So database based integration happens. There are other side of the shop that say, hey, that like other department or other business units needs extracts of order. So end of day, let's like

write a job that will extract all the orders and ship them off. OK, we got one other dependency. There's some other dependencies that say, let's load the prices into the system every night. So there's another job writing prices. All of this is not in your code base, right? Because all these dependencies are live dependencies on the database. That's why people are scared of changing the database. Because hey, I can control what my app does to the database. I can take care of

like remapping the columns, remapping tables. When I deploy this, 10 other jobs that I have no idea of are going to fail. And I do not have a good place where I can look at all these dependencies. So that's why people are scared, especially on the data side, is because of these hidden dependencies that have kind of sprung up on the database over a period of time. This is not how it was intended, but that's what happens at enterprise level.

because people end up giving access and taking access, taking data feeds and that kind of stuff. So the first thing I generally try to deal in this kind of situation is write the instrument on the database layer itself to see if I can see who's reading, who's writing. Simple stuff, right? So then I can know, OK, what are the people reading from? What are the people who are writing to? Can I go talk to those groups? Can I slowly segregate them inside the database?

Instead of talking directly to my schema, can I make them talk to some other schema that has views on my schema? So that they're coming through one other indirection into the system. Now what that does is slowly one tells me all the dependencies. And then second thing, slowly now I have created interface sitting on my top of my database through views, through like other schemas, synonyms, whatever other things your database can give you.

And slowly over time you work with those teams, hey, can I create an API for you to load prices? Right. So when they say yes, then okay, you go create an API where they upload a file. Now my code will read that file and put the prices in. Now slowly the price import stuff has been taken out. Right. Similarly on the extract side, you can write something that does the extract. So this is like a multi-step, multi-time period.

thing that you have to do is one first try to understand what's going on, then slowly split them inside the database itself. Then third, you kind of like slowly start off boarding them. Now what you're left with at the end of this journey is you accessing your own database. Your code may still be messy, but you know that you're accessing your own database. Nobody else is. Right. So that's a place of confidence. I think even though the code may be messy.

It's a place of confidence because what I do will only affect me. Nobody else. Right. That is one of the biggest fear factors is like, hey, what if I do something that will affect others that I don't know of? So you want to get to a place where you know nobody is getting going to get affected once you're in that place. Now, refactoring comes a little bit easier because now by now you have a good enough harness, you have good enough instrumentation, you have good enough test system.

to like now test and slowly start.

Nathan Toups (45:51)

Yeah. Yeah, that's great. I

Pramod Sadalage (45:53)

And this

takes time like an investment from the owners of the software itself. Because if you say I have no time for this, then you are going to make it more worse and worse over here.

Nathan Toups (46:05)

Yeah, that's probably what got them there in the first place. I remember one time we I I was working with a company that had we it was a merger and acquisition sort of period of the time with the funding. And so we brought in a bunch of teams and a bunch of different practices. And that was one of the things that the core company, we had very good observability and telemetry, a lot of knowledge and maturity around that. But one of the companies we acquired really didn't. And

Pramod Sadalage (46:08)

Exactly,

Nathan Toups (46:34)

I just remember when we set up Datadog and we turned on enhanced monitoring with the databases, the conversations that it opened up, you know, there were people like, that's why we've been struggling with you know, it's just and it's funny because these are very intelligent people who're solving really complex business problems and just didn't have the maturity to think through like, I need data to make a decision on the

Pramod Sadalage (46:48)

Yeah.

Nathan Toups (47:02)

system worked. It was just yeah, it was this kind of black box in a lot of ways.

Pramod Sadalage (47:06)

Yeah, yeah, yeah, that's true. Like one very good example that comes to my mind, like around 2018 or 19. There was this big like company that was doing like a lot of things on the SQL Server instance and a login when a person logs in, handled via Okta or something like that. The app used to spin for like six seconds or so, trying to do stuff, right?

So we put like diner trace in there trying to figure out what it does and that kind of stuff. Again, this is in a test system. And then what we saw was once it logs in, tries to go in and see, okay, what privileges this person has, what company do they belong to, what company division they belong to, what things they can access, what profile they have, what preferences they have. A bunch of these stuff was being read from the database. And then all of it was coming back as JSON back to the app.

so that it would store that in memory, 130,000 rows will be pulled from the database just to like paint this thing, right? And presumably you could assume that a user preference, even if it changes like once a day, is not a thing you want to pull back from the database every time, right? So the solution to that was, hey, let's not break anything here.

But keep it as is when login happens, you just go to Redis and pull that JSON back from there. And we'll make sure that JSON is populated on a frequency basis that is useful for the business. And we'll do the 130,000, same exact thing, but through a different method. But now login means login, it immediately has that JSON for it. So that's the first step. Now you can go back and refact.

there's no life dependency on that thing happening, right? So those are the kinds of things that instrumentation gives you, right?

Nathan Toups (49:05)

Right.

Y yeah, similar story. We we were we put in some instrumentation and we just knew that this thing was taking a long time. We realized it was actually it had a weird issue with caching credentials and it was retrying a SQL connection three times. Well, they fix it and then they realize they've introduced a race condition because the delay that that had that it had given it had actually given it enough time for this other thing that again they hadn't

had the right instrumentation around. And so again, it was we we we ended up having again a really nice conversation with real data and a another part of our company, this was n normal, you know, operations, but it was it was interesting to show them sort of like, hey, this is a way things can be. We don't have to just kind of like freak out, have a fire drill, that you can actually, you know, take a real nice engineering approach. And

Pramod Sadalage (50:00)

Yeah.

Nathan Toups (50:01)

Yeah, I love the different shapes of this, especially when it it's it's not obvious. If you don't measure, it's not obvious all the time, especially in a complex system.

Pramod Sadalage (50:08)

Yeah, that's it.

Nathan Toups (50:11)

okay, so I I I love hearing you know, you you again get to see across the industry in a lot of areas. And I'm this is really just sort of like a I'm curious question, which is are there new any like newer patterns or technologies that you're excited about emerging things that are coming up in the data space that you've been keeping an eye on or that you've been an advocate for in recent years?

Pramod Sadalage (50:38)

Yeah, certainly. Like if you are seen at LakeBase, which is a product from Databricks, think it is like Postgres compatible thing that runs on top of Databricks. So you can like imagine you are app developer and you can use LakeBase for like your normal Postgres based development as app developer. Right. So what it ends up doing is gives you the access. Like I have a Postgres instance now to play with and that kind of stuff.

Many benefits from that come because of one, it is in the same ecosystem as Databricks. So if you need data from this app, you no longer have to do like ETL or whatever. You could do like a bunch of interesting things. Let's say you have analytical data that you want to like surface inside your app. You can make that happen very easily and things like that. That apart, the thing I am really excited about is this notion of I can branch my database.

So let's say you're a developer and maybe you're six of us are on a team and each one of us needs an instance of a database to work with because we want to be independent of each other as we are making changes, right? So if you're in a shared development space, that itself is a smell because if you change some table, that's going to affect me immediately without you committing the code, going through PR and I getting the code till that time I'm broken.

because the table is changed, but I don't have the code base, right? So everybody being on their own database instance makes a lot of sense. So this thing, LakeBase gives you the power to do that just at a command. Like all I have to do is Databricks create a branch kind of stuff. It gives you a branch that you can work on. Now that's the first bit. The second bit is as you start doing this, you can like, let's say you have code as well as database migration changes. You commit both of those.

You create a pull request. The moment a pull request is created, you can make your CI create a branch, independent branch of itself, deploy this code there, deploy the migrations there, run the test. If everything passes, then open it up for comments. If those things don't pass, you didn't write the migrations properly. You didn't do a bunch of stuff. So why waste review comment time? First step, right?

The second is once it all there and people say, okay, this looks good. And they allow it to merge. Now we can, when the merge happens, you can again branch from the main, run the same thing again, because in the meantime, the PR review was happening, something else might have changed, right? For example, Carter may have be changing the order table, right? And in the time the PR review happened and that kind of stuff, the table that you're going to change was renamed. Let's say for example, right?

Now when you go to merge into main that migration not run anymore because there's no longer order table there's order details table for example right. So your migrations fail so we want to run tests again there and if that fails kind of stop the PR from merging. Right. All of this is available immediately on this one single create a branch philosophy. Right. The other thing goes back to safety we talked about the safety for experimentation. Right.

Now imagine you have a really large legacy system, where do I experiment if there are only three database instances, production, stage and dev.

I can't experiment on dev because if I break anything there, the whole dev team is going to be pissed at me because I slowed them down. Same thing at stage, like nobody's going to let you touch stage because critical testing is happening there. And obviously nobody's going to let you touch production because stuff is production, right? So now for you to test something and experiment to find the right solution, you're going to just Google around, maybe use chat GPT or something to come up with a solution.

and go with it because that's the first thing that worked instead of the best thing that worked. Right, see the difference here, right? So if I have a space to experiment and I know for a fact I'm not going to break anyone, that's awesome place to be. Fear reduces, then I can try three, four different types of solutions. this looks the best. I can probably pair with others, pair with the data person, data architect, DBA type person, show them my design.

Nathan Toups (54:49)

Yeah.

Pramod Sadalage (55:09)

and then get approval or whatever, and then start the PR class, right? So that, think, the space to experiment is the amazing concept there that helps. And the best thing is, most of this is, it's not like a 100 % copy, zero copy, just like how Snowflake works or other places work, is zero copy. And then whatever changes you make stays local to your branch. They don't go back to the main branch, right? So that kind of stuff can be easily done.

Imagine at the end of the day when people stop working, you kind of drop the staging branch totally. And tomorrow when you start again, you get a fresh copy of staging and start again. Same thing for developers, same thing for everyone, right? So that is the kind of power it gives you. So I'm really excited about to see what's going to happen with this technology.

Nathan Toups (55:56)

That's cool.

I've I've been building a our website, just been working on it in our spare time, actually using some agentic coding and so part of it's on Vercel and I'm also using Neon, which is also using branching Postgres. And just for me personally, when I just don't want to have a side effect, it's been amazing. Even just like team of one is basically me working on it, but I'm like, I can just open a PR and it spins up a branch and even if I have a migration and this change,

Pramod Sadalage (56:14)

Yeah.

Nathan Toups (56:29)

It's been and it's been a cool way for me to kick the tires and really understand the power of this so that I can recommend it to, you know, larger teams or or talk about this. And so yeah, I'm I'm so excited. I I it's cool to see that that it this these patterns are popping up all over the place now. That's amazing.

Pramod Sadalage (56:45)

Yeah, so

many users, right? Like I could have a branch that is used specifically for performance testing, for example, right? So I could spin up a bigger thing and do my performance testing there, or I could have something very functional, right? So some of these pricing engines are like coating engines, need a specific set of data for the pricing to work correctly, right? Like a good example of this would be healthcare pricing works with like, hey, each zip code has a appropriate number of data and that kind of stuff.

Nathan Toups (56:50)

Mm-hmm.

Pramod Sadalage (57:14)

humans living in the zip code kind of stuff. So then you populate that kind of data in that branch and test your stuff against that. You don't need to carry that branch with you all the time. It's just a common branch you take. Apply your changes to that, it. When you're done, you kind of destroy that branch and you're back to. So high functional data necessity. You can use this same thing for other related stuff that you can, anything specific that you need for testing, you could use this kind of concept.

Nathan Toups (57:26)

Right.

Pramod Sadalage (57:44)

and takes away the whole provisioning, takes away the whole, you could do the same exact thing with Docker images and that kind of stuff. But then you have to have one or two specific DevOps type people on your project to help you out with that stuff. You don't need any of that anymore. So it's really interesting.

Nathan Toups (58:04)

Super cool.

Carter (58:06)

Well, promote, we can't thank you enough for joining us today. And again, we love the book. A big recommend to anyone. we would wanted hear from you anything you'd recommend to our listeners for their reading. That can be technical, non technical, fiction, nonfiction, even if it's just a blog post you found interesting. anything for our readers to listen to, we're always trying to get.

Pramod Sadalage (58:28)

Yeah, maybe two things. One on the technical side is mostly trying to keep up with what's going on in harness engineering and that kind of stuff, especially in the LLM land about like how to keep your agents in check and trying to get quality stuff out of it and things like that. No specific place to recommend, but anything that talks about harness engineering may be a thing that's useful in today's land.

And the other place probably to talk about is, is this behavioral economics stuff. am like fascinated by the whole topic of behavioral economics and how it affects decisions we make and that kind of stuff. So two books, maybe one in that thing is Thinking Fast and Slow by Daniel K. Mann. I think it's an amazing book that you can apply like in different places in life, not just in economics, but also in other places and things like that.

that has been really influential in me trying to think about that kind of stuff. And there's this other book by, I forget the author name. It's about like how managers should behave and why like giving autonomy as well as like giving a space for employees to prosper is a better way to talk about productivity and that kind of stuff instead of like just putting them on a task and making them do things. I think it is by someone from

from someone from like a.

the people who do like surveys and things like that. I forget the name of the company. I can send you a link to the book. Really interesting book that talks about like how you enable the people to like do amazing things by giving them freedom and just ensuring that they have what they need to do their job. Right. So that was amazing. I'll send you a link once.

Nathan Toups (1:00:24)

Excellent.

Carter (1:00:25)

Well, fantastic. Thanks again so much for joining us, Promote. for all of our listeners out there, you can always contact us at contact at bookoverflow.io. You can go to the website bookoverflow.io to check out all of our past episodes, our reading schedule. You can find us on Twitter at BookOverflow Pod. I'm on Twitter at Carter Morgan and Nathan and his work with his consulting agency, Rojo Roboto, arrojo roboto.com. again, promote, we can't thank you enough. Such a pleasure to have you back on. And we look forward, you know, if you do any more writing in the future, we we can't wait to read it because it's all been excellent.

Pramod Sadalage (1:00:55)

Alright, thank you, Carter and Nathan. I'm writing a book on data architecture. Hopefully it will be done soon, so we'll see. Awesome. Alright. Thank you.

Carter (1:01:00)

There we go. Okay. We'll keep us posted. All right. I'll see you later, folks.

Nathan Toups (1:01:00)

Amazing. Okay.