• Later and Mavrck
• Ian on Twitter and Threads

Allen: Welcome to It Shipped That Way, where we talk to product leaders about the lessons they’ve learned helping build great products and teams. I’m Allen Pike. Today’s interview is with Ian MacKinnon. Ian is cofounder and CTO of Later, where he has helped grow the company from inception to over 200 employees, to acquisition and beyond. Welcome, Ian.

Ian: Thanks for having me.

Allen: Yeah, thanks for joining. I feel like we’ve had some great conversations one on one, and I figured, why not bring that to the world and share some of the various things that you’ve learned doing what you’ve been doing for the last few years? Or many years really, in aggregate.

Ian: And once we’ve done that in five minutes, we can just talk about anything, I take it.

Allen: Then we’ll just talk about the weather. So before we get into it, it’s good to be humble, but you can’t fool me. I know for a fact that you’ve been doing stuff for quite some time, building companies and learning. So how do you like to, or how would you summarize that story, the Ian MacKinnon story so far, in a way, to give people a little bit of context about where you’re coming from?

Ian: I moved out to Vancouver on a whim after school. That was 15 years ago now. I’m definitely one of these people who moved out to Vancouver like, “Well, we’ll see if I’m here for one more year.” Meanwhile, married, family, have a company here, and just one of these people who never left. But I always knew I wanted to do startups, so I was working for a startup when I first worked here. And then I worked for SAP for a little while. And then to really date myself, I was part of the Grow Lab Accelerator class back in 2011. That was the start of my startup journey. Started making a series of progressively less crappy companies until Later came about.

Allen: Practice makes progress.

Ian: Yeah, you have to have a few. And the one before Later was getting pretty good. That one probably could have, but we were making it at the same time as Later, and it really started to go off and really caught the Instagram boom. It’s a social media management tool mostly centered around visual social media, so it worked really well as Instagram was on its ascent. And yeah, we’re up to over 150,000 paying customers now, and we were acquired last year by Mavrck, the influencer marketing company, and I’ve been CTO of the combined entity ever since.

Allen: Cool. That’s always an interesting perspective that we get as founders, as being able to be in those early days and being part of some of those early decisions when you’re just scrappy, you’re just figuring it out. But then we get to live the consequences of those decisions, both good and bad. So I’m curious, what are some of the things that you did early on at Later that paid off, that you were like, “Okay, we did this thing early, and then now we’re glad that we did, looking back at these layers of scale”?

Ian: A couple. First of all is deciding on continuous integration and continuous delivery on day zero. That was the biggest thing. The speed of deployment, there’s just certain things you can do early on, where if you encode them in your continuous integration pipeline, you never have to add them retroactively. A good example of this is linting. Linting is the easiest thing to add on your first day. And I know this having worked on the Mavrck side now where we’re trying to impose a progressively more aggressive style of linting, it just becomes that much harder when you already have this code base that’s grown. It’s something you just don’t think about. But then seven years pass, and all of a sudden, “Oh, we have to go through it,” and then that’s a massive undertaking. Things like static security analysis, did that pretty early on. Constant scanning for security packages, that’s a lot easier now than it used to be, but just doing those very early on. And making sure your deployment is a one-command deployment, I think it’s called one-click deployments. That has always been the greatest speed enhancer for getting product out there. Because if you’re iterating quickly, a deployment to production should be a complete non-event. It should just be boring, even, I run a command, and all of a sudden my main branch is out on production. Doing that on day zero means you never have to have these discussions about how we get to quicker and quicker iterations. You’ve just always had it. Those were definitely some. Having a consistent coding standard. You never think about these even as you’re … Especially if you’re maybe a solo developer, you’re never thinking about coding in a team. But if your product has any degree of success, it’s going to switch to a team. And coding in a team is very different than coding alone. I think a lot of people realize if you’re a cowboy coding on your own for a little while, as soon as you have to bring other people into that little world, your code looks like a frat house of coding horrors. And if you just think about those early on, especially as you start getting customers, you don’t want to over-optimize for it because I think there’s also a balance that way. But having those styles, having a coding standard, running the security scanner, writing tests even when it’s just you. Nobody has ever regretted writing a test suite. And again, it’s another thing that’s very easy to do when you start doing it, but I’ve definitely thanked my previous self for writing tests that … Once you have a good degree of coverage, you can have that confidence where you can do continuous deployment, which again is just the biggest time-saver for an early stage startup.

Allen: Yeah, you talked about the speed of deployment and the speed of iteration, and that’s something that I think has long been thought of as being a valuable thing. It helps us, especially in the early stages, on your search to, “Okay, are we building something that people even want?” type of questions, our lean startup mentality. But one of the things that I find interesting is, most teams now, certainly that are building web applications, which is most teams, most startups, have some nominal CI/CD pipeline. They’re not just YOLOing from some local file system deploy or curling to a server like we used to 20 years ago maybe. One of the things I’m curious about is, as you’ve gone along, how much investment have you made in trying to keep that speed up? Because obviously, or maybe not obviously, as your team grows and your code base grows, the tendency is towards, okay, it gets slower and slower to deploy, the tests take longer to complete. And that whole process, absent other inputs, will slow down.

Ian: I think you do reach this point where if you’re doing full end-to-end testing using Selenium or something, that test we can start to run 20 minutes, and then you’re realizing even the compute costs of running those CI tests. So we definitely do a round of smoke tests that happen every time. But then we have the nightly, test every aspect of the product, hope we can get it before the customers. But I think in our case, a lot of the glitches that are found by that, they’re not as large. Sometimes you’ll find some button might not work, but if it’s very deep in the settings, maybe it’s fine if that can be out for an hour or two. We want to get it before our customers find it, but we’ve always indexed on speed, knowing that there are very rare circumstances where we are going to need to run the full test suite. And a lot of times the full test suite is just for our own sanity, for our own feeling that this is comfortable. If we were putting out a big push, we can do that, but very rarely, if each deploy is small and incremental, you can have some degree of confidence that it’s not going to affect, that if you’ve architected it with any kind of separation between the different components, that altering one component on a completely different page will not just affect an unrelated area. Sometimes that can happen, you want to account for that. But a lot of times you always want to have a bias towards speed. I know the mantra of, “Move fast and break things” is not really in vogue as it used to be. And as I’m learning from dealing with enterprise software now, it does have to work, but in a lot of cases speed is more important than having every little box ticked. And if you’re running your tests regularly, you can usually get away with a bunch of smoke tests, especially on the core functionality. But sometimes things are going to break, and if your customers complain, you say sorry and you keep going. But ultimately customers want features. That’s what they want. They want to see a product that’s expanding the value that they’re getting from it. But as we grow, yeah, you definitely have to start breaking out the test suites. And eventually, I have no doubt at some point we’re going to start saying pretty soon, “Okay, we do a daily deploy where we run the whole test suite.” I figure once we have one, things work until they don’t, and the second we have a big enough outage that was caused by us, that we’re going to start saying, “Okay, daily deploys and we have the full test suite run before that and no other deploys,” but that hasn’t happened to us yet and I’m here for speed until that doesn’t work any more.

Allen: No, I think that tends to be, as much as you say, “Move fast, break things” has become slightly less popular as a mantra, personally just through association with Facebook and some of their problems that they’ve had breaking things that are fairly important, like society and things like that. But when we’re at-

Ian: Yeah, I’m not talking about breaking democracy, don’t move fast and break democracy.

Allen: Don’t break democracy.

Ian: If you have to break a button on a settings tab, that can happen sometimes.

Allen: As long as you detect it quickly. And I think one of these things that often comes up in these conversations is, you’ll have two people who’ve come from different perspectives. Maybe someone’s from enterprise software where it’s deployed to 100,000 people or 10 million people or 100 million people, and they are really concerned about it exactly working because they don’t want to get sued by the Swiss government or something. And then you have another team that has like eight people trying to figure out, “Okay, are we going to break even before we die?” And this right setting for moving fast and the cost of breaking things is very different in those two circumstances, which is obvious and everyone knows that. But it’s I think a lot more helpful to talk in terms of these trade-offs, just as you were, where you’re like, “Hey, early on moving fast and iterating fast and being able to learn fast was so valuable that occasionally if a button broke that was a net win, or if not all of our tests ran, that’s a win.” And then eventually we’ll get to the scale and we will learn by trial and error that it’s like, “Okay, we need to lock this down, and yeah, we’re going to iterate a little bit slower.” But if you get to the point where you’ve got hundreds of employees and you’ve been acquired and all these other success metrics, then it’s like, “Oh, well, if you slow down a little bit at that point, I think we’ll forgive you.”

Ian: Yeah, and that is the natural consequence. And it also depends the environment you’re in. Fortunately, I work in social media, which is not a regulated environment. Our data security is extremely important, but that is an operational concern. Very rarely are you pushing anything that affects security. So security processes and reviews are extremely important. But when I say move fast and break things, all too often we’re talking about the user interface. And you shouldn’t be trusting your clients anyway.

Allen: Yeah, and I think that, I assume, you haven’t mentioned it yet, you mentioned, okay, you break something and people complain, but I assume, or maybe talk a little bit about what kind of approaches you take for detecting that something is broken before people send customer support complaints. I assume there’s some sort of A/B testing, or are you at the scale that you have rollback and partial rollout of things? Yeah.

Ian: I’m a big fan of the rollback, but every time something rolls back, you should have a moment when you realize, how did this get through? How was this not tested properly? A big way that we test things is using pull request apps. Nothing has really increased the speed and confidence in our deploys as being able to have a completely separated detached front end and test whatever new feature we’re working on before it’s even merged into main branch and deployed. That has just been a great … originally we were using GitHub or Heroku, they have a great pull request system. We’ve since started using AWS Amplitude, which does something similar. But if you have a fully detached single page web application, you can treat it almost like a mobile app, where you’re pointing it to any server you need to. On a branch you can be doing all your testing and then pull it back in, and you can have that confidence for when you’re doing those deploys that everything is going to work.

Allen: I’m a little biased because I’ve spent a lot of time working on mobile apps and that way of thinking about things.

Ian: Yeah, originally the Later mobile app was the most important thing. It’s hard to believe, but I was an iOS developer back in the day. And as soon as we hired one, I unfortunately got to stop it. And Swift was just coming out right around the time that I was starting to do less and less. So I feel like I’ve lost that skill, but when you start thinking of your web page as a client akin to iOS and Android, it really forces you to think about your API design in a fair way. Because you now have these three clients who all need to access your back end if you want to be a true platform.

Allen: Yeah, it makes a lot of sense. So we’re talking about things that you did early on that paid off. Maybe you had a perfect crystal ball and you made all perfect decisions early on. But I’m curious, is there any things early on that in retrospect you’re like, “Hmm, there’s a lesson”?

Ian: It’s funny, it’s not a big deal, but one of the ones that always I’m reminded of constantly is whenever we name the API endpoints incorrectly. Because we’d be working on some feature where we didn’t have the marketing name set up correctly, so we would name something, and then we’d create the endpoints, it gets renamed something differently. And now the endpoints, if it’s just something cosmetic, you don’t ever really want to go back and change them, because by the time all your mobile apps are updated to use the new API, it can be a year. People love not updating their mobile apps, and you just have these clients that are out of date for years on end. And then you’re just like, “Ah, is it not really worth it?” So having a good naming structure and sticking to it and keeping your restful names in place, those are all things that are really good to have. Documentation, having a documentation for your API, super important. We were a little late to the game on that one. Another big one is, if you’re charging for your software and you’re using something like Stripe, I guarantee you one year later someone’s going to ask, “Hey, can we integrate PayPal and all these other?” And just having the flexibility for your payment provider is something somebody is going to ask you. Some European is not going to be able to use Stripe or their credit card’s not going to work or they’re going to want to use PayPal, they’re going to want to use invoicing. And if you architect for that earlier on, it makes it a lot easier than having to go back into a system that’s already running. And this is the money system. So speaking of things you don’t want to break things, the thing that gets you the money in the door is not something that you want to experiment too aggressively with. So that’s definitely one where I wish we had that ability. The big one I can think of early on is, we were using Heroku for deployments. We very famously started at a hackathon. Heroku was great for all those things. It was simple, it was fast, easy deploys. But once your database is in there, it’s a bit of a pain to get out. And the cost of Heroku and a lot of these platform as a service providers, sure, they charge you a little bit on top of what Amazon would, but you can think to yourself, “Well, if I don’t have to have a dev ops person, that’s going to save me so much anyway, so that’s really what we’re paying.” But eventually you’re going to get to a scale where the delta between AWS and your platform as a service provider is greater than what a dev ops person would be. And then you start thinking about how you move, and moving databases is no small feat. And I think that’s the hardest part. When we had to move off of Heroku to AWS, because eventually you’re going to hit there for security reasons, where people send you these security questionnaires and you start needing to have the right answer for things and you need to restrict access in a way that would be pretty expensive on these platform as a service providers, that moving to Amazon, you might not be able to use all the easy tools. And sometimes they don’t want to make it particularly easy to do that for you. That’s your lock-in, and something you don’t think about early on. But moving data, especially when you’re talking about terabytes of it, that can take time, and how much downtime are you willing to have? We can have downtime, by the nature of our product, 20 minutes of downtime, everyone schedules their content on the hour or on the half hour, so about five minutes after the hour we have a nice 20-minute window before anyone’s going to notice that any content hasn’t gone out. So we’re big fans of 20-minute downtimes on Saturday night if we need it, but you’ve got to think about what you can actually get done in that time and planning for the future when it comes to capacity and, is this provider we have now that’s being so great and so fast going to be something that we can move off of later on?

Allen: Yeah, that’s an interesting point, because it goes back to this question, when you’re early on how much advantage you get. Because you can move faster and you have this speed and you have this ability to iterate really rapidly, and you’re saying how valuable it was to have Heroku’s preview deploys and all of that kind of thing. And one of the things that we often hear about is, and I’ve intentionally caused this in my line of questioning, but the retrospective survivor bias is like, “Okay, well you did all this experimentation and then it worked and it was a hit,” and then now you’re at this scale where you’re like, “Oh, great, now we can hire dev ops people.” From your point of view, which is I guess probably informed by having worked on other startups that didn’t have that success, do you feel like going straight to AWS and having that extra dev ops either cost or burden on the existing team may have worked out net better given the cost of potentially taking longer to figure out that you were even on the right path?

Ian: It probably wouldn’t have. Again, speed is going to matter. I think it was more that we didn’t have a plan. I can over-optimize for for having a dev ops person. The reality was, I think as an early stage founder you have to realize you are the dev ops person. Honestly, the switch away from it wasn’t massive, it was just we didn’t have a plan for it and we kept getting scared by it every year when our Heroku came up for renewal. It’s like, “Oh, if we’re going to move, we’ve got to do it now.” And it was always this time pressure thing and it was very intense. And it worked fine, there was no problem with it, but there was a cost involved for hiring a specialist to help us. I got to not think about our infrastructure for a long time, and probably my only actual regret is not knowing enough of that dev ops stuff myself. The problem is that if you hire a dev ops person too early, sometimes people have a habit of making work for themselves. And I feel that same way about a software architect. As soon as you hire a software architect, you are going to get more architectures than you already have whether you need them or not, just because that person now has to justify their existence. And one can make a similar argument for dev ops, that if you have that too early, you might be optimizing for the wrong thing. But fortunately we found that we were onto something pretty quick, about the three-year mark things really started to go as we hit the wave, the big growth wave, especially through early COVID. I think that was more just about my knowledge, but we always wish we knew more when we started things, don’t we?

Allen: Yeah. And as long as we take the lesson that, next time go in with the fast tool, but be aware that you’re going to have to switch away from it, I think that’s a great takeaway rather than the …

Ian: I think if we had a plan it would have been easier, I mean the step-by-step, but in that case we didn’t really have the plan, so it always became this intimidating thing about, “Oh, this year we’re going to switch over and try and save a couple grand a year on hosting costs.”

Allen: Yeah, good lessons there. Although it sounded like maybe perhaps again we have our survivorship bias asking the successful company, “Okay, what were the mistakes?” You’re like, “I don’t know, I think we mostly did it right. Here’s some moderate to small problems.” But Later was not your first startup, you were technical cofounder previously, you mentioned a couple times, slowly improving each time. And obviously luck and timing is always a factor when you’re starting something. What non-luck contributors in your mind played into those previous efforts not playing out, compared to how Later worked out?

Ian: So with my first startup, it was called Placeling, it was a local social mobile, which was all the rage back in 2011.

Allen: Local social mobile, a number of times I heard that phrase.

Ian: Because this is when iPhones are really starting to pick up and GPS, these are the days of Foursquare. And we were trying to make something that was a more privacy friendly version of that. It was going to exist between Foursquare and Yelp. My co-founder and I, we had exactly the right pedigrees. I was a Waterloo computer science grad who worked at Google and he was an NCAD MBA grad who had worked at McKinsey … I mean the Boston Consulting Group. I can’t get those management consultancies mixed up. So on paper we looked really good on any kind of a pitch deck for that reason, but ultimately we were making something that there just wasn’t a lot of demand for. And I think what we had a problem with was getting distribution, and did people want this badly enough that they were going to install the app and use it daily? And turns out the answer was no. We learned a lot through doing it, but we were jumping on a hype train. We got funding because of it, and we figured people who were better at this than us were giving us money so there must be something there. But the people who get it right, it’s kind of like baseball players, if they get it right one time in three, they’re in the Hall of Fame. And most don’t even do that. So you have to just think about, you’re a singular at bat in that person’s career. So it was a cool product, it was a nice looking product, but just people don’t think about apps the same way … The number of apps that people use regularly is smaller than I think most people think, unless they have a specific utility in mind. I think you and I probably talked about this, where I don’t know how many parking apps are on my phone, but I know it’s about half of them right now because there’s no ubiquity among any of them. But I don’t think about them, but I only use them exactly when I have a specific need for them. But if I’m trying to bookmark places around a city I like and see my friends, what they’re into, some would play with it, but there just wasn’t enough interest, especially when you had the dominant players like Yelp and Foursquare taking up all the oxygen in the space. I can tell you about the technical mistakes I made on that one, but they were immaterial compared to just making something that no one had a particular interest in.

Allen: Yeah, I think that’s a really useful perspective that a lot of people, I think maybe we’ve grown up a bit as an industry over the last 10 years, but certainly at that time I was fielding many pitches from people with ideas for social local, global mobile startup ideas. And, “Oh, okay, well, this is a thing that you’ll get ads for the places, the push notified ads as you walk past coffee shops,” and all these various things that no one was asking for, but it’s like, “Well, they could be made.”

Ian: It could be made. Yeah, because what I want more of, push notifications.

Allen: Yes, I want more push notified ads. That’s what everyone was clamoring for. And I think one of the things that you highlighted on there is that yeah, of course we’ll make mistakes in building those things, but at the end of the day, what success was, if you set out with a premise that is sort of “wrong,” you had a business theory, which is probably more attributable to your Boston Consulting cofounder than to yourself, but you had did this theory, was, “Okay, people want this thing.” And success of the startup’s execution from that point is really, how fast do you figure out that people don’t in fact want that thing? It’s really, how much time and money do you spend figuring that out and how efficiently, effectively? Rather than, “Okay, how did we lay out our AWS versus Heroku decision matrix?”

Ian: Oh, yeah. In hindsight it was a very cheap mistake, because we only worked on this for about a year and a half and we only burned about I think $120,000 of investor money, which when you’re in your 20s you think, “Oh my gosh, that is so much.” And now as I’ve seen how investment companies and portfolios work, you realize, “Oh, that’s nothing for them. They gave us that amount of money on the off chance we were successful so they’d be front of line to give us a real amount of money later on.” How we were just a position, arguably a hedge in case this whole area took off. And once you realize the dynamics and the economics of how venture capital works, you realize, “Oh, we were a cheap one.” We probably could have pivoted to something else. I don’t know what it would have been in that moment, what we could pivot to that was close to that area. And it just made more sense to, all right, throw in the towel after a year and a half. We hadn’t even hired any employees. The cost to shut it down was basically nothing. That was a cheap lesson. But the problem was, we didn’t even really talk to our customers. We had so few signups it was hard to really do any kind of user research to understand. And all of our user research was like, you talked to your friends and they said, “Oh, this is a cool app, it’s so good. I will definitely use it.” And then they absolutely do not. As I always say, your friends are the worst source of feedback for your ideas because they’re going to tell you something’s good even if it’s really not, because nobody wants to crap on your dreams. No one wants to be the one who told you, even if they think that it’s a good idea. And as I say, this is why I love the show Dragon’s Den. It is just a parade of people who’ve been lied to by their friends and family about their crappy ideas, and the first honest person they meet is Kevin O’Leary. And that should not be the first honest … Say what you will, he’s honest, so he’s going to tell you what he thinks about whatever crappy idea you have.

Allen: Yeah, this goes back to the, which was I guess popular around the time that you were doing the startup now a decade ago, of the lean startup idea, which is this book, which initially I think shook up a lot of people’s thinking about how to build startups. And then it started to become, as many things do, it gets popular and then you get backlash against people saying, “Oh, I don’t want to build a minimal product. I want to build something that doesn’t suck.” And whatever, it came a bit of a meme. But I think in terms of mentality, trying to figure out, how can we be talking to customers and getting true, honest, brutal signal as fast as possible, as cheaply as possible?” is something that could have been really useful in our previous wave of now dying startups that got funding in 2021 and had burn rates, weekly burn rates of the rate that your one-year burn rate was of this startup that you were able to figure out quite cheaply that your business thesis was one worth putting in the parking lot, so to speak.

Ian: Yeah, I think the big one you can take away from that that will survive the test of time absolutely is, fall in love with a problem rather than your specific solution. Because all too often, any startup I mentor, I tell them, “Don’t get too attached to your solution, because your solution is meant to solve someone’s problem. That’s what people care about, is solving the problem that they have.” And really the lean startup is just a method in order to do that. And you can always try and get engagement by dunking on whatever the latest is. But if you take the principles in that book, it really is pretty good at, what’s the cheapest way of determining whether you want something? Is it to build a fully fleshed-out solution or is it something a lot smaller? When we were launching the first versions of Later, we didn’t have a fully formed app. We had a landing page, and one of the things that we noticed was at the time the search volume for “Schedule Instagram post” was very high, but the competition for AdWords and SEO was basically non-existent. So you can arbitrage Google to find out where there’s opportunity. And that was something that we never did with my first startup, where we thought about, “How are people going to find this?” Because one of the things you realize, that word of mouth does not work as well as you think and people just don’t sit around talking about apps that much, unless they’re dating apps. If there’s sex involved, people will talk about apps, but they otherwise won’t. It’s not like when I’m meeting with my friends I’d be like … But this also cuts both ways. The good news is, if you launch a stinker on the app store, people don’t spend their time going, “You would not believe this piece of crap that I downloaded.” No one’s going to know.

Allen: Out of sight, out of mind.

Ian: Yeah, your app was forgotten, and that is sometimes a good thing. It should give you some idea that there aren’t going to be people looking at your app hunting you down personally and memorizing your name and putting it on a list for having wasted their time for downloading a bad app. They’re going to forget about it in two seconds. So have some knowledge that it’s okay to launch a stinker. We’ve all been there.

Allen: I love it. And I think that’s one of those messages to pass me. I had my string of, “Build a really nice, super polished thing that no one wanted,” and did that a couple times in my career.

Ian: Yeah. At least they were really beautiful turds that you launched, maybe.

Allen: Thank you. I think they were, they got five stars, all eight people who downloaded them. So switching gears a little bit, thinking with a CTO hat now, going back from this founding era to the, now there’s hundreds of people that are depending on you to be leading in a good direction. A common CTO task that generates angst on a lot of teams and under a lot of folks’ radars is picking a tech stack for our new product or feature or some part, component. How do you think about those kind of choices, reflecting on the lessons and successes and maybe partial failures that have come out of the choices that you’ve made there?

Ian: Sometimes I liken picking a tech stack to getting a tattoo.

Allen: Hm.

Ian: Because in theory you can get them removed and changed, but it’s a bit of a process, and you probably don’t want to get a tattoo just because of what’s trendy, because that might not hold up over the test of time. So whatever the latest and trendy tech stack is, you have to think about, “Would I want this logo tattooed on me? How confident I am that this will stand the test of time?” Because this really will start to matter. And the key thing to remember is, your customers do not care about your tech stack. No one is going to ever ask you that question. It’s hidden from them. And there’s a lot to be said about boring tech stacks. We are a Ruby on Rails shop. I love Ruby on Rails, I stand by it, but Mavrck was on the Node side. And there’s nothing wrong with Node, but one of the things I’ve always liked about Rails is it gives you those guardrails. So every Rails app kind of looks very similar, whereas a Node app, you can really have to have a real firm set of coding guidelines for how we structure things, how we architect things. If there isn’t agreement on that, people can start going. And as I say, sometimes if you’re picking ORMs, for example, on Node, the worst, if you’re picking between A or B, some mix of A and B is always worse than the worst of A or B. And I think all too often we have trouble as developers saying no to people or making hard decisions, or kicking the can down the road. But this is one of those things early on where you have to pick an ORM, you have to pick it. And I think there’s this worry that, “Oh, what if we have to change these things?” I’m like, “Those happen so infrequently.” It’s a big undertaking no matter what you do. So you have to think about what you want tattooed on you. Because that’s effectively what you’re doing to your code base, and in theory you can switch, but switching out a tech stack means you’re spending a lot of time not working on features, you’re not working on improvements, and there’s nothing wrong with getting something kind of boring. However, on the flip side, you do have to think about, “How are we going to recruit people who work on this?”

Allen: Yeah.

Ian: If you pick something obscure and steady, are we going to get that? If you’re going to develop on the .NET platform, that’s absolutely okay, but you have to know that’s a special kind of developer we’re going to get. There’s not a lot of people who are flipping between necessarily .NET or Java. And people do, but you’re going to get people who like the frameworks that they’re into. So there’s nothing wrong with going Node, you just have to pick something. And there’s nothing wrong with .NET, but just know when you’re picking that, there’s certain cultures that are around those frameworks. But think about what’s going to work. And sometimes the best thing you can do is look at Hacker News posts from five years ago to see what people were thinking and how wrong it is and how there’s not really the same amount of consensus there. Because I feel like that’s full with a lot of people who have two years’ experience talking way beyond their years.

Allen: Who aren’t yet busy enough with their job that they’re still spending all their time writing posts on Hacker News.

Ian: I haven’t been on Hacker News in years. It’s still a good source, but it’s such a distraction. And people are way too confident on those posts. In real life, most developers should not have that level of confidence about anything.

Allen: Well, Hacker News has the same problem that GPT does, which is that humans will tend to up upvote confidence. So if you have a system that is reinforced by upvotes, then you’re going to probably see a lot of confidence, and that’s hard to fight against.

Ian: Also, we used Ember.js on the front end. We made that decision a long time ago before React had even launched. And obviously React is the clear winner of the front end frameworks, but I still stand by Ember.js, even if we always have to train every front end developer on that. For an app of our scale that big, again, the guardrails that it gives ensures that everyone is on the same page for every piece of the application, especially when you get one as expansive as our front end app. The only times I’ve ever thought about moving off of or changing our front end or backend would be in response to pressure from, “Oh, can we recruit that?” But often it’s just easier to train people on whatever you’re working, if have a culture of upleveling people who are coming in. But again, there’s nothing wrong with these. And just going with what’s ever trendy can really get you into some bad situations.

Allen: Yeah, it’s interesting that you say that, because you don’t want to pick something that’s trendy and you don’t want to pick something that tends to fall out of favor. But then you also gave almost what feels to me as counter example. Maybe I don’t have a complete view on this, but to me, I feel like Ember.js was trendy around the time that you were starting and now it’s not, and you’re kind of saying, but that was fine too?

Ian: Yeah, I’m saying it was fine because a lot of big players are still using it. LinkedIn is, the Apple Music Store is still, and it still has a large amount of support. It didn’t win the hype cycle, but if you look at the merits of that framework on its own and what it does, I have a theory that all these front end frameworks are converging into one thing anyway. In a long enough period of time they’re all going to look more similar, with the right decorators anyway. The real question is, if I had the time machine to go back, would we still have made the same decision? That’s a tough question. Knowing what I know now I can definitely make arguments either way, but the real question is, will we re-platform off of it? And in that case, no, it’s just too big. It’s fully supported. It has a community behind it, and it solves all the problems that we have. I didn’t think it was trendy. It did look like it had some momentum, but you’re going to get these curve balls like Facebook dropping React in 2014, 2015 after we had made that decision. And at the time it looked like it was one. The competitor was Angular and Backbone at the time. Those were the three that were duking it out. And if you look at it, you’re like, “Well, LinkedIn, Apple are using this.” There was enough big players that it definitely had that feeling. Where I think they lost some of the momentum is that the framework started insisting upon itself, where you had to go single page app, all in, whereas React, it was a little easier to do a little bit of React. You’d dip your toe in, you didn’t have to make this full commitment to it on day one. So that was where I think it got a lot more momentum, and it’s absolutely hard to predict, but what I would get at is, if you don’t know why you’re using Rust or something, that’s probably not the right solution for whatever consumer app you’re making necessarily. And I think the more app right now is microservices. I think a lot of early stage startups are so fixated on breaking up their monolith into microservices and they don’t really know why. And the key thing there, just make sure that you’re not looking at …

Allen: Scaling, and scaling.

Ian: Yeah, scaling.

Allen: Hand waving.

Ian: Scaling the app that may or may not actually get 10 users. I think with microservices, I think people really underestimate the operational overhead. Now instead of having one repo where you have to keep its security packages up to date and all these standards, you might have now 10, and for what? So all these individual parts can scale individually? Once you have that problem, by all means. But until you have that problem, that’s a very high gear. If you think of a bike, that’s a different gear setting than when you’re getting started, where speed and need to get going.

Allen: I’ve heard an argument that microservices, the hype, at least from my perspective, seems to have been going down in the last four years, which is great. There was definitely a fever pitch where everything was, “Everything should be microservices from the first line of code,” and that has backed off a lot. But something that I heard, an argument that I found useful, mental model that feels truthy to me, is that microservices are a way not to scale your user base, but a way to scale your organization. And so there’s a certain head count where it starts being better, especially once you get past 150 people and you have 150 engineers, and having teams of engineers coordinate with each other is going to be difficult anyway. Then having them coordinate on separate services starts to become more and more compelling. But whether or not that’s serving a billion users or 10 really difficult users, is more agnostic to whether or not microservices are going to be super helpful.

Ian: Oh, yeah. I think I heard something similar, where you have to think of microservices solving an organizational problem, not a technology one. And where that does make sense is, as we were talking about earlier, if my test suite is now taking 20 minutes to pass for a specific component, it’s getting big, well, splitting it up might make some degree of sense if we have the support to have that. But if you only have a 20-person engineering team and you’re all committing to it, you still might not have that problem. I think the idea is that you want it to be something that you undertake not just because people want it on their resumes.

Allen: Yeah.

Ian: Resume driven development is something that we constantly have to be fighting against.

Allen: It might be helpful for recruiting, but then you have to deal with all of the maintenance that you’ve accumulated. So one thing I want to touch on before we run out time is, I know you’ve been making some use of large language models and GPT4 in particular to deal with some of the stuff we’ve been talking about, which is, we accumulate tech debt. Some people might use that term, or maybe even just legacy code or whatever it is, code that’s been written over five, 10, more years, different teams with different technologies, and that one thing that got written in Rust and et cetera, using GPT4 to help migrate that off to more modern conventions and approaches and things that are more in line with how you’re doing engineering practices today. How has that gone for you, and what have you learned with experimenting with that as a tool?

Ian: So one of the things that we’ve been trying to do to modernize the code base on the influencer marketing side, which is in Node, is things like upgrading it to TypeScript. In a lot of cases we’re using those promise libraries like Bluebird and Async, which look very anachronistic at this point because the control flow gets very hard to follow, and we just want to update maybe not even anything that’s necessarily wrong, but it’s using a library that’s out of date or just some paradigm that’s fallen by the wayside in some chunk of code that’s not particularly important. What we’ve been experimenting with is using GPT4 to rewrite some of our code, especially en masse. By which I mean, you have to understand, and I’m sure you’re aware, but transformers were originally designed for language to language translation, like human language, English to French and French to German. That was their original purpose. And what we found is that they will do that for computer code quite easily, and GPT4 is quite good at it. When ChatGPT came out, I wasn’t particularly worried for my job as a software developer. But when GPT4 came out, I started to, “Oh, okay, this one’s not going to, but I’m starting to feel what a copywriter might in terms of what this can actually get done.” And what we’re finding is that if you put some effort into prompting and you treat GPT4 like a junior developer in terms of how you talk to it, and better yet, if you can give an example of, “Here’s a file before and here’s after. Now go do that same process to this other file,” wrap that in a Python script, run it on everything, and you can do some pretty remarkable lift and shifts for updating things to TypeScript. And it might not get it 100% of the way, but you can take out a lot of the monotony that you’re going to get whenever you’re doing refactoring and upgrading. And I think developers are naturally disinclined to boring work, and a lot of those maintenance tasks fall under that. However, if you give them the shiny new tool and say, “Well, don’t do it, train GPT4 to do it by using longer prompt windows,” you find a lot more motivation among your staff to use cool tools to do relatively boring tasks. For an example, one of the things that we were doing was we had some SQL fixtures for setting up integration tasks, which were extremely brittle, and we wanted to move those into factories. I was impressed with how well GPT4 could take an SQL file and rewrite it as a Node JS factory for proper testing. So it just will do these translations that the copy/paste or cut and paste for a developer would be monotonous and take so much time, whereas 10 cents’ worth of GPT4 can do a full file and with a lot fewer errors than you’d get copy/pasting things.

Allen: Yeah, I love that so much. And when you were explaining this to me, I got really excited when you got to the point where you were saying, “Oh, okay, well, often it can be difficult to get skilled folks to care about this, migrating old code.” But now I have some of my best people that are like, “Yeah, I’m going to totally push the envelope on what we can do in terms of using,” not ChatGPT, but ChatGPT4 with APIs and Python as you described, and getting better and better at translating more and more of this code. Obviously you could assign a bunch of junior engineers to comb through tens or hundreds of thousands of lines of code, and GPT4 will make mistakes, but those junior engineers will probably make more mistakes, it seems like, even a higher volume, and certainly much more slowly than GPT4 would. So I find that really fascinating.

Ian: Yeah. So in terms of what we found GPT4 is good for, definitely for writing tests. If you give an example of a test, especially for something very similar, like a request test where a lot of the setup and teardown is very similar, or if you just give it a file and just takes a lot of the boilerplate out. And what we have in our actual code base is that we have a committed directory of our go-to templates. So if any developer needs to update, “Oh, if you have a file that’s using the Bluebird library and we want to move that to Async 08, here’s our go-to template for reusing that,” and it’s committed right into source code. So everyone can still have access to the latest and greatest template that they can go copy/paste over to the open AI playground, run through. Because I think what a lot of people miss about writing these prompts is just how complicated the GPT4 ones can get, and how if you give it an input and an output example, it’s really good at then applying whatever transformation that you have to some other new input. And you can give it a lot more context than you really might think, because these prompts can get pretty long. So make use of it and reuse them. But in true software style, well, if we have this process, it should be right there next to your CI configuration directory, where, “Hey, here are our go-to prompts for performing all of these actions.” Just keep those prompts right in the source control so everyone knows exactly where they are and what we’ve done. And if you have to add some other prompt in there, it becomes a set of meta tools to improve your code at a pretty rapid clip. And we’ve mentioned there are cases where you want to pipe them all into a Python script and run it on 300 files if you have one change you want to do en masse. What I’m really looking forward to is whenever we have a package upgrade, such as Ember five coming out, and there’s going to be a few changes that we want to do for that, is having our go-to, every major framework update is going to have a few changes, but they’ll in their documentation give you examples of how to do that. You can just copy and paste that documentation right into your prompt to help for how you do these migrations just to help you. I really think GPT4 is really underutilized by developers for how much it will actually do in terms of pulling out some monotony in terms of the kind of work that you’re doing.

Allen: Yeah, I love that, because that’s a great use of, I think, of thinking about this not as a tool that’s like, “Oh, it’s going to take people’s jobs,” but it actually makes your job better, and it can make it so that we do things that we maybe would then procrastinate. Like, migrating this old thing maybe just wouldn’t happen until we are forced to do it, as opposed to, now you have people happily taking little bits of time here and there and making the code base better in a way that is engaging for the developers rather than mind-numbing. So I think that’s really awesome. So it’s been wonderful talking to you, Ian. Thanks for the time. How can people go to learn more about you and your work?

Ian: So the two companies I’m CTO of right now are Later.com for the social media management and Mavrck.co, M-A-V-R-C-K, they didn’t get as many vowels as they would’ve liked, .co. You can also find me on the Twitters as long as it lasts at @IMacKinn, I-M-A-C-K-I-N-N, and on the threads at @Imac.

Allen: Awesome. Well, thanks for being on the show. It Shipped That Way is brought to you by Steamclock software. If you’re growing business and your customers need a really nice mobile app, get in touch with Steamclock. That’s it for today. You can give us feedback or rate the show by going to itshipped.fm/contact. Find us all in the various social medias. Until next time, keep shipping.