How to Run a Remote Big Tech — with Farhan Thawar (Shopify)
Two people working in high fidelity via pair programming, reduces distractions, increases knowledge transfer, removes the siloing of information, less hacking happens, more forward progress happens even though it's less code, actually specifically because it's less code. And all of those things do come at a cost. And the research shows the cost is 15%, one five, which is actually not what you'd expect, not like 100%, not 50%, like 15%.
And what you get in exchange is like less errors, happier engineers, more learning, faster overall throughput and development. Today's guest is Farhan Thawar, VP of Engineering at Shopify and angel investor. With Farhan, we have the unique opportunity of diving into the inner workings of a large remote-first engineering organization.
So we talked about remote versus office culture, asynchronous communication, pair programming, how technical should managers be, and of course, AI. So let's dive into it after a short word from our sponsors. Hey Farhan, thank you so much for being on the show. Nice to be here. Thanks for having me, Luca. So I'm excited to have this chat because in my opinion, and of course I'm sure you agree, Shopify is quite a special company because it's a big tech.
I mean, maybe not at the scale of Google or Meta, but public company, thousands of employees. But on many levels, it operates differently from other big tech corporations. And for example, one of these big ways is you stuck with being a remote first. or I would say digital first in your words, company, as opposed to others that in this moment of time, I would say are more about pushing return to office policies are pedaling backwards. So I'm curious about your thoughts about this. Do you think remote is the future of engineering work and how is it working for you? Yeah, so yeah, to firstly, to start off, I think it's funny that you said Shopify is different.
That's actually one of the things that I think about when I explain to other people what Shopify is. I've worked at many different places, including starting my own company, being an early employee in a bunch of companies. And I would say the Venn diagram of like everywhere else and Shopify has a little bit of overlap, but not a lot. Because we do try to reason from first principles about a lot of things.
And... That's why I kind of, I think I tweeted out one day, like a little bit of a diagram saying like, hey, the things you learn everywhere and then things you'll learn at Shopify, because it is quite different. So let me tell you about how we got into remote work. So we were an IRL company, right? In real life, we had this notion of different offices though, because we were large enough to have outgrown one city. So we had offices in Toronto, Montreal, Ottawa, New York, Waterloo, we were about to open one in Vancouver. So we had a lot of offices and the idea was, kind of got to live near an office.
And of course, sorry, I just talked about North America, but we also, you know, in Europe and in Asia. And then of course, when that pandemic hit, everyone went remote. And what we wanted to do was remove the ambiguity and lean into the future.
And so by removing the ambiguity, you kind of saw lots of companies where even the CEO was still using like an ironing board to like type on their computer and they were still half embracing it saying, we're gonna be back any week now. But of course it was not any week. And so we just said, why don't we remove the ambiguity? We know the future is coming, but it's gonna be smart people who live everywhere.
And we embraced it. So actually pandemic for us was March, started in March. here in North America, and then by May, we had made the decision we're going all remote. - Wow, that's fast! - And we weren't going to... Yeah, it was fast, and we said we're not going to turn back. So what that meant was, okay...
not gonna be remote anymore, we're not gonna be IRL anymore every day, what does that mean? And so what it meant for people was one, get your remote set up ready, right? Get a good camera, good internet, desk, chair, like no longer should you be working on an ironing board, like figure out your remote lifestyle, that's one. And we helped with that, right? We gave people budget for things, we gave them a desk, like this chair and desk and everything came from Shopify. So that was one. And then two, We have to figure out eventually when the pandemic was over, what does it mean for IRL rituals, right? In real life.
Cause remote doesn't, no one, so when I see these postings, right, 100% remote, right? They do exist. But a lot of times when people are thinking about it, and what I like to think about is like 90% remote, or 95% remote. And that's how we think about it at Shopify, where it's not, we are 100% remote, you're never gonna meet with people IRL. We've specifically designated, ways in which you can get together with your team. So here's a few examples.
One is we still have some of those physical offices. We call them ports now. They're not offices.
And the ports are there to use them if you want. So that could mean I'm in Toronto. I might wanna work from the office a few days a week.
I can. That might mean someone's traveling into town and I wanna meet with them. It could mean maybe I'm going to, I know that there are some people I wanna work with in New York. I wanna go there. There are ways in which we have a place for you if you are traveling or wanna get together with our team.
So that's just using ports. We have something else called bursts, which are specifically designated times when you get together with your team around a certain project you wanna work on, you wanna get together to, whether, one example is actually we just had this was hack days, right? So, We have these rituals, so bursts or something you get together, we call it GSD, getting things Hack days are ways in which we can encourage people to come in the office, work together as well. But I think the idea is not only to unlock the power of having amazing people working at your company from anywhere in the world across time zones, figuring out how to work more async, but at the same time figuring out when you can be sync. So can we have a meeting during Eastern time zone? Can we get together once in a while to burst? Should we get together for hack days? So that's on the IRL side. The secondary side is, How do we make remote amazing? So by that, great machines, great environment, lots of different ways to communicate in the communication tool chain, pair programming. How do you do that remote? What tools should you use? How often should you do it? So all of those things are how we try to think about the entire end-to-end employee experience around remote.
And I think, of course, the biggest unlock which everyone talks about is hire amazing people wherever they are. Yeah, that's fantastic already. So as far as I understand, you have kept this option of making people meet in physical places for bonding, for creating stronger relationships and for maybe specific situations and activities where you need that high bandwidth or of being everyone in the same place. Yeah, so we basically try to encourage you to meet up with your team a few times a year, like IRL. And then if there's a particular burst happening around a project that you should be a part of or a re-imagination of an area that you should be a part of, we'll do that. And then like I mentioned, if you live near a port, you can use it as much or as little as you want.
Like there are people who live right near the port, don't use it. There are people who live far who come in every day, right? Like you can kind of use it at your discretion. And then of course, traveling. You can use the ports I always do. I end up working out of the port in any city that I'm in because one, it allows me to get that, you know, random serendipitous interaction with folks. But most of the time, like today, you see, I'm at my house, right? I work at home.
Yeah. And aside from designating this physical versus virtual places, ports, etc., are there things for which your workflows and processes have changed because of the switch to being remote first? Yep, I think there is so Shopify already had a pretty good writing culture. And so we write a lot down. And the reason for that was we already were multi-office, right? So by when you're multi-office, you're already, I mean, once you pass one floor, you have to then figure out a workflow in which works, what everybody, where they can share information. So I think that was already good.
I think that what came from COVID was a much more of an async video culture. Like what came from COVID and remote where We had town halls and we had a lot more synchronous video, I would say, where it was timed. Come here at four o'clock on a Thursday. And I think what changed was a lot more asynchronous video, a lot more demo culture where we wanna show you and not just tell you something. So here's how my experience of using this new feature is going, right? We sometimes call that a friction log.
Let me share with you how this worked for me. Maybe there's a lot of friction, maybe there isn't a lot of friction, but it's you showing an example of you going through the new flow. We do a lot more, Of course, there's still times when we all get together and we have synchronous events like town hall or a summit, or even what we talk about additions when we launch those things, but they can also be consumed async, right? So it's easy, it's very easy to get into this mode of, hey, I'm gonna work on, I have a block of flow time, I'm not gonna interrupt my flow, I'm gonna spend my time actually working, and then instead I'll consume this video asynchronously. And so async video really took off. Yeah, I think asynchronous video is one of those things that really, when you think about it, it had the potential to be there, you know, even before the whole switch to remote, but it really took off. I hear many stories since this whole switch.
And... And by the way, not only for the high fidelity, let me sit down and record a video, but also for, I see, I get a lot of voice notes. I get a lot of videos saying, hey, we don't have time to catch up today, but let me send you my thoughts and it'll be a three minute video or a three minute voice note. So conversation happening in a different medium has also increased during remote because you still wanna get across a lot of information, but you don't wanna type it out. Or you don't want it to be received as just typing, and so you send it as audio or video instead. Yeah, totally.
And so you are minimizing the synchronous part instead, so that the actual meetings and moments where you have to be together. Yeah, you're saying the word minimize, I wanna like maybe be clear. So we have a few things that we do to try to make sure people have flow time. So one is we have limited, so you know, you can imagine a town hall or a big, engineering here is almost 3000 people. So we have big groups. So imagine I said, I'm gonna do, an engineering town hall on like Monday at 9 a.m.
Right, like that could break somebody's flow because maybe that's when they write like to get into work. Or then the next time I do it, I do it on Wednesday at 3 p.m. Like it kind of just is all over the place.
So we actually designated Shopify that, I think I have this right, from 12 to 5 p.m. Eastern on Thursday is the only time you can have a large group get together. Right? And so that means if a group is doing a town hall or engineering wants to do something by discipline, those things can only happen in that window. And then, you know, outside of that, you can plan an entire week of like flow time for you.
The other thing we did was we moved some of our conversations and our communications off of Slack. So Slack is amazing tool. Do you Slack? Yeah, sure So a lot of people use Slack.
It's a great tool. But the feeling we were getting from Slack was that even though it's supposed to be asynchronous, it feels very synchronous. And so what we did was we still use Slack a lot for the synchronous conversations, but a lot of things that are async, so updates, async video of a demo, we move all announcements, we move all that to Workplace, which is like Workplace by Meta, like Facebook for work. And what we found was... because of the different modalities.
I never feel like I have to go into Facebook for work. I never feel the pull. Like if somebody, if something, if I get a notification, I don't feel like I have to urgently attend to it versus in Slack, when someone messages me, you feel like you have to get there. And that has made a big difference in flow time. And also just bringing the.
the pressure down of like the urgency of like, it's an update, it's an announcement. I can read it at a time that makes sense for me versus Slack, it's a conversation, there's an incident, let's do something about it now. So that's really also changed the way we work.
Yeah, it's very interesting how when you think about it on a practical level, it's always notifications, but having them on Slack, which is, you know, it feels like a chat tool makes such more pressure than having them on something like Workplace, which tend itself more to be asynchronous. So sometimes it's completely psychological, but it makes a lot of difference. No, and it has made a big difference because a large volume of messages have now moved to this format. And don't forget, Workplace has like a machine learning feed, right? Like driven by AI. So now it actually tries to surface the things that you might be interested in based on other people's interactions or things you've commented on in the past. Like it's got much more of a, hey, let me go and dye, put my toe into the river of consciousness of Shopify versus in Slack channel by channel, message by message.
You might miss, um, something that you wouldn't necessarily have. And so you do get the serendipity from workplace that you may not get in Slack. For example, again, two great tools. We just use them differently for different reasons. Yeah, I see completely. And some may ask, why do you think Shopify is able to pull this off? I mean, to stay a remote first company or digital first company, while other big tech companies are struggling with this or even, you know, going backwards.
Yeah, so I think that to be fair, I think there's lots of modes that can work, right? Like I think IRL works, I think remote can work. I think what we're doing, which is like 90-10, 95-5, like whatever it is in terms of like, we're not trying to be a hundred percent, we're trying to make sure we get together when it makes sense. Remote, I think there's lots of modes that can work. I think the early thing that worked for us was 2020 May, we made the call right away, so there wasn't ambiguity around what's gonna happen next. And that allowed people to actually sort themselves, right? Like, even folks who knew, who thought the pandemic was gonna end soon and they could go back in or thought it was gonna last forever, like, they can decide, hey, do I think that remote, or mostly remote, right, is gonna work for me? Right, I've now moved out to the country. Maybe that's amazing.
I wanna continue working in this model. Some people are like, hey, you know what? Actually, I wanna IRL culture and... coming in because I can, but everyone else is not there is maybe not for me. So we wanna actually, we call this heightening misalignment. So basically, if we're gonna go mostly remote, I keep saying mostly, let's just say remote, we go in remote, we're going full remote, are, is everyone aligned to that? And if you're not, that's totally fine. Right, like I think it's okay.
Like you can, if you wanna work on something that's not commerce, that's also okay. But there's probably a better company for you. And so we wanted to make sure it was clear, the rules of engagement are clear, and then you can decide.
So I think that's one. I think some companies have kind of like slowly, so to your point, some companies are like doing return to office, but it's not clear, right? They're like, we're remote, and then all of a sudden you get a procedure email that says, all right everybody, two days in the office this week, and then A few months later, okay everybody, we're going to three days, and I don't think it's being clear from some of the companies. And I think if they just heightened the misalignment to say, hey everybody, we really believe in IRL, which is totally fine. We're going to slowly ramp up in office until we get back to five days or three days or four, whatever it is. And uh. What do you think? Like, is that gonna work for you? And if not, like, maybe we should help you find another spot.
And some companies have done that. They said, hey, we're gonna give you three months severance. You gotta work, find a company that works for you. So I think the struggle is in the ambiguity. Yeah. So you think that remote versus IRL or in office will become eventually more like a DNA thing.
I mean, there will be companies who live with the culture of being IRL strongly and they will attract people who are like that and companies who instead start as remote first and then stay on the track. And you think that it's hard to change. while you're running you change track, I mean it's hard to shift from one side to the other.
Yeah, I mean, not impossible. I think companies can, I think companies just have to be clear on what they're trying to optimize for, right? So for us, I mean, so here's a good example, right? I don't think anyone would debate that IRL is better. Like if you and I were in the same room, it would be better than what we have now. Like we can maybe get there eventually with like, you know, Vision Pro or MetaQuest 3 or like these VR tools.
And by the way, I'll talk about that in a bit. We do have VR meetings at Shopify, I'll talk about that, but- - Wow I want to know - I'll talk about that. Yeah, we'll definitely talk about it. But I think the key is just that, like we know IRL is better. So we're like, cool, IRL is better, but what are the trade-offs? Well, the trade-offs are, I can't hire everybody I want because they maybe don't live near an office. I can't come in all the time because my computer is far.
Like there's all these trade-offs to kind of get IRL. The weather is bad. Like there's all these things that kind of force you to make the trade-off. And what we said is, we still believe IRL is amazing. So let's make sure. We use IRL in our toolkit.
Let's still get together. But there are many things that you can do not IRL. And the IRL parts of your workflow can kind of level up the trust when you're not IRL. So for example, if you and I had met at a bar for a drink, and then we had this conversation, the trust that we had by meeting first IRL will continue in our remote experience. So I think that just having an opinion about it and then saying, OK, we're going to be clear about our opinion, I think we'll actually solve a lot of problems from employees and companies, things and over time we can figure out what our mantra is and you can change like we all have the right to get smarter over time right like you know maybe we just maybe we decide to go even more remote meaning like less irl maybe we decide to go even a little bit more irl and you know spend more time in the office i think those are things we can all you know change over time but the what we want to land on is what are we trying to do and is it clear so the people are not feeling like every week some policy is changing Yeah, yeah, I agree.
I think one of the smartest takes that I heard about this IRL versus remote debate was from, I think, a former general manager at Amazon who said that the odds are in favor of remotes being more like the future, if not for anything else, because we have like three hundreds of years of experience working in an office, so it's unlikely to get meaningful any better while we had just a few years of experience working remotely. So you have better odds of we figuring out ways of making it better in like 10x or time. No, yeah, that's a really good point. We've only been remote working. And let me know, like, so this setup, even the advances in the last few years have been immense, right? Like high quality video, high quality audio, getting better equipment, getting better internet, async video, right? Like Slack and Workplace, these tools getting better, Zoom and Meet getting better, like, you know, lower latency, like all these things are happening.
And yes, VR meetings. So like, let me, let's talk about that, right? Sure. So we do something at Shopify that again, we talked about how Shopify is very different.
And one of them biggest differences is how involved Toby is in the company. He's a hands-on person, and we're lucky to pair with him on lots of problems directly, right? Versus at other companies, maybe you're part of a team doing things and kinda giving updates. We're all working together in the team. And so we get together every six weeks, and we go through every project in the company.
And we wanna ensure that the aims and the missions that we're working on are aligned to our overall like, hundred years mission of making commerce better for everyone And every other six week review, so every 12 weeks, we do it in VR. And when we do it in VR, the idea behind it is one, we always want to be pushing the boundaries of technology. Like if you don't know what's out there and what's possible, you'll never be able to see the future. You'll miss it, right? Like if you didn't get an iPhone early and you didn't kind of miss these things until it's everywhere, that's one. And then two, we wanna give feedback on the product and make sure that they get better. And so we use, Vision Pro just came out, so we don't use those, but we use either Oculus Pro or Quest 1, 2, or 3.
We use the Meta Horizons app. And we're not super adventurous, We're not in the middle of a forest having a meeting, we're in like a meeting room in VR, but we do go through the same thing that we would do IRL. It has spatial audio, so people feel like they're around you.
Everyone's looking like they can dress up their avatars, but we're in there for... I think it'll be three days now, but the last one we did was two days. So two days, three hours in the morning, one hour lunch break, three hours in the afternoon.
So six hours every day in VR over three days. So I guess over three days will be 18 hours. And to be honest, like you get, it is amazing. You get used to it and you're like, wow, this is like, this is something different. You do feel like you're around the people. I would say for me, Instead of the avatar, if it could use like a virtual, like take a picture of your face and like even 3D map, take the 2D picture and 3D map it on the face would be better because it looks like the person.
But other than that, the latency of the audio is phenomenal. You can interrupt people while they're speaking. Like it is very, very well done. Yeah, I love that you're talking about this because I think there are so many skeptics about VR and people who say it doesn't have killer applications, I mean it doesn't have an iPhone moment, but I mean this is not the only way, that's my personal belief, that's not the only way that technologies get mass adoption. There are also examples where things just get incrementally better and better and better until they're just everywhere.
And you look back behind that you didn't even notice that, but they're now everywhere. I mean, I'm thinking even the Apple Watch is another thing that never had the moment where it felt like the next big thing probably, but it kept getting incrementally better. And I think the MetaQuest, I have a Quest 3. I don't have many meetings because I'm a full-time writer, so I have meetings myself, but many experiences are incredible, pass through is great. I mean, it's definitely viable for work. I think it's great that we have now such more attention because of the Vision Pro and more so bullish about that. - Yeah, I agree with you
Yeah, and I think the idea is that. Again, you gotta be trying these things. The first time we went in there, it took us forever to kinda get in, and then get set up, and how do we draw on the whiteboard, and how do I see my keyboard pass through? You get used to it, it's new technology. But then over time, the technology faded away, and we just had, we had a three-hour meeting in there as if we were IRL, and we were all over the place.
I think even someone was in Europe. And so it really allowed us to kinda bridge the gap. And these are the kinds of things that you should be doing as a remote company. I don't want to get into too many details, but I'm spending time like two or three companies that are doing like remote telepresence hardware.
- Wow - As well And the idea behind those things is, again, you know, they don't have glasses, they're more like you go into a room and it's a different experience and someone there is someone else is in another room that set up like yours in another place. And you kind of interact with each other and you're like, wow, like the fidelity of these things is getting very good. The latency is getting very good. Now it's different.
You can only probably put that in an office. Maybe you could have it at home one day. But it's again, pushing the envelope of what's possible when you are trying to really figure out.
How would it be together in high fidelity? Get as close to IRL as possible without being IRL. And of course, we still do IRL. But when you are not doing IRL, how do we get closer? And again, so pair programming is another example. I don't know if you spend time in Tuple.
So Tuple, yes I've tried it. It's great. Yeah, so we use Tuple. Again, it's specifically made for pair programming.
Two cursors, you got the video, the audio. It allows people to really collaborate. We use it for anything from just pair programming every week to incidents happening. Hey, jump on this, let's pair. And it really allows people to move very quickly in a low latency way.
Yeah, I think per programming, I'm glad that you mention it because it's one of those practices that actually for a long time there have been like research and studies that prove that it's beneficial to so many things, but it's still, at least it feels to me a little bit controversial or underutilized by many companies. I know you're a big proponent of that. So what are your thoughts about embedding per programming engineering team and why do you think it's still not so commonplace? Yeah, so I had this analogy, like pair programming is like the underhanded free throw in basketball, right? In basketball, similar studies and statistics have shown that actually just walking up to the net and throwing it underhand is like actually superior in all respects to the overhanded free throw, except for one. It looks stupid. Right? Like outside of looking stupid, the underhanded free throw actually wins.
And there's many studies and stats on this and pair programming, similar. It kind of looks stupid, right? Like, why do you have two engineers on one computer? Even though they have two keyboards and two mice, like they're on one computer and like, aren't you losing out on all this productivity, right? And there's this famous funny tweet which says, well, if you have two engineers on one computer, won't they type half as much code? And then the response is, oh, no, we hope they write even less than that. Right. Because the idea is that it's not the volume and lines of code that is the solution. It's the, well, it's the, yeah, it's like the elegant solution, the most straightforward path, the, you know, the, the simplest, most performant, reliable solution tends to come from likely two people working together on a problem.
Not how fast can I type? Right. And so. So for me, I think of it as the following. Two people working in high fidelity via pair programming, reduces distractions, increases knowledge transfer, removes the siloing of information, less hacking happens, more forward progress happens even though it's less code, actually specifically because it's less code. And all of those things do come at a cost. And the research shows the cost is 15%, one five, which is actually not what you'd expect, not like 100%, not 50%, like 15%.
And what you get in exchange is like less errors, happier engineers, more learning, faster overall throughput and development. And it looks dumb. It also is hard to do. Like I remember in my last few companies, we did pair programming 40 hours a week.
And... The first few weeks you do it, people slept more than eight hours a night because it's very tiring to get used to. But it is a high intensity, high fidelity environment.
And we continue to do more and more of it at Shopify. Yeah, I think I read a study maybe from the early 2000s from Alistair Cockburn or something that said that, I mean, it had exactly the figures that you mentioned that the productivity gap was less than like 20, 15 percent, but then the participants to the study, like 90 percent of them after a few months was extremely happy, was meaningfully happier at work because they did pair programming. So it was, I mean, outrageously good numbers. for something that hasn't fully taken off in the next 20 years.
Yeah, it's the number one. If you can do it for people listening, if you can do it in your company is the by far the number one thing that will differentiate you from everyone else because you'll just learn faster and work faster than anyone else. And, uh, you know, just two extra points. One, there was a good article, I think in the New York Times, right? Sanjay Gamowat and Jeff Dean, who a pair program every Friday still from Google.
And, um, and the other one is just that, you know, like there's this notion of like in today's world of distract ability. happens to have a solution for that by having two people on one machine, like it's much harder to get distracted. And so just trying to stay focused is a real win for pair programming. And I mean, the other thing I'll say is, it is kind of like eating vegetables.
Like, what I found is, I've worked with lots of folks who pair programmed exclusively, and then they move to another company that doesn't pair program, and they don't pair program anymore. And so I do feel like there is a, or some companies will say we do it for onboarding, but then we don't keep doing it. And even Shopify, we use it sometimes, we don't use it all the time, right? And I think that's okay. All I'm saying is, it is something that kind of like, going to the gym or eating your vegetables, like you kind of have to kind of do it and then you're happy you did it afterwards. - Yes, yes, it's true.
But in doing it, it is psychologically, it is not psychologically easy to fall into. You kind of have to make yourself do it. Yeah, yeah. And I spoke in the past with companies who did it extensively.
And one of the, on top of this, another great benefit that they reported is that they just have to do less code reviews. So because the code is automatically reviewed by your, by your, your peer and you, otherwise you waste so much time. Sometimes code sits idle for hours waiting for other people review and they view shallow most of the times. I mean, it's not like when you're, when you work with somebody else. So there are.
sneaky aspects that actually increase the productivity. It's like second order effects that you only figure out once you set the practice. Yeah, and I think, don't forget, what is code review for? Code review is not only to review code before it goes into production. It is learning, teaching, asking questions, sharing knowledge, it's all those things. And actually, so just to go back, all of Agile, Agile programming, continuous integration, automated testing.
pair programming, all of these came from the same place, right? They came from a place of a good idea, turn the volume up to 11, right? So where does pair programming come from? It comes from code reviews. It's like, Hey, let's do a code review once a week. And then you're like, this was a good idea. Like, what if we did it twice a week, three times a week, four times a week. What if we did it every day? Well, what if we did it every hour? What if we just wrote the code together? Like that's actually, that's actually how pair programming came about. Same thing with.
you know, TDD, right? It's like, hey, an automated test is actually super useful. We should write more. Well, let's keep, let's write tests for everything.
Well, what if we wrote the test before? Right like it's actually the same thing just turning up the volume to 11 and so pair programming came from code reviews as like a Let's just keep doing it and that's why it makes sense that we're doing it all the time we don't have to do also do extra code reviews at the end, so if you have a Policy that's two code reviews before it goes to production you could potentially drop it to one with pair programming Yeah, it's true. And it's funny that you mentioned it because last week we had Kent Beck on the show and we were commenting just on that, you know, extreme programming, the extreme word come from what you just said. I mean, taking things that otherwise you would do just every once in a while and doing them all the time, testing all the time with the programming together instead of reviewing things once every few days. So I totally agree. And Another misconception I think is that pair programming doesn't work as well in remote environments, while my personal belief is that it works even better.
So I'm curious, I can't imagine what you think about it, but tell me your thoughts. Yeah, sure. So first off, I learned pair programming from Kent Beck in like 1998, like directly.
So I know him, I know him as well. But I would say on remote, yeah, I don't think, so don't forget, IRL always gonna be better, right? We know this, so there's a trade off, right? And the reason that I say IRL is still better is that by looking at somebody's hands on the keyboard and watching a keyboard shortcut. Like there are things that, you know, can translate to remote, but don't translate as well. And then, and using a tool like Tuple, for example, we don't find that there's a big barrier to remote development. In the past, IRL, you'd have to physically get up and find a pair programming room or a pair programming station and, you know, kind of be also, you know, together in a location. Remote allows you to, you know, Tuple allows you on the fly.
And so what I'm seeing, for example, is during an incident, right, which is probably impossible to happen IRL because you're not gonna both drive into the office, maybe in the old days, hey, something's happening, we have to drive into the office to fix it. But today, you can just send a TupleLink and then all of a sudden, you're now in high fidelity working with somebody synchronously pair programming. So I think that's a big win. The other win of course is, here's some funny ones, like personal ones, right? So for example, I always have like a little bag of like snacks here. Not everybody likes that in their work environment, but if I'm pair programming, eating my snacks and nobody cares, right? Maybe if I can mute myself while I'm eating so that they're not like, oh my god, this person's eating and their hands are dirty while they're with their keyboard. Like you don't have to worry about the factors that, you know, humans deal with in person.
So I think that remote can work quite well. And again, I'm glad there's a bunch of purpose-made tools, not only tuple, that help you doing that. And by the way, some people do it on those like Google Meet and Zoom, just like mob program with multiple people. So I think that... Remote is not an excuse.
I do hear it as like, oh, we're remote, we can't do pair programming. I think you send any, a few seconds on Google looking for remote pair programming tools, you'll see lots and they are quite good. I think it's one of the areas where tooling has really improved, you know, by an order of magnitude in the last few years.
And one of, I mean, going back to the fact that we will get better at remote with time, we have just few years of experience. I mean, when it comes to tooling, things are getting better and better at the speed. I mean, that is incredible. Even, I mean, before you mentioned asynchronous video, even asynchronous video has gotten incredibly better with the AI notes that can now summarize a lot of content. So things are getting, yeah. I'll say something on remote pair programming because we have, we can have, the cool thing about remote pair programming is we have data.
So for example, in the past, it was harder, unless people use like the git pair command and they tag both committers, it was hard to know that people are pairing. And so what we know now at Shopify is we can see pair programming hours, number of people pairing, and we can see this in the tool. And what we've learned is a few things, and these are all correlation, you can't really get causation, but for example, For example we can see that those who pair program tend to have a higher impact than those who don't pair program. Now I don't know if pairing makes you higher impact or if you are higher impact because you pair.
I don't know the arrow, which way the arrow goes. The other thing I can say is that we have these questions in our... impact cycle where it asks you things like, is my manager technical enough for the job? And what we notice is those managers who pair tend to score better on those questions than those who don't.
And again, it might be because they pair or they pair because of that. I don't know which way the arrow goes. But again, we see kind of all the positive correlations happening when you pair.
Outside of just low bug count, information sharing, happiness. learning all that stuff. But then now we're seeing it also on the, hey, people feel like I'm more technical.
I also feel more technical because I'm pairing or I might have higher impact because I'm pairing or pairing is making me higher impact. Yeah, I mean, this is very interesting. Do you use metrics to measure impact or does it come from performance reviews? Does it come from, yeah. - No we don't- So again I'm just sharing with you like, We look at it as like activity.
I'm trying to showcase, here are interesting things. It's hard to get any real causation here. So that's why I'm just sharing it as like, here's interesting data.
And so I think it's again, a way for us to share best practices, right? So like, if you see, if you like, you know, you end up seeing a really, really good, really, really good, working with a really, really good engineer and she's pairing all the time. And you're like, wait a sec, maybe, maybe I should pair. Right. - Maybe.. If she does it.. Exactly. And that's what I mean.
So that's the kind of ideas. Like if we share this kind of data that shows, Hey, by pairing more, you might, uh, you might actually become, uh, remain more technical if you're in the management track or you might, um, ship more, or you might, uh, enjoy and learn more about your area, then maybe you should pair more. Yeah, so has it become a device even for managers to stay technical and a little bit more hands-on working with engineers? No, for sure.
So Shopify, I mean, don't forget the CEO is a coder and a hacker. Right? Like it comes from the top. And one of the, I think it's one of the coolest things around Shopify.
And again, I will put that in that Venn diagram I talked about earlier in that, not only do we expect folks to be extremely technical all the way up, we expect them to be in the details. And so what that means is, it doesn't mean that I'm coding all the time, but it means that I could pair with somebody on a problem. I can still have relevant commentary in an architecture review and potentially see, help people see around corners, because I'm looking at lots of projects be only in their one project. And we have, you know, there's that line, right? Cavalry captains must be able to ride horses, right? So everyone comes from a technical background. Actually, I'll tell you something very specific. We do coding.
technical deep dives, pair programming, interviews as part of the interview process for managers all the way up. Like I do them with VP candidates. So I would sit with a VP candidate and we will pair program. And the reason for that is not scary. It's not to scare somebody.
It's to say, hey, even though you might not be coding day to day, we expect you still to be able to not only code but have a love of coding and have a love of getting into the details. And like, if that's you. this could be a good place for you. And by the way, it's totally fine if not, it's just like Shopify's in there for you.
If you're like the person who's like, you know what, I kind of got out of, I got into management to get out of coding, or I wanna be somebody who just does like assembly at this level and I don't want to get into the details, that's fine, that's not a judgment call. It's just like Shopify is probably not right for you as an engineer. Yeah. I'm curious what you ask them. Is it more system design or go through, I don't know, lead codes? I mean, whatever.
All of the above. So a combination of things. So one is, yeah, the lead code type question would be more for like a pair programming, which is just to see like raw. Do they have the, um, Do they have like many, many years of coding behind them? So the muscle memory just comes back. They're like, oh, I'm kind of rusty, but the muscle memory starts coming back.
And then we do a technical deep dive, which is like describe an architecture of a previous system that you were a part of and like, what are the trade-offs? So like, we'll do both sides of that. I will say, we haven't gotten into this yet, is Copilots really do help you get un-rusty very fast. And what I've noticed is even for myself, that... It shakes off the rust by just getting in to VS code with a copilot there.
And you can very quickly build an answer to elite code or even build a prototype of something quite readily using GitHub Copilot or one of the other ones. Yeah, it's very interesting. I think this is a great conversation that is going on in the industry because you see engineers getting more and more productive with co-pilot with AI and many people question the role of managers, right? I mean, from saying that you need less and less managers to saying that managers need to be more and more hands-on to prove their value.
So curious about what you think about this. I mean, what's the true role of managers, engineering managers especially at Shopify? latter is more true, which is that by being in the details, you can be a team member on the team, right? The goal is not for you to sit back and just say, okay, let me oversee my domain area and just pontificate around the things that are happening instead. It's like, how do I help the team given the things that I can see and the details I know about this problem.
How can I help escalate things early? How can I help them get unblocked more quickly? How can I help them agree to throw out all the prototype code, because the learning is the thing that comes with the prototype, not the code. So there's all these things that an engineering manager can do. And management is super useful. I'm not on the we should get rid of management. I'm on the how do we give managers superpowers to do their job better? So that, they can unblock the crafters, right? We talk about this at Shopify.
Shopify is a crafter's paradise, but the managers are there to ensure it remains and still is a crafter's paradise. So your role is kind of to support the crafters, right? Which is, I think how every manager thinks anyway. So we have the expectation in engineering anyway, that you are very technical, understand the details, and work to unblock all the amazing crafters you work with every day. And AI, I think AI and management, is very, very early. All I'm really seeing is like AI copilot, right? And I mean, if you watched a lot of the GitHub universe talks, like they talked about it as a pair programmer, right? As a pair programmer for you, so you don't have to code alone anymore. But I will say like one example on the management side is, we encourage copilot use, not only in engineering for crafters and managers, but also for candidates.
So if you came to Shopify and I said, Hey let's do a pair programming I mean, I would expect you to use a co-pilot at this point, right? Like unless you're in a company that's not, not using a co-pilot. So you, it would be, it would be new and weird to use it in an interview situation. Um, I would readily encourage it because if my question, if the question I ask you is just like very easily solved by chat GPT and we couldn't have a good conversation about it, then it's probably a bad question instead. Instead, if you, you know, If you were able to use ChatGPT and GitHub Copilot and build something interesting, we would talk about it. And we would talk about the trade-offs and then we would modify it.
And then we would say, okay, what if, you know, and then I know some of the interesting areas where Copilot is not good at giving you answers, like meaning it would give you potentially the wrong answer. What do you do in that case? Can you read the code still, right? Every piece of code that comes out of Copilot and then you... merge into your code is your code, right? Like you have to understand it.
You can't just accept it as a given and correct. It's just the same as you can't copy from Stack Overflow. Yeah, exactly.
I think it changes slightly the shape of the skills, you know, that are more or less valuable from you as an engineer. I think, I mean, speaking of Kent Beck, I think he said on Twitter that after trying copilot for a while, felt like that 90% of his skills, the value of 90% of his skills dropped to zero, but the 10% left leverage should shot up to like 1000 times more valuable But did you see the, did you see the reply to that tweet? No Where someone, so I agree with, I love that tweet where he said 90% is useless, but the 10% is super valuable. And somebody replied, which like, which parts are which? And he said, I have no idea. I have no idea.
Yeah. That's the best part. last week I asked him, did you figure out in this time, because this was like a few months back and he told me not yet. So the jury is still out. So did you...
It's true, right? Actually, one of the lines that I like to use, and I probably just stole this from somewhere, is like, be an AI centaur. Be somebody who can use AI, right? Don't be, you don't have to be the, oh, I'm gonna use AI in this situation, or it's a chore for me to reach for, I have to be reminded to reach for it. Like, you should be reaching for this thing all the time, and you should know the limits and how to use it and how to prompt and when it's good and when it's not good. And...
You plus AI is like extremely powerful. I mean, they have this analogy in chess, right? There's like the human chess players, and then they have the AI chess players, and then they have the centaur leagues where it's like a human and AI, and the human and computer chess league, those players are the best. They're better than the computers, they're better than the humans, like they're the merging of the two. Yeah, I think this will just train new generation engineers who are just getting better and better. I mean, just like chess players, now that we have engines, we have that kind of support, are incredibly better than the previous generations because they just have access to more powerful tools. The same will happen.
yeah, you may have seen this right on Twitter. Somebody posted the, like when AlphaGo came out, the types of moves that Go players would use. And then after AlphaGo, it had all these moves that humans now use that they, like the skill level of humans is shut up because of AlphaGo. So I agree, it's all about the centaurs. And so you have to just embrace it. Yeah.
And you, we have talked about AI for coding basically. And you said that AI for management is still very early. Do you see areas where this is changing your workflows already for management or areas of improvement? Yeah, I think it can. So I think, I mean, we're starting to still see the use cases be pretty tactical.
So for example, Gmail helping me write an email, or inside of, I think they're calling it Gemini now, inside of Google Docs, right? It'll eventually be called Gemini. So help me write a. Help me write a blog post or help me write a tech design or help me write, you know, or spreadsheets, help me analyze data or help me give a presentation on this topic. So I think, again, they're very tactical. Now this is a lot of.
This is a lot of people's jobs, right? Which is like, hey, I have this information, I need to get it conveyed across to a different audience. And then the reverse, summarize, right? Summarize these interactions, summarize this document, summarize what this data is telling me, summarize this presentation. So I think the tactical things are there, but what it doesn't have yet is like, which employee should I focus on right now? problem in front of me has the highest risk that I should pay attention to? Hey, I met with 50 candidates. Which one is likely to be the best one for my company given the culture that I have and given their background, right? Like there's a bunch of these things that are still very, very early.
And I'm super excited for them because I think they're all gonna like not work at the beginning and you're gonna just have to like figure out the boundaries, right? So around... Like how do I use this new centaur ability for good? And clearly you want to have like the best people at your company and you want to have the managers who have all the superpowers in the world to do their job super quickly and block all the crafters, you want crafters to have AI. Like, you know, imagine you're an engineer and it would say, Hey, tell me the gnarliest piece of code in my, in my code base that I should probably refactor. Or help me find the biggest bottleneck.
in terms of performance when we're under heavy load. Or help me, these are real questions that AI cannot yet answer, but likely will answer. And I think so, I'm a pretty optimistic person. I look at all this and I was like, let's figure all this out.
Let's keep asking these questions of the code base and let's keep training our models to go more quickly. So we're trying to embrace it. And then of course, all the non-engineering use cases as well, but I'm focused on engineering. Yeah, I think of course one of the things that AI is very good at is taking insights out of data that you have. And you know, there is this big debate sometimes about measuring engineering productivity using metrics and stuff. I don't know your thoughts about this, whether you use some like Dora metrics or cycle time or space exactly at Shopify.
Well, I mean, I think, I mean, we know the answer is don't ask McKinsey. I don't know how, I don't know, I don't know how McKinsey, yeah, it was, it was - Shots fired - Yeah, it was, very strange to see an article from McKinsey on developer productivity. Although, you know, to be fair, like, I like, everybody should talk about it. It's just like, I don't know why there's so much hate, but I read it and I was like, don't know if I agree with this, but it was a cool read. I think in general. It would be interesting to figure this out.
I think the complex, and I think eventually we will sort of figure it out. I think the, you know, here's a good example. We use GitHub copilot right now. I talked about in November, I think that we have almost 1 million lines of code written by copilot in the Shopify code base and people were like, wow, it's - Wow - Wow it's amazing! There's the productivity. I was like, well, what if I had said we have 1 million lines deleted by copilot? What would you have said then? Like, I think, like, we don't know yet what the metrics are.
- It's true And adding lines of code, deleting lines of code. Like, you know, we talked about pair programming and two people will write even less code than them separately, right? Like less than half of that code. So we can't use lines of code. It is a little bit like, what's that line? Like, I know it when I see it. So you can kind of see a good engineer, but you don't really know what they look like.
And you don't know why they're so good, but it could be that they stop people from going down the right path. It could be that they don't allow hacking. It could be that when they write code, that code tends to stick around for a long time without any modification. It could mean that they're a trusted person to call during an incident. There's so many different attributes. And maybe over time, those signals will get added to flow in.
The data exhaust from those signals will flow into some model. And we'll be able to say, wow, that person's a great engineer. Or that person's a 2x engineer. And that person's a 4x engineer. And that person's a 0.5 engineer.
And they just have to learn these things. I eventually think that is possible. I think right now Dora and Space, they're useful metrics. I think here's my take on them. Usually I find that they are good at finding bottlenecks or problems. So you can find a team that's not performing or you can find a person who might be blocked but you can't really tell good from great there.
And I think that's okay. Right? Similar to Shopify, we, you know, we, like I mentioned, we expose some of these metrics and dashboards to help teams kind of find, uh, where they may be blocked. So we have a system called GSD.
It's called get shit, get shit done. And it basically has like a weekly cadence where you put updates in it. And then you can mark things as off track or on track, or I need to make a change of decision or get this reviewed.
And that is like data exhaust that can tell you whether a team is potentially, uh, being, being blocked or going down the wrong path. And then we can all jump in and say, Hey, OK, here's the aim. If we look up here, can we get this back on track? Or we realize that we're down the wrong track. We should just stop the project and start over another time. I think that those things can help you do that. But I don't think yet it can take a 1x or 2x team and compare it to a 10x team.
I don't think you can tell that from those metrics and Maybe you want to see that, maybe you don't. But I think like, if you look at like the 100 meter sprint, you can clearly tell who the top 10 fastest people in the world are. This doesn't tell you that, right? You can't look at it and say, well, who are the best development teams? Now, here's what I'll tell you I do like to look at. One is, I do think tactically, pair programming in general, from what I've seen, does tend to showcase teams that tend to perform better than teams that don't pair because the intensity, no distraction, learning, et cetera, that's one.
And then two, I love to look at weekly demos. For me, weekly demos are the thing that helps me determine whether a team is performing well or not. And the reason you can do that is because you can look at weekly demos across multiple teams and sometimes there's a team that is working in a gnarly piece of code Context will come out every time they demo because you'll ask questions.
Hey, why is this taking so long? What's happening here? Oh, I have to refactor this. We have to upgrade this library. We had a security incident. And then with other teams, you see them moving really quickly. And you're like, OK, the code base is getting into good shape. But for me, weekly demos, and again, we have our tooling to allow this, is a great way to see the velocity of a team as being predictable.
One other thing I'll add before to your next question is, when I was at Pivotal, They have a tool called Pivotal Tracker, which is a great product management tool. And use it. Great. I've used it. - You've used it, yeah! One of the things that they have in there and I haven't used it in a while, but I know they had it before, maybe it's now more prominent is they would look at the volatility of velocity.
So basically, if you deliver 20 story points this week and then three next week and then 30 the week after and five this week, like, they'd be like, why is this so volatile? So they wouldn't look at the number, they just look at is, is the velocity seemingly stable? Okay. And whatever the PM is, their story point granularity is going to likely be similar, right? Cause they're not changing PMs all the time. That's a good there.
And then look at the weekly demos and they can be like, okay, that's a well performing team if the velocities, if the volatility is high, they're like, wait, something interesting is going on there. Yeah, so it's predictability before speed. You wanna have constantly high throughput, which is predictable. It doesn't have to be crazy
2024-02-29 13:05
