Peer to Peer Technology with Pear and Keet by Holepunch | David Mark Clements | Beyond Coding #171
Hi everyone. My name is Patrick Akil and if you're interested in peer to peer software, this episode is for you. And I don't mean Skype, WhatsApp, or zoom. I mean applications that run on my phone, my device straight to yours without anything in the middle.
Joining me today is David Mark Clements, author of Pear Runtime, over at HolePunch, the peer to peer company. I love his energy, I love his stories, and he's doing some fascinating stuff. So enjoy! Beyond Coding. When I looked into peer to peer technologies.
I did some researching and I looked for really good examples of this technology because I want to kind of understand the pros and cons of it. And the examples that I got were kind of Skype, zoom and WhatsApp and I'm like, okay, that is peer to peer, but there's always a middleman there involved. Yeah.
See that's that's the whole thing where Facebook and so on. They, they went from private messages to direct messages or one way or the other, I don't remember, but like either way, like this idea that the messages direct is a lie. That's a lie. It's not direct.
Exactly like your at the very least it's sophistry. You're twisting words to to to indicate something to the user. So maybe that's the thing is that many users think that kite, which is a messaging app built on pair many users think that apps already work like that.
Yeah. And we're just trying to make the app that we think should work like that, because we agree that's how apps should work. And the reason users think that they should work like that is because the companies with all of the money have made sure the the average user does think like that, because that's actually how it should be. But then what is the difference? Because I understand with my technology background that if I send a message on my phone and then I log in on my laptop and the messages appear, that obviously there's something in the middle that just relays, then the message back to me, yes, right.
But that's not the peer to pick technology that you're operating in. no. So so with Kate, we have this thing called identity. Yeah. And it's quite similar to a Bitcoin wallet address. You have 24 words that you record in store.
And then we create sub identities from that identity. actually while we haven't gotten into the tech yet, so but, I think later we should talk about the, peer to peer authoring with identity. Okay, but that's a rabbit trail. but, yeah.
So there's, there's in order to discover other peers, there's something called the distributed hash table, which is nodes. All pairs can be nodes in the DHT. All of our peers can be nodes in the DHT. And they just help you discover other peers by a given key. What do you mean by DHT? Is a distributed hash table? Gotcha. Okay. So it's a way of, of allowing peers to discover each other through, a peer to peer network.
Yeah. So you can imagine that as the central point, if you like. but that central point is that you're inside the central point. So it's not a central point.
No, that makes sense because you're participating. You're one of the nodes. Maybe. Maybe not. Maybe. Right. Yeah.
so, so with, with, identity, you create that, key for yourself. and then when you log on to, when you don't log on, sorry, but when you open the key desktop app and you've already set up your identity on the keep mobile app, for instance, you can then scan, or let's say the other way around, you set up on desktop. You can then scan the QR code from desktop to mobile.
That just just transmits your identity. Yeah. Which is again just like a cryptographic key.
And then, that is then used to allow synchronization of state between your devices, peer to peer. And they discover the devices each discover each other through the DHT. Yeah. So there is no central service. that's saying like, oh, he sent this message, I'm storing this message. And, when they log in to the central service, they'll be able to see the message stored on our servers. Yeah, that's that's not what we're doing.
It's, it's only ever device to device. Only on the device itself. Yeah. What would be the main benefit them? Because I never thought of this. I download the key because I wanted to talk to you, and then I downloaded it on a laptop, probably where I shouldn't have downloaded it. So I wanted to get rid of it. But then I needed the messages and I needed our chat room basically.
So I transferred it to my other laptop. I did exactly what you said. I tried logging in. I set up my identity. I filled in the identity on the other laptop.
I only had to do a, confirmation basically on my initial count. Yeah, and that was it. Then it was connected. That's right. And that's separate from the server where I would just have my sound, I would log in and that would be, that would be that I actually had to pad these devices. Right? Yeah. why would I want that in the first place?
Well, it goes back to why you would want local state on your device in the first place. So, you don't have to set up identity on key. You can you could just have a separate desktop. I, on a separate mobile app, and then the application or the device itself is the identity. Right.
you join them up because you want your state to sync between devices. Right. So it's really about the local state thing in the first place. So the way that, power applications work and cheat is a power application. So this is the generic. Yeah. They, they, they work on, a technology called hyper core.
and hyper core is an append only log. So, that log, everything that you do in the application, all the actions you take are appended to this, to this log. So when you send messages, it's appended the log, if you like. It's appended the log. If you delete it's dependent appended to the log. Yeah.
So deletes are known right. and then that state is private to you. No one else can see that state. So you have an auditable log of everything that you do.
only on your devices. That means that pre-Internet, the situation was that if you were accused of a crime, then you would produce. If you and you were innocent, then you could try to produce evidence to prove your innocence. But you couldn't really be mass surveilled for guilt. We now live in a different era, where, mass pattern matching and all of these different things are happening to, to basically, be then be used as evidence against people and like, that's fine.
And if, if, if we're, if we're putting if we're putting people in jail for things that we all agree are bad, but that level of power for someone to have, for anyone to wield is really problematic. If it goes in a direction that's just like, you know, well, it's not really a crime. What you're saying is a crime isn't really a crime, but the state says it is and they can massively offer it. So everyone you know, you're going to jail because of whatever it is.
Maybe it's being Muslim, maybe it's, I don't know, being super right wing or what, but like, just like privately or whatever, right. or super left wing, like whatever it is. The point is, it's not absolute power is a dangerous thing for a human to have. Yeah. so, the append only log only living on your device.
That brings us back to that time where if you're accused of a crime, you can prove your innocence, like, better than ever before because you have a full log that you can just go. Okay, let's look at my history. No problem. Yeah. but if if, But you can't be mass surveilled under this, under this approach. so beyond that, there's, there's a bunch of other benefits if you're transferring files.
So, like, you, just you yourself know, and you're producing, like, you know, large gigabytes of data, that has all the transfer between devices. Oh, yeah. Yeah. You have to get some kind of iCloud account. So and then you have to pay them a monthly fee and then you might hit limits. Yeah. They complain every single time when we upload new files that we've overspent basically.
Right. And capacity. Well if it's peer to peer it's free. So you can you can send terabyte. You can send a terabyte giga. video file if you want, you just drag and drop it into your desktop. Now that syncs between your devices, you don't even need to send it to the other device.
So if you've got your desktop, you you have to leave your desktop on so you can download from it. But if you want to put a video on your desktop key and then view it in your mobile key or iPad key or whatever, then fine, no problem. and it's is instant, is unlimited.
there's no monthly fee. Yeah. There's no complaining about how much data you've used, because, a lot of the time you're just doing local network transfer. and the fact that, like, it's 2024 and since, since we've had this technology, this is where we've ended up, it's absurd. It's patently absurd. It's a monetization strategy, but it's not it's not technologically needful in any way that it works like that. I mean, it it looks like I need it because I don't know of an alternative.
But if I can sync between devices, then obviously the middleman I don't need, because usually I am transferring files from, for example, the studio in the back room there to my laptop or to somewhere that needs to editing. Yeah, you can also send to friends and your producer, whatever. So if if you producer has he and he needs to send you a terabyte video file, he just pings you and says it's coming. And you just you just get it from him straight away. You can download it whenever you need to.
Yeah. so, that's that's huge. and a lot of people don't really, understand that because again, we've, we've been primed to, to accept the status quo as is by the profit incentives that have been happening since the 90s. at least the 90s is like the when I was a kid growing up in the 90s, the major thing that brick and mortar people were saying, because I was a little bit in tune, you know, like, I was I just got on the I got the internet, went to 12, and it was like, like internet. I 5.5 had just come out. Right. and like a lot of, like a lot of brick and mortar people saying like, well, you can't monetize it.
You can't monetize the internet. It's not monetizable. and then you had the.com boom and bust and all of that kind of stuff that went on. Yeah. but part of that monetizing strategy, of course, was, to create a, a dependency on a centralized approach, because it, it wasn't conceivable at that time as to how you could monetize, the online world without some form of control in the middle. How do you do it without money? And that's still a question to a certain extent today. But I think that we have enough understanding, like we have enough mindshare among the human population because we need that to start off with.
Right. from, from like what's already happened. I don't think it's been bad. No. It's just that it's going towards an extreme now that could get very bad. Right.
And so that in the, in like tower sense. Yeah, I think so. Yeah. Because I've also I've thought of this centralization pattern. Right. And how it originated.
I understand where we are now and I don't think it's feasible also looking towards the planet because I'm now like I, I'm a product manager in G and I'm actually in the ESG space. And we saw and we analyze absolute emissions intensity emissions, which are same but specific to the assets that a company holds. For example. And we saw Microsoft and their chart with regards to absolute emissions nowadays because of the EU stepping in and regulations changing by I think 2040 or 2050, we need to be at a net zero with regards to the companies that operate in the EU. So legislation is making sure that people think about their spent right and start giving back on the planet, not just anywhere on the planet, but actually where they do their own business in that way. And with the AI boom that we had some released late last year, compute went up.
Ridiculous that, yeah, Microsoft was on trend to hit their targets for 2040, for example. And then because of that new technology, compute went up, which means their absolute emissions as the central party that provides that went up as well. And I think nowadays what's interesting is what you're working on now that we have more compute power, let's say locally these machines and hardware have increasingly massive assets.
It's incredible. Then we don't need this centralization anymore. Yeah. But then I'm also interested in with centralization also comes kind of ease of use in some cases.
Right. Yeah. Well it's always a trade off. It's a trade off at the end of the day. But what we're doing is we're making for the user the ease of use, essentially transparently, as much as we can. Right. So, if you use key, it feels like a messaging app. Yeah.
the only real tell is when you set, like, when you send a message, it's fine, because we, we do this thing called, blind mirroring. So if you send. So if you send a message from one device to another and both devices are online, that message is going to go straight from device to device. Yeah. If you're not on line, how does it get to you without a central server. So we have peers that also do and hold an encrypted packet for you, an encrypted blob.
and then when you're back online, relay that blob to you. gotcha. and so, large images or large messages, if you like.
Text is fine. It's always fine. But once you once you go into images like, there's a limit to what a, what a pair was going to want a blind mirror. and so the tell within key is like, sometimes it will say like, oh, this image is download and key workspace pair or something like that.
So like that's an example of the sort of trade off. Right. But like we also have to think about how the system can grow over time.
because instead of paying however much you pay for cloud and having all of those issues we discussed, if you want to send images to another peer or even say, terabyte video files or whatever to another peer while that peer is offline. Yeah. then you could, participate in a marketplace where peers who are willing to lend their capacity will do so for a few SATs or whatever it is. And so you can commoditize the resource.
You're not you're not looking at, you're not looking at, you know, this, this monthly subscription model that's like, yeah, costs and stuff have factored in, but like, what's the margin on that? You've no idea, right? If you commoditize the resources like it's just a market for commodities. Right. Interesting. we have this concept in the energy market because companies I mean, I know this because I'm in the space companies can do, for example, the electricity as a byproduct of whatever their supply chain is, is most likely going to fall into their scope, two emissions, and they can offset that. Scope two is usually broken down into market, and location and location is onsite wherever they're factories, for example, the electricity usage would be there and that would be the absolute emissions.
If they then buy energy certificates from people that have solar panels or bigger windmills, they can say they have green energy and that would go into their, location now that we're going to their market, absolute emissions. So what you see is on site, you have your location emissions, your factory uses electricity. And if I can buy enough energy certificates that say that my energy is green, I basically have green washed all my electricity usage. And a company can claim that they use green electricity, right? But they only do that by buying the energy certificates. So they're paying for the compute or for paying for the usage that someone else has done.
Yeah. And me, if I have solar panels, I can actually contribute to that grid. I can say I have these energy certificates and I can sell them to Google or whomever. I mean, as long as those energy certificates are valid, I mean, it's kind of not the worst thing ever. No, no, better than not that that's the existing system that we have. Yeah.
I've never thought of it to be used with compute. because that looks like a missed opportunity or an opportunity we now have with hardware. We have it with hardware, and we have it with a peer to peer system, because if you try to do this via a centralized entity, there's no incentive. Yeah.
And it's just it's just going to be very boxed in. And you know, you might get some users, you know, if it's a really large, let's say Microsoft tried to do it or something like that, you know. but I think it would be a very self-contained thing. And, and I think what I'd rather do with, with the P2P approach is, create markets and participate in them. That in itself is a monetization strategy. Yeah. but the, the I thing you brought up as well plays in here because, With centralization, as you noted, investment in AI with a goal to increase centralization has gone bananas in a few years.
Yeah. and it's sucking resources from everything else. and, I don't know, maybe I hit a limit. I don't know, I don't know, it's seems to have slightly plateaued. Maybe. Yeah.
but. Either way, I think a much better way to do I. Let's I always been thought about for, like, decades. Right. And you got all these, like, you know what? What are the logical extremes of AI? Like what scenarios can we end up in? and the hyper centralized scenario is like one of the bad ones because I gained extensions. probably unlikely.
The bigger threat is a human controlling an AI because an AI can do mass scale kind of things. Yeah. Let's say that the humans bidding. Yeah. and again, that's, that's an issue of, you know, as it's very Lord of the rings, but it's, you know, it's an issue of having way too much power, you know? So, within a peer to peer system, as you mentioned, we have devices that are capable of running local alarms. Right.
so you could have a peer to peer based alarm, the that works on your devices with your identity, syncs between devices, knows everything about you, but on your device. So it's the same as just having data on your device. Whereas an AI that knows everything about you, that's someone else owns, well, you're vulnerable if that's the case.
Yeah. And that is the case. We're all vulnerable. so being able to have, sort of an AI that's sort of limited by the capacity of your device, it's good. But like, you know, it's not going to compete with the what they're putting into very powerful AI because it comes down to how much compute power they're going to front it right. but but if we have this marketplace of peers and the commoditization of resources, then I compute is just another resource.
Yeah. and from there, users can maybe hook their, allow them up. Sorry, Dells. Yeah. with with another, peer to give them additional compute power for that.
Like the, the technological possibilities of this, like whether this is actually possible or not is like debatable. Right. And I was this is a research, thing for, for hole punch.
It's not something that's going to be around for a while, but like, like in terms of envisioning like how how that can go in a way that's not hyper centralized. I think we need to do something like this. once once you have that, then AI becomes, our tool. Yeah. Not a, not a hyper, centralized, like, elitist sort of control tool, you know? Interesting. I was thinking, so we talked about it for a long time, and I can see it from a messaging perspective, as well as kind of the audit of my data being on my devices or as you mentioned, with that mirroring mechanism being relayed back to me whenever I need it on a new device and my device is just broke, an EMP hit my house, and I don't have any devices anymore.
Like worst case scenario, I still have my data just by virtue of having it be relayed through other devices. and then I was like, okay, but what about compute, right? This peer to peer network, I can see it for video messaging, I can see it for file sharing. But file sharing is a problem I have just because I do podcasting and I have huge gigabytes worth of files. And I would like to keep them ideally somewhere. Right now it's on a drive and I don't mind, but I don't know if I could have it by myself if I choose differently, basically because I don't have an option currently. Or maybe I do and I don't know about it.
You do have an option if you use key, exactly which is something new to me. But then you talked about decentralized compute, and I was like, first of all, this sounds very complex, but it still sounds possible, right? Because in in essence, that's what the cloud also does with horizontal scaling, with auto scaling in the first place. And sure, they might have bigger machines on a vertical level here and there. You pay a lot. You pay a big premium for that horizontal scaling is basically what they do in essence most of the time.
Yeah, yeah. so with a centralized model, what you're doing is you're you're, you're sort of designing and building a system, and that system sort of is a self-contained thing, right? Yeah. what we're doing with power is we're power is a is essentially, I like to call it a software fractal. like, it's a self repeating piece of infrastructure.
Okay. So for a piece of infrastructure that builds the system from itself through participation. And if you think of that as a primitives, all the way up to like communication markets and different things like that, then the same, principles can be applied to, to, to storage capacity. I compute all the way through.
Yeah. It's just you're turning the system inside out and looking at it from the inside. And if you do that, we are the system.
we all participate in the system. And the system is, like, essentially equally working for all of us. do we need to get to a tipping point in adoption before that can happen? 100%? that's what I'm thinking. Yeah, definitely.
And that's why I said earlier, like, you have to think about how this can play out over time. because those markets don't exist yet. but the potential for them does. And it's been unlocked by the power technology. Yeah.
it's a it's a bit of a journey, you know, and there's, there's multiple avenues that we're that we're where, we're exploring, around all of this, at the minute where we're at is, we are getting a lot of adoption of kit. it's very difficult to to measure that in a peer to peer system. we, of course, have mobile downloads. Yeah. So I go through the app stores, but, the, there's a, the, the DHT can, you can sort of see the, the global DHT that all peers use, not just kit.
Yeah. So any pair application can uses the same global DHT. but the we, we sort of roughly know who's like what what apps and whatever using the DHT and key the presence of key. It's so large in terms of adoption that we can see that spiking DHT usage. So we know that the keeps growing. you won't see a lot of reviews on the App Store and what kind of stuff.
But like if you think about the type of early adopter, early adopters of like private secret messaging technology, you're not going to get a lot of reviews. I there's not the people that lead, people that read reviews. So interesting. So if it was not a global DHT, then you would be able to see your usage, right? Because then it would be specific for yes, the network and the participants there. Yes. Interesting.
So if you launch your own you can launch your own pair up. Okay. using path know using the same, same DHT, but you'll just be like, unless you, unless you can grow bigger than key, you're not going to be very observable at all. No, it's the biggest sort of players that are going to be observed as observable. And that's, pretty much key to the minute.
Interesting. I wonder if you ever get more insights in usage than because that's still from an adoption standpoint, and especially when we're talking about that tipping point where more becomes possible and you have this decentralized opportunities kind of arise by virtue of participation. You want those insights? I would say no. theoretically, but philosophically, not so much. Because, as you know, privacy, is a is an important concern for us. And we don't want to, we have to be very careful.
the technology, we can't we can't make trade offs in the technology that allow for the kind of monitoring that you might want here. Yeah, right. We can't do that. And we won't do that. So we're actually happier to not know. Yeah. And just that things play out interesting. I and we're not we don't use the, we're sort of back into early internet kind of development model. We're not doing a B testing.
We're not doing all that kind of stuff. but it's a communication up. Right. And so we have, we have, plenty of very s different, key rooms, which, by the way, keep room can scale infinite. It's not infinitely, but we can scale to plan a population size.
Well, so, theoretically, there might be UI bugs that would stop it from getting all the way that theoretically, in terms of the algorithm underlying it, we can scale to the planet. so we have like we have a, we have a Bitcoin room that's got like, 1700 people in it. Last time I checked. Might be more now. Yeah. No, but, but so we have rooms where people are giving us feedback constantly about the application, and they're using it to give us that feedback. So we have the advantage of like making a chat app and people being excited about it and then using it and giving us feedback. So, that that works very well.
It's more qualitative. and, and certainly more of an art than a science, though. Yeah. I like the, no compromise on technology because when I said it, I was like, this does sound like my view is kind of anchored by the centralist systems that we use by solutely.
And then I was like, okay, yeah, you can measure your usage. And that's the only reason you can measure is because everyone goes through this kind of funnel in traffic and then, you know, usage. But if you don't have that, then how do you measure it? compromising the technology to be able to measure it. But that would be compromising the technology. Yeah.
I mean, there's, there's maybe there's maybe other things where we can infer things, you know, very anonymously without compromising the technology. Right. Yeah. But it's it's definitely, an important, an important, principle to keep that we don't we don't compromise it. Yeah. but I think you said about being anchored in the centralized way of thinking, like, 100%, pretty much everyone is.
I think it makes sense. Like myself, the systems we use, of course, I myself also. And I still have these things where I go, no, wait, I don't need to do that anymore because x, y, z. Right.
and actually what you realize as you go, as you start to go down this like peer to peer path is there's, a, an unbelievable amount of complexity in the current status quo that we all accept as reality. when you go peer to peer, things get simpler. Do they? Yeah, that blows my mind. Yeah. Because I was thinking, okay, how do you do this? Like on a software level specifically? Okay.
So somewhat sometimes on the software level things don't get simpler. But once you get it to a use a place or a way of reasoning about things, you have to go. So it goes through here. And then that goes with that. And it's just like I used to be, you know, rightly.
Yeah. Yeah, yeah. But from a software level you were saying. Yeah, that's what I mean. That's where my mind was because I know you're in the driver's seat and you're creating part of the protocol as well as using it for this application. Not so much the protocol.
My, my thing is the, the Pear Pear runtime. Yeah. sort of a platform for both running and developing peer to peer applications. So, yeah, it's a lot of it's the dev kind of side of things. It's it's pulling all of the technologies together.
Yeah. To make a coherent, journey user journey, for, for, for doing this. Yeah. and would I be able to do that as well?
Yeah. It's really simple, to be honest. Okay. you basically you say this, it's a command line based thing. You say pear run and then a link. And the link can either be a file link like your file URL, or it can be a pear URL. If it's a pear URL, then that's the first part of that is the key. That's for the app that's discovered through the DHT.
Okay. and if it's a file, it runs from disk. So that's running an app to, to get an app to be discoverable through the DHT, you have your, your project folder and you just run path stage and you give it a channel name. And then that generates a key for your app. And synchronizes the disk contents into a hyper core. I mentioned earlier the then you do pear seed and you you use the same channel name once and then it just seeds from your machine.
Then you can go to another machine or someone else can run it and they can do pear run and the pear link that the stage command in the C command gave you. Same thing. that's it. Okay. That's it. That's the whole thing. there's other stuff too.
So that there's pear released for like managing release markers and things like that. Yeah. So you can say like this. So it's an append only log, right. Every, every file that you synchronize to it increases the length of the log.
So whenever you say okay, well, this length, that's the official release, but I can keep on staging and then I can look at preview releases so I can do Pear Run, check out, staged and check out staging to see the preview release of of a product. but when I just run it with Pear Run and I'm just looking at the, yeah. Beyond that, what we do with key is, we have a downloadable DMG for Mac, and that's what I used. Yeah, yeah.
and all that is, is, a Genesis pear build and that Genesis pear build runs. Pear run does exactly that. Yeah. That's it, that's it. Wow. So, we got we haven't got this yet, but we've got a pear build command coming. There's some complexity on windows
to still figure out because windows is windows. Oh, yeah. anything that takes us like a day on Linux and Macs takes like two weeks on windows. It's a little bit. Yeah, a bit weird, you know, push that back. Yeah, yeah, yeah.
but we, we have a pear build command coming so that once you've, once you've staged your app and whatever, you can just say, okay, now I want to build it and it all it does is create the same, wrapper files that just do pear run of your app key, and then users can download it and it just feels like a normal app to them. Same like he does. very interesting. I'm definitely going to play around with it, I think, because I love what you're doing currently.
I love when people are working on newer technologies that might actually be truly disruptive, right? Because people usually label technologies as disruptive, but I think it's always debatable. But I think something like this, which would change the way that we operate, as well as even the incentives and the structures, the huge pillars that we have. and what kind of decentralized them, I think is fascinating.
Yeah. Because within all of that, that I just said, there is no infrastructure. Yeah.
Other than the machine that runs it and the machine that cedes it, that's it, that's it. That's a peer to peer situation. Yeah. with I did quite a bit of consultancy and in order to, you know, you had to you had to have a bit of a margin at the beginning of a lot, of, a lot of consulting gigs because you, you first few days you can't do anything because you have to figure out, oh, oh, there's a VPN. Oh, okay. Oh, yeah. so, oh, I need to set up my exact house file so that redirects this IP to that.
Oh, okay. Who gives me. Oh, how you doing? Oh, they won't give me access because I'm not on the employee register. Oh. Okay. So how do I get around that. You know, and, and so like, there's so much friction in the way even just for onboarding new devs into, into centralized companies.
the wastage, the, the cost on, on, compute and all of that. It's, it's really wild. I think in some countries I think in the USA and so forth, it's written off as, you know, you got CapEx and opex, right. And the tax incentives on, opex are different on CapEx. Right? So, to a certain extent, the there's even within and I don't fully understand this, but like my I might be wrong on this, but my understanding is that the, the, the systems themselves sort of, incentivize this approach of just spending tons of money on infrastructure and staffing infrastructure.
which is interesting because it's like, you know, a centralized government is, is, is trying to incentivize centralized companies. But I don't think that was an explicit, attempt to do that, but it's just an inherent bias in the system. Exactly. How did you end up kind of collaborating with hole Punch and doing what you do now? Because I think it's fascinating, this vision.
And it sounds like you're really driven by it. Like I talk to you now, I talk to you in previous conversations. I love the energy and the passion that comes from it. But I'm really curious how you ended up here, because I I've seen this kind of on the sidelines, but you dove head in.
At first it feels like. Yeah, I mean, it's kind of a long story. I would do in the fall, I guess. so I, I started programing when I first saw a line of code when I was five years old. and, this, this guy who was into computers, came round and plugged, a Commodore and and he wrote ten print. David.
20 go to ten. And I was like, mine blown. I was like, whoa, that's my name.
And it's doing it over and over again, and all you need to do is type that. And like, I was thinking about that for a long time. when I was nine, I got my hands on, an Atari, and, I had a book fair that's like a garage sale, beef flea market, all the same sort of thing.
Okay, people who have different things, but where I, I grew up in, the southeast and the corned beef as there. Yeah. and, of, of the UK and, the and I got and I got my hands on a programing basic book. So I sort of taught myself basic through, you know, a nine. Yeah. Incredible. and then when I was 12, as I said, like, I 5.5 had just come out, right.
and I had a single single mom, but she sort of recognized my, like, pension, you know? So, yes. The computers, and she, I think she had two jobs and she had two jobs. So she got me the computer, and she got me a dial up modem. Yeah, I'm back then, like, dial up was super expensive and stuff, so I dial up, load a page. back in the day, like, there were these websites called JavaScript goodies and HTML goodies.
and I use those websites to learn about JavaScript, CSS. Okay. And I just like dial up, load a page and then cut it off till I try and save money.
I later found out that the dial up is the most expensive part. You to do that over and over. Yeah. that's hilarious.
And like, throughout throughout my childhood, I mean, I destroyed, my uncle's computer, I destroyed my grandma's computer, I destroyed my own computers many times, and I'd always just be able to, Well, I would destroy something and then fix it. And that's sort of how I learned as well. but I definitely, I definitely, irritated, the extended family with my shenanigans, but on a software level, or, like, physically destroyed the I didn't take a baseball bat or anything like that.
I would just be like, oh, what's this? Can I have it? Go on your computer, please? And then I'd be like, oh yeah, sure. He's a bit of a whiz kid, you know, and I just. And then I'd get into like, oh, what kind of, what kind of computer is this? Like? My uncle is an engineer and he had some kind of, I don't know, I think it was maybe Posix based or something, but you could run lemmings from the command line on it. And I was trying to do that.
And then I just got distracted, you know, and I was like, what's this do what's I do? And then the next thing broken, you know, I was the driver of curiosity and how powerful that can be. Absolutely incredible. I think ultimately that's the question of how I ended up here. Like, that's the ultimate route, you know, following curiosity and also just saying, like, hey, you know, just do we have to be like this, you know, do we have to? I don't think we do.
Like, you know, humans aren't bees. Bees have to be bees. They have to be worker bee. They have to be a whatever be they are. But humans self-organize and take up roles as they see fit and like so I don't know, I just think there's a really interesting connection between technology, communication and how we organize ourselves, you know? Yeah. but so, so skip forward to like, about age 23 or so. Okay.
I, I didn't realize that being able to do these sorts of things with computers, with computers was actually a valuable skill. Okay. in my head, I was just like, well, anyone can do that. It's like washing dishes or. You know what I mean? It's probably normalized so early on. Yeah, it was just normalized. Yeah.
So I mean, I was working in McDonald's, and then, I worked at it while I worked at various of places I worked in, like, a bank for a while as a cashier and like, McDonald's. and then after that, I worked in, a homeless hostel, and it was 12 hour night shifts and 12 hour days. I do two 12 hour days and two 12 hour nights and, and on the nights I started to build a website for the hostel, which was, which was cool. And then I start and then I started getting like, people sort of pay me to do websites for them, and I sort of do that little, little bit on the nights because like, you'd have like this five hour sort of period where everyone's asleep, not always because lots of things sometimes went went mad because it's a homeless hostel. But like, sometimes, and then, I started thinking I was doing PHP and stuff, and then I started thinking, I need to, I need to find a way to, to make these, to make these applications more real time. You know, I started looking at Ruby on Rails, and then I came across Node.js, and at this point it was about 0.8.
Okay. and I was just like, whoa. and at the same time, I came across Bitcoin and Bitcoin at the time was like £0.02 a Bitcoin. I really yeah. And I looked at both of them. I was like, well I don't really have much time.
So I can only really go in one direction. And I'm trying to like, you know, build websites and stuff. So I'll go I'll go look into the node thing and I'll leave the Bitcoin thing like decision right there. I even at the time I was like, oh, if I can just put a ten, 10 pound into it or something like that.
But even at that time 10 pound was a lot to me. for various other reasons. Right. so, so so I went into looking into, to node and at the time there were no books on it. there were, there were, but there were just rehashes of the docs and the well, one of them was just garbled nonsense. The other two other were rehashes, the docs.
So I contacted the publishing company because like, well, I'm looking at the raw code here and look at the raw code of express and like figuring it out. so I wonder if I can get a publisher to just pay me to do that, and then it'll incentivize me to do it. so I did a proposal with, with, PAC publishing and then, it's just, I think the timing was just right, and there were absolutely.
Yes. Let's do it. So I ended up writing Node Cookbook. and then I did three more editions, over the years.
Awesome. But with that, because I was living in Northern Ireland at the time, I did a bit of, asking around on the Google because at the time it was like the Google forums, Google or something. Yeah.
and they said, oh, there's actually a conference happening. Someone said, there's actually a conference happening in, in Dublin. And it was it was the first node conf. Okay.
And so I, I can't remember exactly how it all happened, but basically I went down and they, they gave a shout out for the book while I was there, you know, as a thing. And because you were there or just in general. Well, I sort of talked to them before I went down and they're like, come on down and do a shout out for the book stuff, you know, good stuff. at the time, I was working for this company called pace, which do, like, this guy set top boxes, like TV, set top boxes and stuff like that.
Okay. And, they had interfaces in JavaScript and HMO. Yeah. Nice. So I was building like, sort of Netflix style kind of navigation stuff for them. but the people that organized know comp was a company called Near-tum, and their form were very early in that in, at that time. but they, there were, they were bullish on node and there were shortly very shortly after bullish on microservices.
very ahead of the wave there. so I ended up working with them through knowing them from the conference, going down and speaking at meetups and different things like that. And they they hired me as the head of training. Gotcha.
but I ended up going way further into the consultancy side of things with the consultancy team. so therefore, at the time was building, really like really cool stuff for, for companies that needed, you know, to update. and we went into sort of the realm of digital transformation and that's like, so you're going from like node, because node is, JavaScript in general, as an invented language is very good for mediation. that's why that's why Pear's also JavaScript. So the being able to mediate between, within an event ID and an event loop within an event ID language, the cohesion is weak, right? so, once you start thinking digital transformation as you come across Conway's Law and Conway's law is the, the idea that, there's a relationship between communication and technology. If you have dysfunctional communication in an organization, that organization produces dysfunctional takes up technology.
And I'd argue that most of the technology we use today is and in some ways dysfunctional. and, it just dysfunctional to others as, as people, some in some cases, but not necessarily dysfunctional to the organization because it's intentional that they, for instance, don't allow you to, give feedback on certain things. They control the app experience to that level, right? That sort of thing. but even even the unintentional stuff leaks out, you know? So what we did with the digital transformation stuff was outline a path for companies to move on to a technology, architecture that that tweets the way they communicate at the same time. And if you can do that, you can kind of double bubble it where you you give them you give them the digital transformation.
They need the technology, but you've also got to work with them, at the, the sort of front lines and the executive suite levels, to basically be like, this isn't just technology, this is culture. Yeah. and honestly, from my perspective, you need that to kind of grease the wheels because it's like the, you know, when something is very, very large.
I don't think you need that. But when it's, certain scale, even when it's quite large, you still need that sort of greasing of the wheels, as I say. But but I think from my observations, what it seems like to me is that a lot of the time it's to do with the major factor is a technology change, because executives change over time. Front line is change over time. But the culture and the way things are done are constrained by the technology. So one of the things that we did, I did quite a bit of was the was creating, conventions of how to develop within an organization.
Okay. But then enforcing that through actual tools and code, which I'd written specifically for that purpose. So, or use from the open source world, like a really, really obvious one is like a pre-commit hook, which, as an example, like if you if you need to lint your code before you commit it right now and you say, okay, everyone, we're all linked in our code before I commit it.
Cool. Everyone goes, yeah, no problem. And then no one learns the code. Right? And so okay, so you put CI in maybe and then now you're constraining it so the indented code doesn't get into the code base. But you're still using up CI resources. By not doing it you take it onto the user's machine, the Pre-commit hook. Sorry, the the pre-commit hook runs and and now they can't commit unless they fix the linting issues.
Yeah. So that that, technology constraint like as a small as small as it is, can have a massive difference as you scale it up. if you have thousands of people suddenly using a pre-commit hook like the, the reduction on CI compute is massive.
Yeah. but also the communication changes. People aren't mad at each other anymore for not linting the code because it doesn't happen as often. Because it's not happen at all. Because they can't commit without unless they force it.
And then that's a whole different. That's what I would. Yeah I mean there's that. But then like that's, that's different to being like oh sorry I forgot. Yeah. No that's like I deliberately. And so now the communications clearer and you can deal with it more.
Yeah. The technology is clarified. and now there's less dysfunction. like that's just a super small example. But you see how, like, something small can have a big effect? Yeah. and so, with all the consultancy and all of that, I also met, I was also speaking at conferences, and I spoke at a conference in Oslo about microservices.
And someone else was that, doing a talk about, Merkle DAGs. Okay. Very specific.
Yeah. so, Bitcoin uses Merkle trees. I've heard of this before.
I hope I'm not me. I'm probably going to butcher this explanation. Okay, but but but basically it's to do with, you know, you've got the blockchain and you've got like, the hashes of the hashes and all that kind of stuff. Yeah. Like somewhere in there there's this, there's this thing called the Merkle tree. That's part of that whole blockchain process. Right.
Okay. Merkle dogs are a kind of inverted tree. something something graph. I can't remember which is.
Which is funny because it was, Matthias Booth who gave the talk, and he's the CEO of Whole bunch now. and we had a chat at the end, and I was like, asking him more questions about the Merkle doesn't, but that's, pretty funny. I can't explain it now, out of curiosity, purely, yeah. Well, I was like, I was like, wow, this is really interesting.
But but remember, like, I'd looked at Node and Bitcoin at the same time and I'd read, the solving of the two generals problem white paper. It's a totally white paper on that kind of stuff. Yeah. Like very, very early on. And so when I saw and talking about this stuff, I was like, oh yeah, that's I've been all doing the consultancy thing. Like what? You know, I would do three month gig.
So I'm going in and out, in and out, in and out, in and out. Completely different. Yeah.
And and so but I'd also I also had an awareness of Bitcoin and what it was doing and stuff like I don't know personally I guess I'm a maxi because I don't, I don't really write the other coins like the breakthrough with Bitcoin. Everything else is like extra. Yeah. It's experimentation and and and
yeah. so, so. That then led to well that didn't lead to it, but then skipped forward a bit further. And Matthias is actually working with form also as a consultant and as, a specialist in general.
Okay. we also did performance work myself, Mateo Cleaner and Matthias Booth were performance specialist as well. Like, how do you make the code really, really fast? That's another thing is like, if you can, if you can speed up the speed at which a technology runs within an organization, then you still have an effect on the communication, right? Yeah. so, so Matthias, had been experimenting with peer to peer stuff from his Danish. Right.
So, like the Danish, the Danish sort of education system is, is fantastic. and you sort of given time to explore and all of that. So he's come out of it out of the education system.
Got his, I think, a math degree or whatever it is. And then he's been like, oh, I like, I like BitTorrent or whatever. And you start messing around with that, right? Yeah. And going from there, fast forward a bit later and he's created this thing.
well, there was this thing called Dot, sorry, which was, which was an academic funded program for transferring large amounts of data because academia has a need, for that. Right. And they, you know, back then there wasn't even really cloud offering. So this was like, really novel.
So, so Dot was born from that. And then the, the, and there's like a whole done community, around this that exists online. That's a peer to peer community.
shout out to the DAC community. but within that there's, hyper core. So the, the, the primitives of that, which after a while sort of sort of dwindled off. Okay. they went into, a company that Matthias then started, called Hyper Division.
Okay. So Hyper Division, was, a consulting company for peer to peer stuff. So they're able to work with, like, crypto trading platforms and all that kind of thing to, to create, like peer to peer systems that work within that company. Because, a peer to peer approach, you know, can actually be a lot more efficient.
and when you're talking about trading in different things like that and you're talking about robustness and anti fragility, these things come into play with peer to peer systems. So that can that continued the investment in into the stock. fast forward a little bit later and I decided I also did this thing with near from where, or with the Open Jazz Foundation, the Linux Foundation. I've written the JavaScript node application developer and JavaScript node Services developer certifications and exams. it saw it as a community project and all of that. But I'm the author and lead.
and it came to a point with their form where it seemed like it was time to sort of step away, but carry on doing that stuff. So I stepped away, carried on doing that stuff, and then went freelance for a bit, just did some freelance sort of stuff with, with I work with. Tell us a little about the Canadian telecom. Okay. some other stuff. and then Matthias contacted me and I was just in-between kind of gigs, and he said, you up to much? And I was like, no. I, I'm from now I went to Denmark and met, this guy, our CTO called Andrew.
Okay. Andrew. well, he's, he's from a, sort of technology neuroscience background. and his really the, extremely good, algorithmic kind of programmer. Okay. in terms of the peer to peer algorithms and stuff, as is Matthias, but they bring different, different things. Right? Interesting.
So, we started talking then, and there's one other guy, sorry, who wasn't there at time, but this is the other guy that's involved was, Chris, Didrikson. He's like, guy from a chemistry background who, found out that he really liked, thinking through cryptographic algorithms. Well, yeah.
so he's our cryptographer, but he can also code. so he's a coding cryptographer, whereas Andrew is a coder who writes the, the things on top of the cryptographic algorithms and like, the, the sort of communication algorithms, and a bunch of other stuff. He's the CTO. So anyway, we started talking about a microkernel.
What's what we thought at the time. Okay. Which is, essentially what, what Pear became. And it's just a platform for a cross-platform course operating system, platform for running peer applications or for running peer to peer applications.
We we didn't call it power at that time. I love the name, by the way. Yeah. Me too. Vampire. Yeah. So, yeah. It's so good. Yeah.
that was, that was Paolo, our, chief strategy officer. he's, CEO of tether and a bunch of other stuff as well. he came up with that name. but, If, where was I? Sorry. Microkernel. Micro. Yeah. Sorry. So no worries. So we started talking about that, and that's, that's, that's sort of where we sort of formed the, the basis of what we were going to do, with Hole Punch. And we hired, our first front end guy.
JB right in the early days. And what I did was, is I just wrapped electron in a command line thing where we before powerschool parrots called hole punch. and you just wrote hole punch dev.
And then that just opened electron with, like, a, file watcher reload thing. Okay, so cheat was started was when we started building, we built it with, with with this hole punch dev command. Just like straight in electron. And so key development could start in parallel. We had immediate utility.
Well, and we could keep on building this microkernel. So, that's what we did. And we and we kept on we kept on with it until we, until we got to, like multi-platform. What multi app. Sorry. so we made it so that pack and run like multiple applications with a sidecar situation. And it started out all it could do was run one one pair up, you know.
Yeah. but yeah, because the thing is, is like it's, it's really it pair of runtime really is, is very sort of it's supposed to be as thin as possible sort of lateral lateral piece of code because most of what we want to do is we want to keep all of the, core functionality and everything outside of the core. We want that in ecosystem modules because we can iterate those as fast as we like, you know what I mean? And like, you've got samba control and all that kind of stuff is, is the normal sort of development process that people can use. Yeah. And we want to keep the, the core as small as possible because that minimizes breakage. we don't have any like codependency issues or anything like that. Right.
So, you we could stop build once we once we could support the native primitives in electron, for, the piece of stuff. Then we could just use the modules that are already available as opensource modules, like Hyper Cool, hyperdrive, Auto Base and so forth to build the kit up. And at the same time, the, runtime platform is being built using those same modules to, to facilitate like, say path stage and pass seed and those things I mentioned earlier.
Right. All the, all the sort of dev acts and, and, and deployment features. and then once we are at a place where we were like, okay, when was this in time, by the way? 2020 is where, oh man, I'm so bad at timelines. So we we went open source in Feb on February 14th of this year. This year, so Valentine's Day, that's when I saw the first Valentine's pair. Yeah. Nice.
but we decided that we would go live, I think roughly October the year before. Gotcha. And that's where we were like, we've go live now, but we'll we'll do it really well, you know. so, so yeah, that's that's really how it all came to be. It's almost been a year that it's live that also. yeah. Yeah, yeah.
And, we've got a, I think we've got a release coming out today. Okay. 1.40. I should be going out there, or at least this week. but that if you watch me, I'll often say we've got a release coming this week and some not. but, yeah, we're releasing is like at the minute. We've I've had a few like, big releases because we've had to wait for something else.
And so while you're waiting for something else, more things are happening. And so it's like chunking other release. But I want to get it to like, smaller incremental releases as soon as we can. So after this one, probably try and do more rapid releases. Our release strategy is really cool as well, actually, because, when you stage two, a drive, we also have this command called pad dump.
Okay. So we have, a development key that we just use internally and then we dump, so then I use that development key to verify everything's fine. So I run from that key, and I can dump from that key into another folder.
and then I stage that folder. So then that folder is now my staging key. Yeah. Which is like what we use, like for everyone in the organization, QA and everything checks that. Everything's working on the staging key. And then we dump from the staging key to the release candidate key.
same thing, same process again. But the release candidate keys different. And this is something that we need to put into the dev of pad. We don't have, like a really easy way to do this, right now, but the release candidate key is a multi-sig key.
Okay. And we've got this. We've got a key room that, where the stakeholders are in the key room, and you need three out of five signatures in order to sign off on a release on that key. Interesting. sorry. This is.
That's sorry. That's the not the key. The RC key is the one before the production key. The production keys. Multi-sig.
Sorry. there we go. Dev key. Stage key. Asking release kind of gets gets the most rigor applied to it. Yeah, and that's the one with the change locking. That's the final thing. and then that goes on to the production key, and then you have to have three out of five, mode, six signatures. Again, this is all decentralized.
It's all completely peer to peer. Yeah. and that and that's how we control gate production. Nice. Yeah.
Was that there from the start or how did it grow to that organically? Because that's quite an elaborate approach. And I love the multi-sig at the end as well. Yeah.
So it started out with just my de
2024-08-27 06:04
