{ "episodeId": "SLP588", "speakers": { "stephan": { "name": "Stephan Livera", "role": "host", "tag": "STEPHAN" }, "orange_surf_from_the_mempool_space_team": { "name": "Orange Surf from the mempool.space team", "role": "guest", "tag": "ORANGE" } }, "segments": [ { "speaker": "stephan", "time": "00:12", "start": 12.19, "text": "Hi everyone and welcome back to Stephan Livera podcast, brought to you by Swann dot com, the best place to buy Bitcoin. Joining me today is Orange Surf from the team over at mempool, mempool dot space that is, and mempool dot space also have a research report that's out on empty blocks, so we're gonna be talking a bit about that. A lot of people are curious, well, hang on, why did these things happen? So joining me today is Orange Surf. First of all, welcome to the show." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "00:37", "start": 37.26, "text": "Thanks very much, Stefan." }, { "speaker": "stephan", "time": "00:38", "start": 37.96, "text": "It's great to be here. So let's start with this, what spurred this research report from you and the team? Yeah, it's a good question." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "00:46", "start": 45.73, "text": "So at Mempool, we, we get lots of questions from users of Mempool.space, and people running their own Mempool instance, about why they're seeing these empty blocks appear on the block timeline. and I think especially if people are waiting for confirmations on a transaction that they really need, to get confirmed, it can be a bit frustrating, and a bit surprising to see that there's a block that comes in which has no transactions in there apart from the Coinbase. And so we've, you know, generally tried to direct people towards a very small, succinct answer that explains it. but recently there was a bit of discussion about whether that answer was sufficiently detailed and whether it really captured the nuance, and we thought actually the best way to really address this was to just do a detailed analysis of the historical trends and explain, \"This is what empty blocks are, this is why they've happened over the years, and this is why they're happening now,\" and to basically just give a bit more space space for these kind of ideas to breathe rather than just the, the small soundbites that people can shoot back and forth" }, { "speaker": "stephan", "time": "01:47", "start": 107.15, "text": "online. Yeah, okay. And so your process of researching this question also was quite interesting. I mean, obviously we can get into the report and the report will be linked in the show notes for listeners, but, can you tell us a little bit about the process there of trying to come to an answer here?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "02:03", "start": 123.08, "text": "Yeah, sure. So my background's in mechanical engineering, doing a lot of different data analyses, and the, the kind of starting point for this was just to collect a load of data, in a nice kind of manageable way. So we wanted to have data for every block on three different things. So we wanted to know was the block empty or not, which is really easy, you can look how many, transactions are in the block, and if it's only got one, then it's an empty block. the next thing was which pool, mined the block" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "02:33", "start": 152.74, "text": "Information using just data that we can see on chain. And so we've got the Coinbase address and we've also got the Coinbase signature, and they can both indicate to us which, pool mined a block. Now, the Coinbase address, in theory, a pool could direct hash towards, their own pool, but they could pay a small amount of Bitcoin to the Coinbase address of another pool, so there's a potential there for a false positive, similarly with the Coinbase signature, in theory, a mining pool could specify a different pool's Coinbase signature, but really there's no real incentive to do that. if someone didn't want to announce to the world that they were mining, they could just not include those, well, they could use a different Coinbase address regularly and they could, use a nondescript signature. So we're quite comfortable that we can identify the vast majority of blocks, which pools mine them, especially over recent years, whereas back in the early days of Bitcoin, things were very different and it was much more common for individuals to mine without specifying, you know, a particular tag in the Coinbase, signature, which makes it very hard for us now to retrospectively see who mined them. So in the analysis, we had early blocks with very High percentages of unknown mining pools, but in recent years it's ninety-nine percent odd, we're able to identify. And then the third thing was we needed to look at the block interval, which is how long it had been since the previous block was found, so the duration effectively between blocks. And, there's a great data source for this, from, zero x b ten c, and he, or she had managed to pull together lots of different sources of this block interval data, and aggregate it together. So Mononot helped me there, pull all this data into one dataset, and then we were able to just start looking at it from different angles, and try to learn Were there any interesting trends here? And we tried to look at it with a very neutral lens, just going off the data, and over time, we were able to build up some interesting insights into the empty blocks and how they've changed over time." }, { "speaker": "stephan", "time": "04:34", "start": 274.45, "text": "Yeah, interesting. And so, yeah, again, we'll, we'll do our best to keep this accessible for, listeners. and so Just the kind of super basic stuff. In Bitcoin, on average, there's a new block every ten minutes on average. Of course, it can be faster and can be slower than that. And in practice, what we see today is this world where we have individual miners, or some people call them hashers, and then mining pools. And then, as you've said, those, Those mining pools are being identified at the block level by looking at what address the Coinbase, you know, payout is going to, and then also the signature, because as you said, in early years, there You know, there wasn't mining pools until I think a few years into Bitcoin, and so I think as I'm understanding nowadays, the pools basically just openly put that there so they can say, \"Hey, look, I got this, you know, I got this block,\" and maybe there's a little bit of an advertising aspect of it too to kind of show off, \"Oh, hey, come and join my pool,\" because obviously the pools are competing with each other, they want peop-- they want miners to point their hash rate there, right?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "05:39", "start": 338.73, "text": "Definitely, and it makes it very easy for hashes" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "05:45", "start": 344.78, "text": "and they know what proportion, they were providing of hash rate to that pool, so they can get an easy estimate of, you know, if they provided one percent of the hash rate to a pool that earned a hundred Bitcoin, then they expect to earn a hundred Bitcoin less the fees. And so it also makes it quite easy to track that data over time." }, { "speaker": "stephan", "time": "06:01", "start": 360.99, "text": "Right. And I guess it arguably helps with transparency for, individual miners or hashers to understand, am I getting roughly something in line with, you know, what the pool is winning, or am I-- or are they somehow I guess that's one thing that you could use there on the, at the individual hasher level. Yeah. Yeah, definitely. Okay, so let's get into the, the obviously the topic, the obvious question, what is an empty block? Sure." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "06:26", "start": 385.62, "text": "So an empty block is a block that contains only one transaction, but it's a special transaction. It's a transaction that creates new Bitcoin. this is the Coinbase transaction and it allows the miner to, send to themselves out of nothing, if you like, the block subsidy, which at the moment is three point one two And so that is a unique transaction in the block that doesn't require a corresponding input value which matches up, and that's how the, the miners are able to earn the vast majority of their revenue from mining today. Normally in a block you'll have other transactions that are Bitcoin users sending Bitcoin to each other or to themselves, and there's typically a few thousand of those transactions as well. But an empty block doesn't contain any of those, it just contains this payout to the miner. And historically there were You know, about ten percent, if we look back through the whole of the history of Bitcoin, ten percent of blocks had these, this unique property that they didn't contain any additional transactions, but more recently in the past two years or so, it's varied, but it's been less than half a percent of blocks have been empty. And so When blocks are empty, it's, the, the work is still being used in that it's confirming transactions in, like, deeper into the chain. So if you had one confirmation and then an empty block was found, your transaction would now have two confirmations, but it isn't adding any new transactions into the chain. So it's confirming stuff that's already happened, which is obviously good if you're, you're waiting on, say, six confirmations for a lightning channel, but it's not great if you're expecting to get your transaction confirmed in the next block. And then an empty block gets found, which is often" }, { "speaker": "stephan", "time": "08:01", "start": 480.99, "text": "why we get asked the question. Okay. And so another obvious question, why do empty blocks happen? What's the normal cause?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "08:09", "start": 489.24, "text": "Yeah." }, { "speaker": "stephan", "time": "08:09", "start": 489.48, "text": "So" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "08:10", "start": 490.0, "text": "in the early days, I think it was the first, hundred thousand blocks, eighty percent of those were empty. and that was just because basically no one was using Bitcoin. There weren't any transactions waiting to be mined since the last block was found, and so they were empty just because there weren't any transactions available to go in. But obviously nowadays, there's always- Is someone who's sent a Bitcoin transaction since the last block was found, and we've obviously got a mempool backlog that's been going on for quite a while now. So we, we definitely know that that's no longer the case. so the explanation really is just, economics. So if you send an empty block template as a pool Then your miners or your hashes, I should say, are more quickly able to start doing useful work, work which if they produce a block, it will be a valid block that builds, builds on top of the chain tip And extends the chain and earns fees. So that will happen slightly quicker if you send an empty template than if you send a full template, because a full template also includes within the job that's sent to the miner a Merkle branch, for, well, a number of Merkle branches that can be used to compute the Merkle root. And so If you send people, if you send your hashes, a full template instead of sending them an empty template, it'll take slightly longer for them to get started and you'll earn slightly less Bitcoin, all else being equal. But the nuance comes in when actually, if you send an empty template, you can replace it with a full template. So you can send an empty template first and then immediately follow it up with a full template. And if your miners switch very, if your hashers switch very quickly to this full template, then you'll have the best of both worlds. You'll earn the extra small amount of revenue if you can quickly find a block while that empty template is being worked on, and then you'll earn the kind of the full fat block with all the transaction fees if you mine it after that period of time. but the reason that this has been a bit controversial, I think, is, it's, it's all quite nuanced. If you send an empty template and follow it up with a full template, there can be quite a delay be-tween-before the individual mining ASICs will switch over to using the new job And we'll touch on this later in a bit more detail, but that at a high level is, is kind of why people are sending these empty templates, and it's a choice of the pool. A pool, if they want to, can choose to send only full templates, and they'll find absolutely no empty blocks, and there's an opportunity cost there in that they'll ever so slightly delay how long it takes their hashes to start doing useful work." }, { "speaker": "stephan", "time": "10:41", "start": 641.35, "text": "Yeah. Okay. And just to sort of zero in a little bit on the useful work point that you made, which is that Ultimately, you know, a very simplified way of thinking about Bitcoin mining is it's like a decentralized lottery, and so people have got this, you know, their ASIC machines, their SHA-two-five, double SHA-two-five-two-five-six, and they're, they're doing this hashing to try to- Target to basically find a successful block. And so, I guess what you're referring to there is this idea that until the individual hashers have been given a valid, template from the pool, at the pool level, they're not doing any useful work, and only once they've received that Then now they're actually doing, quote unquote, as you said, useful work, and there may be, you know, a few seconds of lag, but, you know, once you kind of compute this out across hundreds of thousands of Bitcoin mining rigs and machines all around the world, it's meaningful, right?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "11:35", "start": 694.81, "text": "That's it, yeah. So if you, if you look at the way that miners typically work, they have a cache of transactions, of jobs, which are waiting to be processed, because you don't want the individual mining chips to be undergoing a huge thermal cyclic load where they ramp up to do a load of computation and then they cool down, as they wait for a new job to come in. Ideally, you want there to be a, a backlog of, of jobs waiting to go into those chips to be computed. and if you have this backlog The question is, when you send a new job to replace it, do you clear that backlog out and replace it immediately, or do you wait for that backlog to be kind of processed through? And if the time to process the backlog is of the order of, you know, less than a second? Then it may well be worth just waiting for that to work its way through and having the new jobs get added at the back. But if it was gonna take you a minute for those jobs to get worked through, then it may actually be worth Telling the mining, ASIC to just, abort all those old jobs and only work on the new jobs. Now, if you're working on a completely new, block Which is on a new chain tip. So there's a new block that's come in, and the old work that you were doing is no longer going to be valid because there's a new chain tip, and you're working on a stale, chain tip. Then you don't want to do any more work at all. So the miners will send a clean jobs equals true flag that basically tells the ASIC, \"Drop all that old work, you're not gonna get credited for doing any more work on that old job, just immediately switch over.\" But in this scenario that we're describing, where you switch from empty to full templates, it's not as extreme as that. The old work is still useful in that if it's still valid. If a block gets found using that old work, then it will still be a valid block. But if you don't switch quickly enough, then you'll continue to expect to earn slightly less Bitcoin. So there are some miners that will send a clean jobs equals true Flag just to clear everything out and switch over, not just when they're switching to a new chain tip, but when they're trying to get the miners to switch from a, an empty job to a full job, if the differential between those two fee revenues are, are significant. If it's a very small difference, then it's probably not worth sending" }, { "speaker": "stephan", "time": "13:57", "start": 837.01, "text": "that at all. So let's get into some of the research report content as well around the differences in empty block performance. So as I understand, you, you were trying to go through and statistically analyze, okay, what is the actual- Empty block rate, but then also you have to compare that with the rate at which these pools find a block, which is also again correlated to what percent of the network hash rate do they have. So can you elaborate a little bit on that process and, and what were some of the, the insights you found there?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "14:29", "start": 868.61, "text": "Yeah, sure. So if we were to go just off which miners found the most empty blocks, then it would- Basically unfairly, target the miners that are finding lots of blocks, because even if you've got the same percentage, empty blocks, because they find more blocks, they would look worse. So we thought it's only fair to compare the, the rate at which, miners find empty blocks as a percentage of their total. And so When we did that, we found that we're-- there were almost three different categories, three groups of miners. There were some miners that found absolutely no empty blocks. So in the past two years, about thirty-three percent of the, the hash rate, hasn't found any empty blocks. And so these are pools, that include Foundry, Maro, Pegapool, Nicehash, Whitepool, and, and a bunch of others, and they all add up to thirty-three percent of the hash rate. So it's quite a lot of the hash rate is choosing to forgo this extra potential fee revenue. But then you've got another- So before" }, { "speaker": "stephan", "time": "15:27", "start": 927.02, "text": "you move on there, just quickly, are they-- So I guess that's the implication, they are quote-unquote leaving some on the table by doing that, right?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "15:34", "start": 933.89, "text": "Yeah, yeah. So if all else was equal and they were basically just only sacrificing the extra fee revenue from, if they, if they were to send empty templates, they would earn slightly more. So if everything else was equal, they would be foregoing that. But Because of these other complications that we'll discuss, it, it's actually not quite so straightforward, and actually some of the pools that are, mining empty blocks are mining more empty blocks than you'd expect, so they're also foregoing revenue, but they're doing it in a different way." }, { "speaker": "stephan", "time": "16:05", "start": 964.94, "text": "I see, yeah. So I guess the answer is it's kind of a bit complicated. There's not really like a black and white simple answer there." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "16:10", "start": 969.66, "text": "Okay, go on. Yeah, like, we'll, we'll, I think we'll get onto that. Yeah. So the, the second group is They're, they're mining, they're sending empty block templates, but they're not finding an abnormally high number of empty blocks. So they're finding less than one percent empty blocks, and this is the vast majority of all the remaining pools, so about the remaining seventy percent or so. And They're sending empty jobs, they're following that up with full jobs, but occasionally they're finding empty jobs like twenty seconds after the previous block. So this is when, you know, alarm bells should be ringing. It's been twenty seconds since the last block was found, You would expect that these miners would all have switched over to using the full block templates that they were presumably sent by now, but for whatever reason, they're not finding full block Full blocks, they're finding empty blocks. And so these are, F2Pool via BTC, Binance, Luxor, and Ampool. and there's two additional pools that also find these empty blocks with s-significant delay, but they find far, far higher proportions of empty blocks. so this is Spiderpool and Ocean, and there was lots of controversy just while we were making this report about Ocean and whether, you know, they were producing a, an abnormally large number of empty blocks. And so the point you alluded to about how does the empty block rate compare for a mining pool compared to the rest of the network, this was to try to kind of normalize it. To say, okay, if, of, on average, people are finding one percent empty blocks, if we look at each pool, are they finding significantly more than one percent empty blocks or significantly less? and you can do a statistical test that I won't go into the details of now, but it basically says like, is this result really unexpected? Like, it could just be that unfortunately for this pool, they found lots of empty blocks and that was just unlucky, but it's not statistically significant. And so we did this statistical test And we only found that two pools were statistically significant, and that was, as I mentioned, Ocean and SpudPool." }, { "speaker": "stephan", "time": "18:16", "start": 1096.47, "text": "I see. And so one other area that came up, and I think, Bitcoin Mechanic from OceanPool was talking about this, I'm sure you're familiar with it. H- he was coming up with this I guess theory or answer, he was saying that there was a problem with, well, I guess problem is, subjective, but he was, I think in his conception, he was saying Antpool has this particular firmware for their Bitcoin mining rigs that they don't necessarily update to the latest block, and that that was his reasoning for why You know, this thing exists. Now, I guess on the other side of that argument, there were people saying, \"No, maybe that's more like a software configuration side, that's like an error on your side,\" or \"Is there some other answer here?\" I'm curious, what your thoughts are there?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "19:00", "start": 1139.61, "text": "Yeah, so I think The speculation was, an interesting one because the implication was that Ampool had a minor firmware version which they were using internally that didn't have this delay. It was able to switch over from mining on empty work to full work as soon as the new template came in And by doing that, they were able to maximize their profit, whereas the implication was that they were releasing a version of this firmware, that deliberately didn't switch quickly, that would cause anyone else using the firmware to have a, an economic disadvantage. and so we looked at that and we found actually the percentage of empty blocks found by Ampool, has been increasing recently. So We think if, if this was true, then actually ample would be finding fewer and fewer empty blocks, but actually we're seeing the opposite. So it's possible that the, the producers of the firmware and the, Ampool themselves, you know, they're, they're not the same entity, Bitmain and Ampool, so there, there is a proportion of Ampool hash that is presumably operated by the people making the firmware, but it's not a hundred percent, and there are other miners who are contributing to Ampool, so it could be that it's getting lost in the noise a bit there, but we didn't see kind of a smoking gun that you'd expect if, if this was to be the case. and I In sending the jobs with this clean jobs equals, false flag and not forcing the hardware to, to flush the cache and to start working on the new jobs. The, the proba-probably most likely thing I think is that Ocean and maybe even Spidrop have a large proportion of hash rate which is, using this particular firmware or Devices that are using similar versions of firmware and that because they were unique in that they weren't sending this, flushing with clean jobs equals true, they It kind of had an unusually high percentage of, jobs that were being found that were empty. So, might not have explained that very clearly, but if I don't think necessarily that we can attribute it to malice, I think it could just be that the firmware did have this kind of issue, but that the other mining pools weren't experiencing the issue the way that Ocean was, because they weren't doing the same, they weren't, didn't have the same approach in terms of sending clean jobs equals true." }, { "speaker": "stephan", "time": "21:30", "start": 1289.9, "text": "Back to the show in a moment. This show also brought to you by Mempool.space, the leading Bitcoin and blockchain visualizer. I use it all the time when I'm checking transaction fees and trying to understand And what is the state of Bitcoin's mempool? You can search transactions, historical as well as unconfirmed ones, and also see a great way to visualize things across the Lightning Network, Liquid, mining, and so much more. Also, for those of you with an enterprise, they offer a mempool dot space enterprise program. So for those of you as part of that enterprise program, you might wanna get increased API limits, and you might wanna have increased access to the team in terms of feature requests. want to have special branding in terms of how your custom instance of mempool.space looks. So to sign up for that, go to mempool.space/enterprise. This show brought to you by CoinKites dot com, the creators of the best Bitcoin hardware security devices such as the Coldcard Mark IV and the new Coldcard Q. Now, we use Bitcoin hardware security devices to keep our keys offline, our private keys offline. Now, the way these work is you can do that setup, write down your twelve or twenty-four Four words on those, the seed word cards and keep that secure. Now you can use this device to interact with the Bitcoin network using software such as Sparrow Wallet, Electrum, or Becto Desktop or Nunchuk as a few examples. Now you have a range of security features that you can use with these devices such as passphrase, you can use seed XOR, or my favorite is multi-signature. Now if you're starting in a basic way, just start with the device and the USB-C cable You'll plug it directly to the computer and use it that way, and then later improve your setup. But I believe these devices are great at helping secure your coins, especially as you start to migrate up into multi-signature security. But don't be disheartened or don't be, scared away. They are accessible, and I think you actually do learn about Bitcoin in the process. So to get yours, go to coinkite dot com, use code Livera to get a discount on your cold card. And now, back to the show. I see. So it's a, it's difference then, like that other pools were using clean jobs equals true, and we're, I guess, you're saying Ocean and Spiderpool were essentially configuring with clean jobs equals false." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "23:50", "start": 1429.93, "text": "Yeah, I think maybe a combination of that and the fact that they had maybe a high, higher than normal, higher than average proportion of their hash rate that was using those devices. So just like the perfect storm of these two different things just coming together might have made them particularly uniquely- Set up to have a very high empty block rate. so we, we had, we did see that, for example, BrainzPool, which presumably a lot of their miners are using, like a BrainzOS, different firmware, we did see that although they were finding empty blocks, they weren't finding them with this more than twenty-second delay. So it's possible that, for example, BrainzPool, would be able to provide a bit of an insight there as to whether they had, you know, a, a, a configuration option in their firmware And move on to the new work more quickly. and it'll be really interesting actually if there's anyone out there who's willing to kind of have a talk with us and, and explain this, because it's all, speculation really at the moment. We're able to measure certain things using data on chain, but at the end of the day, if, if anyone is willing to kind of explain these things using their kind of knowledge from behind the scenes, then I think that would really help everyone understand what's going on. I see," }, { "speaker": "stephan", "time": "24:59", "start": 1499.09, "text": "yeah. And so I guess bringing it back to the question around who, who are, are there pools that are quote unquote leaving something on the table by either sending out these empty, or I guess the implication would be maybe, there are some pools leaving something on the table by not sending out the empty block template and only sending out a full block template at all times, such that they only want to have a, blocks that have transactions in them versus other pools who maybe say, \"Oh, I'm willing to kind of take the-- I'm willing to just kind of take the more, the increased risk of an empty block by sending out an empty block template.\" Yeah." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "25:37", "start": 1537.3, "text": "And, and actually, it all comes down then to, what is the fee percentage as a total of the block reward? So at the moment, we're about three percent of, the block reward is coming from fees. And You can play around with the different parameters, but with the numbers that I plugged in in terms of how long it takes to get an empty block out to miners and how long it takes to get a full block, to switch over to a full block, if, if it takes more than, if you got three percent Fees coming from three percent of the block reward is coming from fees, then you can wait for, over fifteen seconds for the miner to switch over. But as you increase this percentage of the block reward coming from fees, that time that you're able to wait decreases. So when you get up to ten percent fees that we saw very recently, you were only able to wait for seven seconds. If you, if it took longer to switch than seven seconds, then it would be more profitable To go with the other strategy of just sending only the full templates, and if you get up to fifty percent fees, maybe, you know, in the, in the future we'll see a much higher proportion of the block reward coming from the fees, then you'd have less than one second of allowable time to switch over. So Whether or not it's economically the kind of rational decision really just depends on the fee level. And so I think, if you look at how long it's taking some pools at the moment to switch over and what the current fee levels are, you might find that at the moment there's lots of, the, lots of these miners who are sending empty templates are doing the economically rational thing. If they could reduce the time to switch, it would be even more profitable. But at some point when the fee level increases, it will become more and more economically worthwhile to just send the full templates, and at some point, you know, it's not gonna be possible to switch over quicker than, than this minimum time, and then everyone will be mining only full blocks." }, { "speaker": "stephan", "time": "27:30", "start": 1650.24, "text": "Yeah, really good explanation. So effectively it comes down to, as you said, what proportion of the block reward is made up of transaction fees versus subsidy, right? So again, in the early days, it was fifty Bitcoin every block That was the subsidy plus transaction fees, and then, as the halvings have happened, it's gone from fifty to twenty-five to twelve point five, six point two five, now to three point one two five. And so over time, theoretically, mo- let's say most of us are expecting that the transaction fee component rises while the block subsidy goes down as a percentage. And so as that process carries out, as you said, it's gonna be more and more Making sense to just really try hard to never mine an empty block because you're giving up all this extra transaction fees. so I guess that's kind of the strategy at the pool and mining level that they don't wanna risk having an empty block because it's so much foregone revenue. now today, as you said, it's about three percent, but as that ratio creeps up over time, maybe in the next-- I mean, none of us knows, but let's say ten years, let's say, at that point, it'll be so much more worth the mining pools To never have an empty block or at least try to do, to not have that." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "28:43", "start": 1723.38, "text": "Yeah, definitely. And, and actually, you can do it on a per-block basis, so you can monitor as a mining pool live how much are you foregoing, by, by having one approach over the other. and you can choose at any point to send the new job with clean jobs equals true, which will wipe out this old work and, and replace it with the new work. And if the difference between how much you'll earn with the old template versus the new template is, is great Then you might be happy to suffer the, the delay there, and there are some mining pools that are doing that at the moment. They're kind of monitoring and, and they're only sending these clean jobs equals true replacements if they think that it's economically the sensible choice. But I think there's lots of pools that aren't doing anything as sophisticated as that, and they're just sticking with one strategy and, and seeing how it'll play out, which I think is, yeah, it's interesting to see that there's this potential, extra profit that you could earn By maybe being a bit more actively choosing. But I think it's also worth pointing out that the overall difference that you like would expect to make, because the fees are so small at the moment The overall extra profitability is, is really quite slim. So, I had a look and only 0.2% of blocks were found in less than three seconds, over the past, I think it was two years. So, very, very small percentage of blocks were being found in this, this narrow window of time. And, and even in less than three seconds, we're still seeing about forty percent of blocks being- Full and, and not, and not empty blocks. So i-it's not even the case that that point two percent would all exclusively be found by pools going with the strategy of sending empties, But it's, it's a very small extra amount, so I think there's been a lot of noise, and part of this research report was to, you know, provide a, a detailed analysis for historical reasons and, and looking back over time, and I think people find it interesting. But at, as a kind of sanity check, it, this isn't a massive impact to the network, it's a small proportion of blocks and the amount of, change we see in the difficulty adjustment is regularly, you know, much more than point two percent, so the difference in how long you get confirmed, just because we're having a, a run where, you know, we're, we're seeing more transactions coming in, so more blocks coming in than we'd expect, and then we have a difficulty adjustment down, like that, that will have more of an impact on how long it takes your transaction to confirm than empty blocks and I, I think no one really ever considers the difficulty adjustment and how that will affect the time. It's always ten minutes. Ev-everyone just considers it to be ten minutes, even though we know if we dig into the details, it will be slightly different. ten minutes is, is fine. And so in the same way, the presence of empty blocks at the rate at which they're happening today Is, is so infrequent that it shouldn't really change how long, you would expect to wait or anything like that." }, { "speaker": "stephan", "time": "31:40", "start": 1899.84, "text": "Yeah, interesting. So it's almost like all this work of, of, not much, not much kind of impact, but I guess in a way you're sort of helping prove out that actually, yeah, there's not, that much of a difference, like at the end of the day, all things considered, because as you said, e-e-yeah, we say ten minutes on average, but actually on average Therefore, the actual rate is slightly faster than ten minutes on average, and we're slightly, quote unquote, ahead of schedule per se. Exactly. And," }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "32:11", "start": 1930.95, "text": "and then when you're waiting for your particular transaction to confirm, even if it's in the next block template, the ten minutes is Is only the kind of most likely time, but it's also highly likely that it will take eight minutes or twelve minutes. So We're, we're basically just, we're, I think a lot of people are arguing about these relatively small differences, and I think there is a perception, you know, if, if someone finds a block that's empty, there's a perception that that could have been full of transactions and that as a result, people are You know, not having their transactions confirmed and that that's not good for Bitcoin. And I think that maybe stems from the fact that there is an attack on, on Bitcoin, hypothetical attack, whereby miners only mine empty blocks and that stops anyone from transacting and using Bitcoin And so I think maybe like historically because the empty block attack was a known attack, there's this association with that and that it's seen to be particularly negative. And if someone's mining empty blocks, you know, twenty plus seconds after the last block, then that, that is worse for the network because those transactions could have been confirmed and instead they're not. But when you've got these blocks that are found very shortly after the previous block, it's, it's not really damaging the network, it's just adding an extra confirmation. and so I think everyone, everyone's right in, in a way, like the, the pools choosing to mine, only full templates aren't really foregoing much in terms of the re-potential revenue, and so that's not a terrible decision, and, and overall, the amount of extra profitability is so small that maybe it makes sense to just keep things simple and just send full templates. And then you don't have to worry about any of these switching times or anything like that. And the pools that are choosing to send empty blocks and then replace them, you know, maybe they're eking out an extra bit of revenue and, and obviously mining's very competitive, and if they can fix this issue with the firmware that causes the delay, then maybe it's that extra marginal, profitability is worth it for them. and then on an individual miner side, you know, if, if you can choose to upgrade to firmware that fixes this issue, it seems like that" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "34:18", "start": 2057.61, "text": "So I think, yeah, that, that's kind of the high level summary of everything." }, { "speaker": "stephan", "time": "34:21", "start": 2061.11, "text": "Yeah, that's great. it's funny because, maybe some of this is maybe a little more inside baseball, let's say, but it seems that, when empty blocks happen, there's kind of, if you're watching social media, some of the industry participants might, sort of heckle each other and sort of-- I mean, the implication will sort of be like, \"Oh, look, there's another-- look at this other pool, they did an empty block, Just banter and not, that important basically based on the kind of how big it is in the grand scheme." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "34:56", "start": 2095.57, "text": "I think probably the reason that we saw this kind of back and forth, in the context of Ocean was because they hadn't found that many blocks and a reasonably high proportion of them were empty. And as, as they found more empty blocks, that proportion, you know, was, was pretty significant, like over ten percent of the blocks were empty. So it was more like, if this continues, then it suggests that there's a, a- problem, and although statistically it might not have been significant when there were one or two empty blocks, once you got to kind of four, it then had a statistical significance and it was quite likely that there was something unusual happening. And actually, it was only from that discussion, or only following that discussion that the firmware update got announced. So I don't know if those things were, were linked or not. so I think the, the discussion is useful, but it's also worth just bearing in mind the, the broader context of it and the fact that it's not, it's not destroying Bitcoin, it's not causing people to wait, you know, an extremely long period of time to get their transactions confirmed. it's just, a kind of a quirk that has happened for that throughout the history of Bitcoin, and is probably gonna become less and less significant over time. Yeah." }, { "speaker": "stephan", "time": "36:02", "start": 2161.7, "text": "Okay. and so I guess the other big topic that people might be interested, just if you have any personal views around mining decentralization, there's been a bit of chatter about, well, obviously the proxy pool, drama. I'm curious if you have any actions on that or any thought on what that means for Bitcoin mining decentralization?" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "36:22", "start": 2182.25, "text": "Yeah, so I mean, it, it predates me, but obviously in the early days of Bitcoin, you didn't have pools, you just had individual miners who were, providing their, their hash power to secure the network and, and earn fees, in Bitcoin. And obviously we've moved away from that quite significantly now, and there's a relatively small group of, of companies who are doing all of the transaction selection And, you know, in the, in the earlier days of Bitcoin, you had thousands of different people, building the templates that were being used, and finding blocks, whereas today you're probably down to probably six entities being responsible for seventy-five percent, if, if you assume each of those entities is independent. And so I think- Although there is lots of interesting work to be done in looking at whether these entities are actually independent or not, even if they're independent, you know, we've still got quite a high, high centralization problem, in my opinion, and, and I think that will change over time. I think it's good that people are discussing it. I think the kind of drive for people to, to look to move their hash rate away from larger pools to smaller pools, is very interesting. I did some research into how much capital would be required to back a pool that was doing FPPS payouts, because for context for your listeners, the reason that there have been, The, the number of pools has reduced over time, is partially due to the fact that it's very attractive for miners, who are providing hash rate, to get paid a regular, consistent amount Instead of suffering the volatility that comes with the, the random process that is mining. And so instead of getting a random payout and some, some months having very little income, another months having a lot, these miners will have an agreement with a pool where they supply hash rate and the, the pool pays them out regardless of whether that hash rate Was associated with finding blocks or even whether the pool, in total, found b- any blocks. And so in order to do that, the pool needs to have a significant amount of capital, to be able to afford, you know, a bad luck streak where you have, you know, maybe many days or many weeks where they haven't found anything but they've been paying people out. And as the pool gets bigger, you need more and more capital. but in doing the research, I, I could see how basically- At about thirty percent or so of the total hash rate, the extra capital that you need, actually goes negative. So you, you need less and less capital because you've got this economies of scale. And so some pools are very well funded, for other pools, they're just using this as a, as a lost leader. They're happy to make basically nothing on the pool side, pool fee side, because they're selling these ancillary services or they have other revenue sources. and so it's really kind of Led to this centralization, and another option would be for, you know, wealthy Bitcoin companies or individuals to work together to provide financial backing for pools that want to do this FPPS payout, and allow maybe smaller pools to, to have the confidence to offer, offer these schemes. and I think we've seen more discussion about that in recent weeks, which is quite interesting. But the, the alternative at the, kind of at the moment, if you're not doing that, is that you just have to have quite a lot of hash rate in order to have- Nice smooth payouts, and you have to have a lot of liquidity available, to make sure these payouts happen. And we've seen pools in the past that have ceased operating, presumably because they, they ran out of funds after a period of bad luck, And so I think that's like a bigger problem than, well, they're both big prob-problems, but if, if we assume that the, and the pools that are listed on mempool.space are completely independent, then I think we still have to identify that that is quite a lot of centralization and it's a small group of pools. but the good thing is there's lots of work being done to identify Solutions to that problem. So, for example, with Stratum v2, there's the opportunity for hashes to go from being just hashes that are working on a template provided by a pool to actually construct their own template. And there's lots of other discussions to have around the pros and cons of this and whether economically it will make sense for miners. but I think a lot of miners will choose to do this from an ideological point of view, especially if it's not significantly less profitable. and we've seen, for example, with Ocean, there are some pools that are-- some miners who are willing to forego the extra revenue that you would earn from, mining certain transactions that Ocean Templates will exclude, it's not the economically rational thing to do, but we are seeing that there are some miners, not a huge proportion of the hash rate, but there are some miners who are willing to do that." }, { "speaker": "stephan", "time": "41:08", "start": 2467.85, "text": "Back to the show in a moment. The lead sponsor of this show is Swan Dot Com, and Swan has a mission to onboard millions of people into Bitcoin. I also work at Swan, helping on some educational content for the team. Now, the team have also put out a new version of the Swan Bitcoin application, which is available on your smartphone, whether it's Apple or Android. Now, this app has a really fast onboarding experience. It's now just a few minutes to go from zero to Bitcoin. So if you are standing there with your family or friends and you might have been having trouble trying to get them onboarded, well, try this now. Recommend Swan Bitcoin, and you can do this while you're standing next to them, they can click through, and most of them will be able to set up and do this in just a few minutes. Also, the team at Swan have rolled out a new promotion. There are zero fees thousand dollars of Bitcoin buys. So this is a great way to go from zero to Bitcoin and in a guided and managed way. So reminder, go to your app store or Play Store and search Swan Bitcoin to get onboarded with Bitcoin today. And now back to the show. And actually, as I understand, there's still four different templates even on Ocean, so it's more, I guess it's a bit more complicated because it's, it's more like there'll be some hashers who choose, you know, the \"quote unquote\" spam-free, templates, but it's still one pool at the end of the day. So maybe you would argue, maybe the argument would be that outside observers looking into Ocean might sort of perceive that, \"Oh, it's gonna be a less profitable overall pool.\" Maybe that's the kind of the steel man argument" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "42:40", "start": 2560.03, "text": "here Pay everyone out regardless of which template they were working on. Right. even if you're working on the most profitable kind of Bitcoin Core standard template, you will earn less than if every-- because there are other market par-- other participants in the pool who are mining with the other templates, you'll earn less. Now, in Ocean's particular case, they've got, you know, they've claimed that the profitability of mining with them versus other pools is actually higher because their fees are currently at zero percent and, and various other things. But at a high level, You know, if, if people are directionally, if, if some people are willing to forego some profit in order to support decentralization, I, I think it seems from what I can see that there's There are some people willing to do that, and at the moment, the only way for those people to express their desire to mine, with their own templates, is to use very small pools. But as we see the So as we, if, if we see some larger pools offering, miners the option to use Stratum v2 and to do their own template construction, then it'll be very interesting to see whether we do, migrate away from the pools doing all of the block, template construction to individual miners. and even if it isn't a large percentage, you know, we have so few people constructing the templates now that it doesn't need to be a vast number of, of people who switch over for it to materially- improve the situation, especially because, you know, the real threat is that particular transactions don't get included, and you only need one block to contain that transaction for The game theory to play out where it's kind of in the interest of everyone to work on that block. So even if at the moment you've got a handful of pools who aren't interested in mining that transaction and then one One pool using Stratum v2 or just an independent solar miner, if they mine, you know, this one transaction into a block, it's probably quite likely at the moment that all these economically incentivized pools will then mine on top of it rather than, you know, try to reorganize" }, { "speaker": "stephan", "time": "44:44", "start": 2683.88, "text": "that out, yeah. Yeah, interesting. Okay, yeah, so, yeah, lots of things, to consider. Hopefully, you know, we see more progress on things like Stratum v2 and some of these efforts to, let's say, switch the, the pool Mempool.space, I know there are some new features and things, rolling out, things like, you know, the ability to use the goggles to kind of look at what's happening at the, at the block level" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "45:12", "start": 2711.68, "text": "Yeah, so that, that was interesting actually, 'cause it, it corresponded well with doing this research report. we've added lots of new functionality for introspecting, transaction data and, and bringing a lot of things to the surface that weren't visible before. So if you could read Bitcoin script, then you could open a transaction and, and eyeball it, and maybe you'd occasionally spot something that was familiar to you, but for the vast majority of people on Mempool.space, you know, it was a bit of a black box, and, and unless they were making the but all of this data, is, is public, and so if you, for example, are doing a, a multisig or something like that, you can tag that, using the data on chain, well, depending on the type of multisig. And so what we did is we brought all of those transaction level tags and showed them in the overview for the block. So when you're viewing a, a next block on Mempool.space, you can click on the goggles and you can then select, \"Okay, I want to see transactions that...\" Are consolidations where there's lots of inputs and a few outputs, and I also want to see transactions that are, I don't know, SegWit. And so I think bringing these kind of data analysis tools To people via the, the goggles system and being able to just eyeball kind of what's happening on the network is also really useful, because, yeah, previously it would have required something like a mempool research report to explain, you know, this is the frequency of consolidation transactions, but now you can just go on mempool.space and click on th-those toggles and you can see for yourself kind of what's happening and, and what's in the mempool. And there's a few other things that are being worked on actively at Mempool, I don't want to kind of jump the gun on any of those things now. I'm quite involved with Mempool, Doing the enterprise side of things. So we have mempool enterprise for businesses that are using, the mempool API or have particular demands. So if anyone has any, you know, questions, then by all means, reach out to me and I'll provide some information. and then we're working on some other research reports at the moment, but I'd rather leave those, leave those for now and, and we can maybe, maybe have a chat another time." }, { "speaker": "stephan", "time": "47:21", "start": 2840.77, "text": "Yeah. and I know there's also a community sponsor program as well. So I saw, Simon mentioned That, BTC Prague, Dev Day, where, you know, even if you aren't an enterprise user, you can, you know, be a sponsor and get some, get some gear, get a mug, get some, yeah, absolutely. So get some shirts. Yeah, the, the" }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "47:40", "start": 2859.97, "text": "enterprise, sponsorships are, are for businesses who are looking for You know, a partner to work with, to address particular problems they're having, and to solve problems for them. The community sponsorships are just for individuals or, or businesses who want to show their support for the project and get some swag. and so, yeah, you can go to the mempool.space, website and you can, you can see all these things available. I'm sure we can provide all the links in the show notes. I think- We're at the point now of beginning to, to roll some of this, merchandise out. So hopefully we'll start seeing people wearing it and, you know, we've, we always used to get questions from people asking when they could get hold of the t-shirts and, and obviously we've got the new mugs and things like that now. So, yeah, I think we're, we're pleased that people, you know, really like using the project so much that they want to, you know, wear the merch and, and show their support." }, { "speaker": "orange_surf_from_the_mempool_space_team", "time": "48:34", "start": 2913.82, "text": "I think the and we, we get lots of requests from people for new features and new functionality who are doing that, and it's just great to see people, you know, using, their own data, their own node, doing everything in their own, you know, environment and not having to rely on mempool.space, because although, you know, we, we put a lot of effort into making sure that mempool.space is very performant and very reliably accessible around the world, there's, there's nothing really that can beat having your own node and, and" }, { "speaker": "stephan", "time": "49:08", "start": 2947.52, "text": "So listeners, check out the links in the show notes. Mempool.space is the overall website, slash sponsor for the individual sponsors, slash enterprise for enterprise customers, and then the empty block report, again, that link will be in the show notes. So Orange Surf, thank you for joining me today. Thanks very much, Stefan. Great to speak to you. I hope you enjoyed the show. If you did, make sure to give it a thumbs up and share it out there with your family and friends. Check out my website at stefanlivera dot com, and I will see" } ] }