{ "episodeId": "SLP730", "speakers": { "stephan": { "name": "Stephan Livera", "role": "host", "tag": "STEPHAN" }, "pieter_wuille": { "name": "Pieter Wuille", "role": "guest", "tag": "PIETER" } }, "segments": [ { "speaker": "stephan", "time": "00:00", "start": 0.18, "text": "I would go as far as saying this is the entire reason why Bitcoin has proof of work. The point of having proof of work is that anyone can join the mining market. Bitcoin's decentralization and censorship resistance ultimately boils down, if things go really wrong, the final protection is you can become your own miner, maybe even at a loss, but hopefully not at a, at a big one. What is it that incentivizes existing large miners to be Basically accept every transaction. It is the knowledge that if they start reliably censoring some transactions that have economic demand, someone will just pop up and mine them anyway." }, { "speaker": "pieter_wuille", "time": "00:43", "start": 43.46, "text": "Hi everyone, and welcome back to Stephan Livera Podcast. Today, I'm joined by Peter Weller. He is, a long-time Bitcoin developer, researcher, and he is working at Chaincode Labs, and we're gonna be talking a bit about Cluster Mempool, which is, a new project that is landing in the new version of Bitcoin Core v31, I believe. Correct. So, Peter, welcome to the show, and let's start with, start with why. Why were you interested in this project of Cluster Mempool?" }, { "speaker": "stephan", "time": "01:10", "start": 69.65, "text": "Yeah, so you call it Just about three years ago, and the immediate motivation for this project was a talk that Suhas gave here at, at Chaincode Labs about some of the existing issues with the current mempool design. like the immediate motivation was this example of actually, you, you can end up in a state today in a mempool, obviously not likely, you would need a pathological adversary-ly constructed situation, but where the very first- Transactions that would be evicted if the mempool fills up, the first thing that gets evicted is actually the very first transaction you'd want to mine. And this is obviously undesirable. It's undesirable for miners, but it's, it's actually far worse than that. When the network is sort of unable to reason about the proper profitability of transactions. And because obviously you'd want eviction and mining, even if you're not a mining node, like you would want the thing Things you evict to roughly correspond to the very last things a miner would want to include. And so that, that, that was the motivation, sure, like actually the current design we have is broken in many ways, that doesn't necessarily mean exploitable, but it sort of makes it hard to, to, to reason about things, and we wanted to address that, came up with a cool design, and it grew into something that That ended up replacing everything in the mempool basically, internally touching on some relay policies that change along with it, and yeah, so it's been merged now, it will be in Bitcoin Core thirty-one, which is slated to be released next month, I think. So pretty excited" }, { "speaker": "pieter_wuille", "time": "03:02", "start": 182.21, "text": "about that. Great. And so just to motivate this for listeners, why does this project matter for everyday Bitcoin users? What is the impact that they might see or the-- you know, what is, yeah, basically, what is the impact that an everyday Bitcoin user will see here?" }, { "speaker": "stephan", "time": "03:17", "start": 196.81, "text": "Hopefully nothing. I, I think the, the motivation is, is more a long term health of the network question than it is directly improving, things for users. Now, they, they might see a few changes. The most noticeable, I think, is how today, so up, up to the current Bitcoin Core release, we have relay policies that include ancestor limits and descendant limits. That means Any transaction together with any and all of its unconfirmed ancestors can't be more than 25, and similarly, a transaction together with all its unconfirmed descendants can't be more than 25 transactions. And this Sometimes you see these peeling chains of, of someone makes a transaction, sends a change back to themselves, use the chain to do another payout, sends it back to themselves. Hopefully they would batch this and turn it into a single transaction, but if they don't, this, this sort of chain is limited to twenty five transactions at any given time, right? It's, it's not that it, it's impossible to peel further, but you'll need to wait for some of them to confirm before the later ones propagate across the network. And this changes. With cluster mpool, there are no more ancestor or descendant limits, instead there are cluster limits. Think of it as, it- It's your, we'll continue with the analogy of ancestors and descendant in a family tree. Cluster is just the, the widest possible extended family thing. It's like your parents and children, your grandparents and grandchildren, also aunts and uncles, nieces and nephews, their parents, their children, anything that is related through any combination of is parent of or is child of. And you, you can, and sometimes do see fairly complicated things where you have one user that pays a number of other users, and then they spend those coins together with other, another output that they got from yet another party. They, they all become related in a cluster. And so in Cluster Mempool, we have a limit of sixty-four transactions in a cluster. That, that's a number we picked based on performance characteristics of the algorithms inside. Similar to how the ancestor and descendant limits of twenty five in the past were picked based on, well, the-- these are the things we can sort of efficiently reason about still without the computation time for nodes blowing out of the water." }, { "speaker": "pieter_wuille", "time": "06:02", "start": 362.22, "text": "Yeah." }, { "speaker": "stephan", "time": "06:03", "start": 362.74, "text": "So from, so the most obvious direct impact is probably you could build these peeling chains up to sixty-four rather than twenty-five, but it's not like just increasing the number because you also have this extended family thing that gets counted along with it." }, { "speaker": "pieter_wuille", "time": "06:21", "start": 381.0, "text": "Okay. And so can we just overview a little bit on how the mempool works today, and then what are the main issues that you see with that? I think you were touching on some of that around the ancestor and descendant limits, and then we can sort of go into some other concepts, to help make this understandable for us. So just overview on the mempool today and the issues." }, { "speaker": "stephan", "time": "06:41", "start": 401.42, "text": "So Bitcoin Core uses a design where it maintains every node, not just miners, maintains a set of transactions that It expects to be mined in the short to medium term, up to roughly a day worth of transactions with default configurations. It maintains the set as a consistent set, so will never have two conflicting transactions in the mempool. It just becomes awfully hard if you try to reason about that. So its purpose is that there are many uses for this. One is obviously for miners to, to pick their transactions from, but maybe more importantly, it's for nodes in the network to get an idea of, of what the transaction market at that point is like. It, it, it's used for fee estimation. It, it's used for deciding what replacements to make. Alright, if you see two conflicting transactions, one will need to replace the other or, or not, you stick with the old one. How, how do you reason about, well, is, is this actually an improvement? if we can, can decide that this new transaction actually looks better to miners, we should assume it will make their way to those miners and thus for us to be able to predict the future behavior. We want them in our mempool too. It's basically a modeling of what we think the block space market is like at this very instance. And the way, the, this is guided by, by a whole number of heuristic algorithms today. So before cluster mpool, for example, mining used something called an ancestor set based mining, which is For every transaction, you compute the set of all its ancestors, which, as I explained, was limited to twenty-five at past, and look at the average fee rate of that whole set. And the reasoning is like, say, I pay you, and now you take the-- and but I, I attach a very low fee only, and you wanna spend those coins for some reason, and you want to do so urgently, so you take those coins, pay someone else with it. But my transaction hasn't confirmed yet, so what you wanna do is, is bump the fee. This is called child pays for parent, where you attach a higher fee than you normally would to sort of pay for the missing fee I have. Because miners aren't allowed to include your transaction without including mine as well, and so by attaching a higher fee, you incentivize them to include both of the transactions at the same time. And this is sort of modeled using this ancestor set based mining, where the, the mining logic or the, the block template building logic, this is long before there's an actual miner involved, will look at the two set transaction combined because they form An ancestor set, and sees this is the average fee in this whole set, I'm gonna try treating it as a package that gets mined in one go. So this is what makes child pays for parent work today, and it has since, since twenty fifteen. Practically how it works, we precompute for every transaction in the mempool, the set of all its ancestors, compute the average fee rate of all of those ancestor sets, and sort the transactions by their average ancestor set fee rate. And we pretty much eagerly include the highest ancestor set fee rate package and then start over until the block fills up. This is a pretty good approximation, but the opposite, we would like to do the very opposite be-cu-because again, not every node in the network is a miner, but every node has reasons to try to predict that behavior is a fiction, and in a fiction we do the exact opposite, which is we compute the descendant set for every transaction, because whenever you want to evict a transaction, you must also evict all their descendants, they become invalid. I paid you, you paid someone else. If my transaction gets evicted, you don't get your money anymore, so your transactions need to disappear as well. So we do, we precompute for every transaction the descendant set and the average fee rate of that descendant set, and when evicting, we pick The descendant set with the lowest fee rate, and this feels very much like the opposite of what a mining algorithm is, but it turns out you can construct weird edge cases that involve diamond-shaped dependencies. You have one parent and two children, and something that spawns those two children that causes things to be counted double, and as a result, you, you, you-- these two orderings, like highest ancestor set first and lowest descendant set first. are not opposites of each other. But there, there are many more heuristics involved. For example, for RBF, we've had for the longest time, the BIP 125 rules that sort of give a number of conditions on, on, if you satisfy this and this and this, your transaction will be accepted for replacement. And it tries to heuristically answer, does this actually make things better for miners, in terms of fee income? Because if it is, they'll want it and- We don't want to introduce a reason for people to submit their transactions to miners and make the block space market basically private. So to, to, to avoid that, we, we, we need to be able to, to reason about this profitability, and Bitcoin 25 was an, an attempt at that in," }, { "speaker": "stephan", "time": "12:48", "start": 768.17, "text": "Pre cluster mempool. There's more things like fee estimation uses mempool, there's decisions like if, if you suddenly get a flood of transactions at the same time, we don't want to overload our peers, so we trickle them out. Which ones do we send first? Well, the transactions that we think are better. So we try to send the ones with a higher fee rate first, and so forth. So that, that's sort of how I think if I had, had to summarize today is, is like we- We have Mempool is an attempt at approximating the current view of demand for the block space market, and it has a whole bunch of heuristic rules to decide what to keep and what to evict and, and what to send and what to receive that try to approximate this economic reality. I'd say Cluster Mempool is replacing all of that with a framework that can actually reason about things properly, taking dependent-- arbitrary dependent Tendencies into account and do away with all the heuristics and replace it with one rule is, does this make things better or not?" }, { "speaker": "pieter_wuille", "time": "13:55", "start": 834.76, "text": "Okay. And so as I'm reading you then, it's like, in simple terms, okay, we've got this mempool, people put their transaction in this, in there, but as you said, there are certain heuristics around ancestors and descendants and eviction, what we're saying, and Some of these heuristics aren't necessarily always correct because they're heuristics, and maybe as a quote-unquote rule of thumb, they might kind of mostly work, but then there are some times where maybe the heuristic goes awry or goes wrong, and there are examples where maybe a miner would have or should have wanted to include a particular transaction and the descendants of that transaction, but actually because of the heuristics, they would cut them out. Yeah. Is that what you're getting at?" }, { "speaker": "stephan", "time": "14:34", "start": 874.47, "text": "Yeah, yeah, yeah. And, and this is undesirable because we do have- You have to make a caveat here is that ultimately only the things that people actually do matter. Like if, if nobody creates super complicated dependency graphs of transactions, there's no real economic demand for that. It doesn't really matter how things-- like it, it doesn't matter that we treat those suboptimally because it's only an economic reality that matters. But the problem becomes not just that our decisions may be suboptimal But they may be inconsistent, and inconsistency can be a lot worse, because now you, it's not just okay, ma-miners make in this weird edge case that nobody really does make a tiny bit less money, who cares? But if there's- If we're talking about like eviction rules, maybe you, you, you get some-- Well, someone may wants to make those transactions and they're not propagating. So now you're incentivizing market players to move towards-- I see, you're" }, { "speaker": "pieter_wuille", "time": "15:41", "start": 940.52, "text": "driving towards private mempools again. Exactly." }, { "speaker": "stephan", "time": "15:43", "start": 942.84, "text": "And, and I, I think that is really the, the core issue here is like, it's a very tenuous situation how we're trying to make the public peer-to-peer transparent Current mempool market as good as possible, and, and the hope is that if it is sufficiently good, there's just no incentive for people to bypass it. And, and we see attempts at bypassing it, right? There are, what is it, transaction acceleration, exactly," }, { "speaker": "pieter_wuille", "time": "16:14", "start": 974.1, "text": "different accelerators that a lot of the pools have had for years, it's not a new thing." }, { "speaker": "stephan", "time": "16:18", "start": 977.59, "text": "Yeah. and, and I think it's, it's worrying. I don't think it, it's necessarily a problem to- Today, I, I think that the total income from those services is mostly negligible, except maybe at certain times. But if we were to end up in a situation where just the-- there is so much economic- Activity going through these private transaction rails where the network just doesn't have visibility into it, it ultimately makes it harder to enter the mining markets, because say there's three big companies that, you know, have a private mempool that you can submit things to, and as a miner, you really have no choice but to contract with one or most of these, because otherwise you just miss out on a substantial- Financial portion of fee income and are unable to compete with those who do, it just makes it harder to, especially anonymously. Like, we would very much like it to be possible for, anyone in the world. Obviously, you need access to internet and electricity and hardware, but you don't need permission from someone to enter the mining market if you are If you think there, there's money to be made, or if you are unhappy with the set of transactions that are being censored by others, then now we get to the core issue, censorship resistance. Right? And, and it- Like so much so, I, I would go as far as saying this is the entire reason why Bitcoin has proof of work. The point of having proof of work is that anyone can join the mining market. Like Bitcoin's decentralization and censorship resistance ultimately boils down, if things go really wrong, like the final protection is you can become your own miner, maybe even at a loss, but hopefully not as a, the big one. And so- What is it that incentivizes existing large miners to basically accept every transaction? It is the knowledge that if they start reliably censoring some transactions that have economic demand, someone will just pop up and mine them anyway. And But this, all of this only works when access to the stream of transactions that users create and have economic demand for is public. You don't need to go to a private party to get access to that economic activity, because if you do that, they become the gatekeepers. They could enforce a rule and say, \"Well, you're an anonymous miner, you, you don't get our stream.\" And so I think this, this is ultimately the, the motivation for all of this. work. We, we want to make it so that the public network is able to reliably relay transactions that users want to create and miners want to mine out into the public so everyone has access to them. Unfortunately, in a decentralized system, denial of service resistance is a very hard problem. We can't just relay every transaction. I'd stream my video over it and code it into transactions, that they're all conflicting. With one another, the-- there's a network out there that will freely relay every transaction, but that doesn't work. So what is-- and this has for a long time been sort of the philosophy behind the mempool design in Bitcoin Core is, under the assumption that we can reliably predict w-what will be mined, because miners are economic actors, that then our denial of service protection is basically, well Transactions in our mempool will get confirmed and they pay a fee, and this is the protection, they pay a fee, even if it's not to us as a relaying node, they're paying the fee to someone, so there's a finite resource they have to spend. So we need heuristics to decide what will be relayed, what won't be, and those boil down to predicting profitability of those transactions. And all the examples I gave about mining and eviction and deciding the order in which things go are Sometimes faulty heuristics that try to reason about profitability, and so with Cluster Mempool we can just do that much more reliably and consistently." }, { "speaker": "pieter_wuille", "time": "21:00", "start": 1260.05, "text": "Okay. Yeah, so just replaying some of that, the understanding I'm taking from it is instead of using these heuristics, moving to a new model of cluster mempool which allows, let's say, that, that node runner or that miner to more accurately, let's say, get an idea of what transactions are likely to be, the most profitable. And as you said, it's not just about optimizing the profit, it's about making it so that it's an, you know, mining remains an open- thing that a new miner can join. Making it predictable, but" }, { "speaker": "stephan", "time": "21:35", "start": 1294.67, "text": "even predictable in the face of possibly complicated constellations of transactions. So as a very simple example and not an attack or anything is, so I've talked about child pays for parent, where you can have a single child that bumps the fee of a parent because the parent paid a, too little fee, where you can't have to, and you can have one, one child that bumps multiple parents at the same time too, but you can't have multiple children that bump the same parent. Because of this ancestor set framework, each of the children has its own ancestor set, and they're never treated as a single set. And, and, and I think this was not the original goal of cluster mpool, because I haven't really talked about what it is yet, but the goal wasn't so much improving or adding, You know, more crazy use cases that someone could construct, like children pay, pay for a parent, it was just making things consistent. But then, a-along the way, we discovered that there were actually much nicer algorithms that became available that, that can take these much more complicated constructions into account and still deal with them almost optimally." }, { "speaker": "pieter_wuille", "time": "22:56", "start": 1375.6, "text": "Yeah. Okay, and so as I was trying to research and understand where this will impact, so as you said, mining is an obvious one, Lightning and L2s, as I understand, it might have some impact for them, and then maybe as an example, the exchange withdrawal case. So as I'm understanding, some exchanges, they might do like- They might, let's say, you know, let's say I'm the exchange and I've got like a hundred customers, I might be doing a big withdrawal, but then I might be continually doing either CPFP or RBF to continually add new outputs to that transaction to go out to the, the customers. And I presume then Cluster Mempool might help in this context also because they can more reliably do that." }, { "speaker": "stephan", "time": "23:39", "start": 1419.4, "text": "Yeah, so the, there were, there's a, a number of things I've seen exchanges do. So they, they might do batching where you have one transaction that pays out tons of people at the same time. I don't think that, that will be affected too much, or you can have the, these, these changes. You're, you're talking more about like continual replacement. So I, I don't know if they will be affected, but it's, I mean, the, the rules change there because, with Bitcoin 25, it, it used to be the case that there are Certain things that are like obvious improvements, like you, you, you make a transaction that, that just pays a higher fee, it replaces another one, it makes some changes," }, { "speaker": "stephan", "time": "24:29", "start": 1468.72, "text": "But wouldn't be accepted. And, and the other way around, where, where you have something that, that's like clearly not an improvement, and yet it would be accepted. I, I, I think At a high level, things become simpler in the sense that you can adopt a strategy of, \"I'll just bump the fee.\" Like, I do a replacement, it doesn't make it its way through, I just bump the fee a bit more, and if it still doesn't work, I bump the fee more, replace it with another transaction, and this will just work. Like, the transaction will go through as soon as it's actually better, where in the past you could try to follow One twenty five rules, but as far as I know, nobody actually does. In practice, people already adopt this, you know, try and see approach. Like, you bump the fee, it doesn't work, I'll, I'll bump it some more." }, { "speaker": "stephan", "time": "25:27", "start": 1527.09, "text": "Now we'll go through when, whenever you've actually improved the situation. Right." }, { "speaker": "pieter_wuille", "time": "25:32", "start": 1531.72, "text": "So it's kind of like in practice, people were just doing sort of a trial and error approach anyway on these things. Yeah." }, { "speaker": "stephan", "time": "25:37", "start": 1537.12, "text": "And, and, and I think this, this is- In theory, I think this is the, the biggest downside to the new approach is I can't give you a simple set of rules anymore that are of the form like, \"If you satisfy these rules, the fee rate of this is bigger than the fee rate of that, and the size of this is less than that, and whatever, I guarantee replacement.\" That, that's no longer the case. There's no simple rules. It is, you have to make things better, basically." }, { "speaker": "pieter_wuille", "time": "26:06", "start": 1566.31, "text": "Yeah, okay. so, I mean, we've set a bit of a context, and I think it was Listeners, let's, let's actually go now to, you know, how does cluster mempool actually work? So can you take us through an overview on that?" }, { "speaker": "stephan", "time": "26:17", "start": 1576.8, "text": "Yeah. So the, the main observation is In order to reliably predict how, you know, make judgments about are these transactions better than these, will this one get mined before this one? Today with the algorithm we have, if, if I wanted to actually figure out, let's talk about the eviction case, if I wanted to actually reliably figure out today, what is the last transaction that would be mined? Imagine no new transactions come in, whatever. Give you a, a mempool of four thousand transactions, four thousand, and you wanna figure out what is the very last one that would be mined, because that's the one I want to evict if I'm somehow resource constrained. Well, the only way you have today is you run the mining algorithm, but not for a one megabyte block, but for a hundred thousand or nine hundred and ninety-nine transaction block, and you see what's the one transaction that doesn't go in. That's The one I would want to evict. Sadly, that's just computationally not feasible. We, we can't any time we want to evict a single transaction, run this entire, you know, block template building thing to see what goes in last. So what is the obvious answer? As a computer scientist, well, we wanna precompute things, we wanna cache it. We, we, we, how about we just maintain at all times the mempool in a sorted order from good to bad, and then mining becomes well, take from the front, and eviction becomes drop from the back. unfortunately, it, it turns out that, you, yeah, I don't, I don't, don't think drawing it on a whiteboard here, here, here is helpful. For, for listeners, but I've given examples in, in write-ups about this where, where you can construct pathological mempools where you have like a whole sequence of transactions where, like, they're, they're all like, every parent has two children and every child has two parents and they're sort of connected as a trellis where the optimal ordering is going from left to right, the best transactions are in the left, the worst transactions are in the right And now a single new transaction comes in that pays a huge, huge fee and attaches on the right, and now suddenly the best order becomes you go from right to left. And this is pretty unintuitive, but it's true if you work it out. So the observation here is that a single transaction attaching to a huge cluster of transactions can basically completely overhaul how good things are within that cluster. So this notion of we can just precompute what the order is and, and make small changes to it doesn't work because a single thing can come in that requires us to recompute" }, { "speaker": "pieter_wuille", "time": "29:18", "start": 1758.15, "text": "I see. And just, I mean, out of curiosity, how does something like Mempool.space do it? I mean, they, they kind of project out what they think the blocks will be. Is that just like really computationally expensive or like just not feasible?" }, { "speaker": "stephan", "time": "29:30", "start": 1770.08, "text": "So I don't know what they run in the background. I, I assume they have Bitcoin Core nodes that they pull things from, wh-which do something, but it maybe will be suboptimal. I don't know, like the, the, in practice you make an approximation for, for everything, anything, and, and you'll get some output. It, it just may not correspond to what is the actual best thing to do. And from this observation that a single transaction, in, in a, you know, hypothetical adversarially constructed mempool, we, you won't see, like, you know, I don't know, ninety percent of the mempool is one or two transaction clusters. There's no problem here. But, in worst case, y-you can have a single transaction comes in that requires you to, to recompute everything. And to be clear, when I say recompute everything, th-this may be- Tens of milliseconds or something is just not something we can do for every single transaction that comes in, right? It, it's, it's, it's not like I'm, I'm claiming this will take hours to compute. It's like tens or hundreds of milliseconds, but that is too much to do all the time, every time a, a change comes in. And so that's where the idea of the clustering come, comes in, because it, it turns out we don't actually need the ordering of the entire mempool, we just need the ordering within every cluster of transactions. I, if I precompute, like, I have a, a bunch of related transactions with dependencies and whatever, but it's a small group, and I know how to order those, and then I have another cluster, and I know how to order those, and yet another one, and I know know how to order those, you can very efficiently sort of merge them into a single view of the mempool. And this is where the idea of cluster mempool comes in. If we were to just able to limit how big those clusters can get, then any individual transaction that comes in will just affect the order of that one cluster. And sixty-four transactions is the number we ended up with as, as our proposal, because for such small clusters They're not tiny, right? Sixty-four transactions is significantly larger than what, what, what things are supported today. Yeah, a new transaction comes in, you d-- you will just need to do the re-computation for that small group of clusters, and we don't have to worry about a single transaction, you know?" }, { "speaker": "pieter_wuille", "time": "32:05", "start": 1924.71, "text": "Like appending the whole block kind of thing. Exactly. Instead, you're just reconsidering that particular cluster, isn't it? Not," }, { "speaker": "stephan", "time": "32:10", "start": 1930.5, "text": "not, not in, in my previous like hypothetical example, I wasn't just talking about o-one block, I was talking about the entire mempool. Could, could like, like hundreds of blocks worth of transactions could need to be appended in a hypothetical situation. Example. And, and here it, it's, you know, it becomes limited. And so cluster mempool is Switching to a policy rule that imposes a limit on how big the clusters can get based on our intuition, belief, observation that users don't really care about huge clusters, and then it becomes possible to precompute the mining order within every cluster, and from that you can implicitly define a whole mempool ordering, and now you decide is this better Than this, is this an improvement? What's the best thing to pick? What's the last thing we should be evicting, and so" }, { "speaker": "pieter_wuille", "time": "33:09", "start": 1989.32, "text": "forth. And then the idea is that these clusters, you'll have, I guess, is it chunks, and then you assess one chunk against another to say, \"Oh, the fee rate in cluster or chunk...\" B is higher than chunk A, so I'm gonna, you know, preference that in favor of, you know, et cetera." }, { "speaker": "stephan", "time": "33:26", "start": 2005.62, "text": "Yeah, so the chunks are sort of the generalization of CPFP groups of transactions. Whenever you have some transactions paying for other transactions, and because, say, we, we have a, a single parent transaction that pays both of us, you and me, both get an output, and but it's, it's a fairly low fee. Transaction and I wanna accelerate it, so I spend my output, but I spend it with a huge fee. You spend yours, but you don't really care, you spend it with a low fee. What you want is my bumping transaction to bump the parent, but yours doesn't, in the sense that yours is low, you don't care. So even though this is one cluster, it will become two chunks. The first chunk will be the parent that paid both of us and my transaction that bumps it, and then the second chunk is yours that becomes dependent on it. And it's very related to the ordering. I don't think I should go into the details too much here, but if I give you an ordering of transactions for a cluster, you can find what the chunks are, like which things need to be combined. And actually, finding the optimal ordering is the same problem as finding the chunks, it turns out. And so it, it's, I think the simplest way of describing what a chunk is, t-take your cluster, which is a whole group of dependent transactions, whatever, can be dozens. You Call a topological set is a set of transactions that includes all its own ancestors. So in our example, just the parent would be a topological set, the parent together with mine would be one, the parent together with you would be, or all three would be topological. And then, among all the topological sets, pick the one with the highest fee rate, and that becomes your first chunk. Remove it from the cluster, and now you re- We do that same computation on what remains of the cluster, try to find in what remains the highest fee rate topological set. I think that intuitively it makes sense because obviously you can only include topological sets, you, you cannot include something where a parent is missing. So among all those, pick the one with highest fee rate. So that is the chunking problem and that is the linearization problem." }, { "speaker": "pieter_wuille", "time": "35:53", "start": 2152.9, "text": "And when you say fee rate, what we're talking about is like Not just absolute fee, what we're talking about is fee per, per V-byte, right?" }, { "speaker": "stephan", "time": "36:02", "start": 2161.8, "text": "Yes. And, and average-- and I'm talking about the average fee per V-byte in the whole set. So it's the sum of all the fees of all the transactions in the set divided by the sum of all the V-sizes, or weights. Actually, internally, all, all the computation is done with weights." }, { "speaker": "pieter_wuille", "time": "36:20", "start": 2180.04, "text": "Okay. Yeah, I think I'm, I'm slowly getting the idea. And then, I guess the idea is, yeah, so you're, you're expanding out the, you're simplifying transaction group limits, you are Taking away certain previous limitations on how many unconfirmed ancestors, because now instead of twenty-five, we're going up to sixty-four, but that's at a cluster level. And then I'm, I guess, kind of thinking about L2s and this thing, I'm thinking about maybe it's not the exact precise technical term, but I've heard people talking about these concepts like unfurling or unrolling, this kind of thing, like maybe in the future this would be relevant for that when people are trying to, let's, like, exit out of- Of some kind of shared UTxO. Is that, is that related to this?" }, { "speaker": "stephan", "time": "37:10", "start": 2229.8, "text": "Yes, but I'm not the expert at that." }, { "speaker": "pieter_wuille", "time": "37:13", "start": 2232.57, "text": "Gotcha. Right, but I guess in theory it would be, it might ease the pathway for the, some of those kinds of L2s, that kind of thing. Yeah. Yeah. Okay. and then, let me just see. So, so this notion as well, like we, we're, we're touching on this before, do you believe that mempools should roughly align across the network? Is that, like, that's the rough idea?" }, { "speaker": "stephan", "time": "37:36", "start": 2256.26, "text": "I, we can't, right? There are many reasons for-- Obviously, no, they can't, or we can't guarantee it, because if they would, we wouldn't need a blockchain in the first place, just accept whatever transactions are in the shared mempool first, the whole consensus problem is solved. We just can't do that due to, you know, that there's no centralized party that can impose a single ordering on everything. Everybody has their own view. You might have conflicts that arise from even just, double spends. Like, what if someone has an output, spends it in two different ways, gives one version to me, gives another to you, if one is obviously the better than the other, then a replacement might be possible in one or the other direction. But what if they're equally good? That, there's, there's no- objective measure of picking one over the other. This is worsened by the fact that we need denial of service protection rules because so even if we could define a mathematically optimal set, we can't really guarantee convergence because it might be computationally infeasible to do the work to figuring out if it's actually better or not. And then o-obviously there's policy differences which have good reasons, like different nodes might have different- Resource limits. no, we can't guarantee things, and this does mean that there are limitations to the quality of the reasoning we can make. So I, I wouldn't say the goal here is making mempools uniform. It's, it's really everyone for themselves is now given a better framework to reason about profitability, and there are still other limitations that mean not everyone will- Find the same outcome, but, yeah." }, { "speaker": "pieter_wuille", "time": "39:28", "start": 2367.83, "text": "I see. So as I understand, no, no, I don't, it, yeah." }, { "speaker": "stephan", "time": "39:31", "start": 2371.46, "text": "I think we shouldn't, you know, gratuitously introduce inconsistencies between them. But the goal isn't uniformity, the, the goal is, you know, make the best reasoning you can locally and" }, { "speaker": "pieter_wuille", "time": "39:46", "start": 2386.2, "text": "Okay. And so as I understand, and from what I've read from different Bitcoin core developers, there seems to be a lot of concern about things like block propagation, making sure that it's very, you know, very low latency. So I guess that's kind of a related idea then that if, you know, you-- I guess in theory, it would be nice if they sort of were roughly the same so that there's less of a problem around that, but you can't guarantee that. That's-- Is that-- Yeah." }, { "speaker": "stephan", "time": "40:12", "start": 2411.94, "text": "Yeah." }, { "speaker": "pieter_wuille", "time": "40:12", "start": 2412.4, "text": "Okay." }, { "speaker": "stephan", "time": "40:14", "start": 2413.88, "text": "if you've seen it, there's been A relatively recent idea by Anthony Towns to, to, to this, block template sharing, which, which addresses some of this in, in a limited way, but it, it's a pretty simple idea. So the, the idea is like, add a protocol extension that nodes can negotiate with each other, it's not a consensus change or anything entirely opt-in, where, you can ask your peer, if you were to- Create a block right now. What, what would it contain? Or you can even go a bit further, like, what are the first two blocks you would construct with your mempool right now? And I try, if it includes transactions I haven't seen yet, I'll try to insert them into my mempool. Obviously, why wouldn't I? I'll try to validate them if I can. But even if I don't, I hold on to them. So even if somehow you have a somewhat different policy or there's a double spend that results in us being unable to, to come to you, you know, an, an exactly identical view, it will be the case that I ask you for, hey, what's your, what would you put in a block right now? You give me these transaction, like, huh, some of these I, I, I don't even wanna validate or outside of some policy I have, but I can still hold on to it. So if it then turns out that some of those transactions are found in a block, I already have them and I get the, the proper- Propagation speed benefit, even if not the validation speed benefits from having them fetched ahead of time. And, and so this works as long as you have a network where-- and I shouldn't say policy differences, because there can be good reasons for mempool divergence that aren't related to policy differences. So w- whenever sort of the, the boundary of mempool divergence, everyone is connected to someone across- The border. Like if, if you end up in a world where, there's two big groups, but they're very clustered together, you know? group of users A are very much connected to, mostly to other users A, and then there's B who are mostly connected to each other. The, the template sharing will only help on the border between them. It's somewhat related to an older idea called, called weak blocks, where miners are allowed to propagate blocks that almost meet the definition of proof of work. They're not quite there, but we, we can relay them, and because there's proof of work, they could be relayed across the network everywhere and given preferential treatment, like there's no DOS concern because there's proof of work, and proof of work is expensive, even if it's just ten percent of the real difficulty. Indeed, there's a substantial amount of proof of work, but it's a much more complicated idea. I've seen some, some objections against it too, but even ignoring that, it, it's much more complicated in, in that it like interacts with miner infrastructure in a much more low level way than this. Tell me what you would in a, put in a block right now. So I, I like this idea." }, { "speaker": "pieter_wuille", "time": "43:36", "start": 2615.53, "text": "Yeah." }, { "speaker": "stephan", "time": "43:37", "start": 2617.27, "text": "It helps somewhat with propagation speed and mempool convergence, but again, it, it doesn't guarantee anything." }, { "speaker": "pieter_wuille", "time": "43:44", "start": 2624.15, "text": "Okay. So coming back to, you were touching on this before, but as I understand, cluster mempool, it's a Bitcoin core thing. How does that work when we're talking about, let's say, other implementations like, I don't know, BTC-D or whoever else is out there? What does it mean if a Bitcoin core v31 node is talking to other implementations of Bitcoin?" }, { "speaker": "stephan", "time": "44:04", "start": 2644.5, "text": "Yeah. Well, or, or, or even what if they're talking to a Bitcoin Core 30 or 29 and older core version, yeah? I've touched on this before in that it, it, it's about improving local reasoning. The fact that you are connected to people who, who use different, Probably, I don't know, inferior. Reasoning doesn't prevent you from making better decisions yourself. In terms of predictability, in that, once it comes to a point of like, Can network users start soft relying on cluster mempool on being deployed on the network to relay maybe somewhat non-trivial things that will depend on deployment?" }, { "speaker": "pieter_wuille", "time": "44:58", "start": 2698.18, "text": "But I g- I guess we would, you would assume that in a few years' time, most of the network would be running v31 or later, and then at that point, I mean, maybe that is a more fair assumption to make." }, { "speaker": "stephan", "time": "45:09", "start": 2709.27, "text": "Yeah. right, but this is up to individual users really to, to assess. I mean, it, it does go to-- If it doesn't, then maybe we have failed in this, you know, goal of censorship resistance through making the public market transparent enough. If there's real demand, there, there has to be demand for some complicated constellation of transactions being relayed on the network, and it doesn't due to non- Deployment of this, or maybe it's due to limitations of cluster mempool, we still have anti-dos rules that result in, in certain pinning problems. Those are, some, some are affected, but they're not gone. So for whatever reason, if, if it turns out that there's economic demand for transaction constellations that we can't reliably relay due to non-deployment or due to in-effec- Effectiveness or, or whatever, sort of fails at its goal of, you know, keeping this market public." }, { "speaker": "pieter_wuille", "time": "46:18", "start": 2778.0, "text": "I see." }, { "speaker": "stephan", "time": "46:18", "start": 2778.48, "text": "But, but that is to be seen, and I like how the, the cluster mempool Framework just makes it so much easier to do this reasoning, right? In, in, in, in the past there's, for example, been talk about package relay, we have some limited form of package relay now, there are questions about like, can we do package RBF? But before cluster mempool, like, I don't think we could really conceptualize what rules would be involved in a package RBF, like you have multiple dependent transactions in a mempool and you- Get another set of dependent transactions in, and some of those might replace some of those. Like, it, it, it, it's a very hard-- It's still pretty hard even with cluster mpool, but at least there's a, a framework to talk about those things. And I guess that's what I like most about it, e-even not The, the deployed capabilities that the network gains, but the, the fact that we, you know, have a mental framework to, to assess these things." }, { "speaker": "pieter_wuille", "time": "47:25", "start": 2844.68, "text": "Okay. So yeah, I was gonna ask, I mean, you mentioned, any other, you know, while we're here talking about mempool stuff, other related mempool ideas. So you mentioned package relay, v3 transactions, anything else that you're interested in or focused on that you wanna discuss there?" }, { "speaker": "stephan", "time": "47:40", "start": 2859.73, "text": "Yeah, so I, I touched on, on, on the, the, the block template sharing, I think that we can think of that as a transaction relay, aspect. There's some, some renewed interest in, in early as well. These are very orthogonal improvements, right? They're, they're not about reasoning about, you know, how good transactions are. They're more infrastructure level, improvements to transaction relay. Yeah, I'm interested in, in those, but, maybe helping out with some of those is, is something I'll be focusing on in, in the future." }, { "speaker": "pieter_wuille", "time": "48:18", "start": 2897.69, "text": "And I guess zooming out a little bit more, like out of just, let's say, mempool and cluster mempool What are some other areas that you're seeing as interesting, like in Bitcoin development, protocol development, just more, more broadly? Is there anything else that is interesting to you?" }, { "speaker": "stephan", "time": "48:33", "start": 2913.02, "text": "Sure. Yeah, my, my interest shift, shift over time. Like, I, I, I can't tell you the things I'm thinking about now. I'm considering looking into m-making validation more asynchronous internally, w-which has some bearing on, on, in Bitcoin Core specifically, this would be very much an implementation detail Another protocol level change, but it would have some impact in, in, in terms of latency where you can have-- What if I send you a slow transaction? Can I slow to validate transaction? To what extent can I slow your node down? That, that is today, an, an uncomfortable reality due, due to how synchronous and, and blocking things are done, I'd like to look into that, but yeah. Yeah," }, { "speaker": "pieter_wuille", "time": "49:22", "start": 2961.56, "text": "and so is that-- when you said, when you mentioned validation, are we talking like at an IBD stage or just in general?" }, { "speaker": "stephan", "time": "49:29", "start": 2969.06, "text": "I mean, in, in general, but I think the impact would mostly be on, on the steady state." }, { "speaker": "pieter_wuille", "time": "49:35", "start": 2975.3, "text": "Okay, as in like staying at the chain tip correctly is kind of loosely understood. Yeah." }, { "speaker": "stephan", "time": "49:41", "start": 2980.76, "text": "And the interaction with, with, with the network, you, you being a node connected to many others that give you messages like, how quickly can you respond to them? Like, isn't about making things faster, it's more about making them snappier, so to speak, if you can do multiple things at the same time, even if Dumb example, you give me a very slow transaction, someone else gives me a block. Maybe I wanna interrupt processing your transaction and give priority to that block. This isn't something we can do today." }, { "speaker": "pieter_wuille", "time": "50:12", "start": 3012.06, "text": "I see, yeah, I see, yeah. Because at first when you said that, I was thinking about like IBD stuff and like, you know, like Swift Sync and Assume UTXO and this kind of, that kind of, or UTXO like that kind of area, but as I understand, this is a bit, a bit of a different area." }, { "speaker": "stephan", "time": "50:25", "start": 3024.81, "text": "This is, this is much more" }, { "speaker": "stephan", "time": "50:29", "start": 3028.8, "text": "That's something I'm thinking" }, { "speaker": "pieter_wuille", "time": "50:30", "start": 3029.7, "text": "of. I guess, I mean, other kind of things I see people talking about are things like, you know, covenants and things like that. Is any of that interesting to you at all or not really?" }, { "speaker": "stephan", "time": "50:39", "start": 3039.26, "text": "I've, I'm, I'm trying very hard to sort of a-after, to an extent after SegWit, but even more after Taproot, I am trying to stay out of all consensus change discussions. Yeah. Okay. Yeah. I, I just feel like I, I've had my fair share there and, you know, it, So, I'd rather stay out of that." }, { "speaker": "pieter_wuille", "time": "51:01", "start": 3061.3, "text": "Sure. Yeah. Okay. so I guess let's say cluster mempool, you know, comes in in this next E31, and I guess next thing for you would be maybe looking at things like you mentioned the block template sharing idea from AJO Towns or this parallel validation. Those are probably the, the main areas that you'd be interested in?" }, { "speaker": "stephan", "time": "51:21", "start": 3080.64, "text": "Yeah, they're, they're, they're things that are currently on my mind, but who knows, maybe in, in a month I'm thinking about other stuff." }, { "speaker": "pieter_wuille", "time": "51:29", "start": 3088.76, "text": "Yeah. And I guess just zooming out as like at an ecosystem level, even if, even if it's not consensus related, what other things do you think people, what other things do you think the ecosystem needs work on? Like, is it wallets? Is it self custody? Is it, you know, mining stuff? Is it L2 stuff? Like, do you have any thoughts on those areas?" }, { "speaker": "stephan", "time": "51:47", "start": 3107.1, "text": "Not really. There's lots of good work, like you're, you're talking about mining, I'm, I'm happy to see the evolution with Stratum v2. I, I hope it takes off. I haven't had too much contribution myself there, but, nothing really comes to mind right now. I'm-" }, { "speaker": "pieter_wuille", "time": "52:04", "start": 3124.49, "text": "Okay. Alright, well, I guess let's, wrap things up then. I guess maybe just give a quick, like overview or just kind of a quick, what's the, the key insight for people to take away on cluster mempool, you know, if there's one takeaway, what should they take away with understanding cluster mempool? The" }, { "speaker": "stephan", "time": "52:21", "start": 3140.75, "text": "reasoning about dependent transactions is hard. We now have a framework to do that. Bitcoin Core will be able to use that reasoning in the future, and hopefully that results in us, key- Keeping the block space market public for longer." }, { "speaker": "pieter_wuille", "time": "52:40", "start": 3160.2, "text": "Excellent. Well, listeners, the links will be in the show notes, so, you can follow Peter's work online, and read more about Cluster Mempool. Peter, thank you for joining me." }, { "speaker": "stephan", "time": "52:49", "start": 3168.71, "text": "Yeah, thanks for having me. It was great." } ] }