{ "episodeId": "SLP54", "speakers": { "stephan": { "name": "Stephan Livera", "role": "host", "tag": "STEPHAN" }, "justin_camarena": { "name": "Justin Camarena", "role": "guest", "tag": "JUSTIN" } }, "segments": [ { "speaker": "stephan", "time": "00:08", "start": 8.27, "text": "Welcome to the Stephan Livera podcast focused on Bitcoin and Austrian economics. Learn about the technology and economics of Bitcoin by listening to interviews with Bitcoin's best. Today, for episode fifty-four, my guest is Justin Camarena, software developer at Bitrefill. Today, you'll learn a little bit about Justin's journey learning Bitcoin Lightning development and Py- Pioneering Bitrefill acceptance of Bitcoin via the Lightning Network and also a bit about Lightning channel management. Also, for anyone interested, I interviewed Sergey Kotlyar, CEO of Bitrefill, for episode twenty-five of this podcast. Here's my discussion with Justin. Justin, welcome to the show. Hello, happy to be on. Yeah, so Justin, I've heard you've been doing a lot of cool stuff, and just for listeners, I've also previously interviewed Sergei Kotlyar, also of Bitrefill, and I thought it'd be really cool to talk to you, Justin, and get a sense on what your experience was like with starting out, learning Lightning, and then what you guys are doing now with Bitrefill, at Bitrefill, with things like Thor. But let's just get a little bit of background on you first. First, just for the listeners, Justin." }, { "speaker": "justin_camarena", "time": "01:19", "start": 78.66, "text": "Yeah, so, my name is, Justin Camarena, I've been in Bitcoin not as early as most, but my first touch was in 2011, but I really started delving into the community in late 2013. It was, at around the time I was in college, so I actually switched, my major from political science into computer science because of how much I was interested in Bitcoin. So And it is a sort of foray into learning how to program to actually start building stuff around Bitcoin or with it." }, { "speaker": "justin_camarena", "time": "01:56", "start": 115.51, "text": "I think that's my intro." }, { "speaker": "stephan", "time": "01:58", "start": 117.73, "text": "Sure, okay. And let's talk a bit about how you got started working with Lightning Network." }, { "speaker": "justin_camarena", "time": "02:06", "start": 125.53, "text": "So originally I, was very interested in the payment channel technology, There was a company called Twenty One previously that sort of kind of got into this field. They provided a payment channel tech, a centralized version, but they basically, started getting interest in the Bitcoin community in building APIs that utilized Payment channels. Well, their scheme was sort of, clunky in, in that the channels were time-based and it only allowed you to send from, one node to another node. It didn't have this sort of- Multi-hop or sending through the network, there was no network so to speak, it was between two participants. So I first started getting interested in building APIs on, on this, just because of how easy it was, and they had, they built a library out of this. so with Lightning, when the paper was released in twenty, I think it was twenty sixteen, I was invited to a MIT cryptocurrency bootcamp in the middle of twenty sixteen, and there I met Taj. And at the time, I think they were building LND, but it wasn't at a state where it was running or many testers. So after that event, I sort of started becoming a tester and helping out with LND, and I was in university and I wasn't working in any company at this point. So at that point, Lightning was very, limited. The node software ran, but you were only able to send between two nodes, there were a lot of issues. So in my, my own way, I'm not a Go developer, but I thought I'd help out by Testing out and, and, filing issues, so to speak." }, { "speaker": "stephan", "time": "03:43", "start": 222.78, "text": "Yeah, I think that's a great example as well where you don't necessarily have to be the developer yourself, but you can contribute by being a tester. So tell us a little bit about how you went about learning Lightning and then where you went from there." }, { "speaker": "justin_camarena", "time": "03:57", "start": 237.17, "text": "So initially it was, just starting up a node, understanding all these moving parts." }, { "speaker": "justin_camarena", "time": "04:05", "start": 244.86, "text": "at the time, you would have to run a full Bitcoin node. I think that's sort of still the case now, but the, the API was really simple as it is now. just learning how to create invoices, learning the software, let's see." }, { "speaker": "justin_camarena", "time": "04:22", "start": 262.44, "text": "And from there, just building applications on, on top of it. from there, I sort of got In interest from people on Twitter, I was active on the, on this community, and my boss, Sergey Kolyar, reached out to me about interning at the refill, at this point in time. So that's how I sort of forayed into testing Lightning and sort of an intro point to a company and sort of being able to integrate Lightning into a, a Bitcoin based company at this" }, { "speaker": "stephan", "time": "04:48", "start": 287.98, "text": "point. Excellent. And then, so let's go a little bit further now, and then you were mentioning how, well, as most people know, Bitrefill were one of the pioneers with Lightning Network, both using it on the test net and also on main net to actually sell the product. Tell us a little bit about your experiences there." }, { "speaker": "justin_camarena", "time": "05:07", "start": 307.32, "text": "Yeah, so I- At Bitrefill, we are a, mo- we were at the time just a mobile phone top-up company, we've expanded to selling gift cards, but it was during the scaling debates and the increase of the fee market, so it was at the, at the time when transactions would cost five or ten dollars, we were interested in, in pushing out anything that could help us accept payments while still In these times of, high fee period, high fees. So, I was sort of, the engineer behind integrating this into Bitrefill, and at the time, I had been working on moving away from using third-party APIs. I know that, you had a podcast back with the Bills and Francis, how they discussed how they went through, like ten different APIs accepting Bitcoin until moving on their own to accepting it through CipherNode. Using their own node. So initially that was my first task at the company before getting hired, moving, the company from being using their own self-hosted node, and, from there I was able to easily expand to, add Lightning. At the time, this was only testnet, so I pushed it live and we were able to have people on testnet pay orders that wouldn't fulfill or anything, but, to try it out so to speak. And at this time there were mo-no mobile wallets, it was more of a just Command line. So the experience around building that, integrating that wasn't really that, that difficult. It was mostly on the node setup, which to me would be spinning up a Bitcoin node. Well, fortunately for us, we already had a Bitcoin node up and running just for our, our orders already. So the, the difficulties in spinning up a Lightning node wasn't as difficult earlier as it was today. I, I think it's pretty similar right now. Basically just installing Go, setting up the configuration, but integrating into BitPay for the website itself was pretty easy. It was just a few API calls, being able to understand the paradigm of invoices. So instead of addresses, we would have invoices. It was, it was pretty easy to integrate, in my, in my opinion." }, { "speaker": "stephan", "time": "07:22", "start": 441.97, "text": "Sure. And just give the listeners a sense of the timing here. So my understanding is roughly late 2017 was kind of fiddling around with testnet, and then sort of early 2018 was when you went mainnet?" }, { "speaker": "justin_camarena", "time": "07:36", "start": 456.41, "text": "Yeah, so early 2017, I've been testing out the software, so I was already familiar with it. mid-2017, I actually already had a Testnet integration, but at this point, the software was not as stable, it was crashing often, it wasn't something I could even push out for Testnet. mid-2017, the late, later end, that's when we started pushing Testnet live for, for users to, be able to test us. And I believe the end of 2017, we had a sort of a private, beta users testing us live on mainnet, and in 2018 is when we-- early 2018 was when we pushed it actually live." }, { "speaker": "stephan", "time": "08:16", "start": 495.82, "text": "Fascinating. And can you tell us a little bit about some of the challenges you faced as you were moving through that, I guess, that procedure of starting with testnet people and then moving into mainnet?" }, { "speaker": "justin_camarena", "time": "08:29", "start": 508.79, "text": "I think at first it was just having the node be like stable. at the time, there were a lot of breaking changes and the node would always crash for X or Y reason. the challenges there were sort of keeping it stable, being able to have it restart automatically. I mean, even today we have some sites that- Like they're not even crashes and they don't even have the, it implements it to start on its own. other challenges were sort of how to, present this to the user. Like, as a separate cur- separate currency, separate payment option. I think today we're sort of seeing many companies merge those options so you can pay with Bitcoin on-chain or Lightning. so that's been-- it, it wasn't really so much of a, of a challenge for- Me too, Adam. I would say integrating Bitcoin itself was a lot, a lot harder than integrating Lightning." }, { "speaker": "stephan", "time": "09:22", "start": 562.26, "text": "Right, I see. And what kind of setup were you working with back in the early days compared to what you're working with now?" }, { "speaker": "justin_camarena", "time": "09:31", "start": 570.55, "text": "I think, early in the days we, there were a lot of APIs that weren't ready or weren't, used, like I couldn't set fee rates for channels, Sending wasn't really, sort of a thought at the time. Nobody had sending transactions, there were no, mobile wallets or any, Sort of set up to be able to pay mobile wallets at the time, so it was only people that ran servers at the time. But setup-wise, it's, it's still pretty similar to how it was back in the day, the main APIs for accepting lightning haven't changed too much, at least for LND. So we use, LND, which is one of the three main implementations currently in use." }, { "speaker": "stephan", "time": "10:15", "start": 614.7, "text": "Excellent. And I think the cool thing with this stuff is it's just a, a fascinating opportunity that Doing Lightning Network and coding with Lightning Network has given you, so tell us a little bit about what you see there for opportunities for other people who might be considering taking up Lightning development." }, { "speaker": "justin_camarena", "time": "10:34", "start": 633.94, "text": "So what I saw a couple years back was, many people wanted to get into Bitcoin development, but there were many barriers to entry. they had ideas of what they wanted to implement But it wasn't really feasible doing this all on chain. Like what we're seeing now is more of a, all those possibilities that were, were not feasible before, sort of, sort of, sort of starting to become feasible with Lightning, and the development certainly is easier to, get yourself up and running compared to Bitcoin. there are a host of other like APIs, node setup guides, there are boot camps and, and so all these, all these kinds of things that help you And communities to sort of help you, bootstrap a lightning node and teach you how to use it and develop. Like we're seeing a much more dev focused community at this point." }, { "speaker": "stephan", "time": "11:24", "start": 683.78, "text": "Right. And the cool thing there is, yeah, as you mentioned, there's a lot of cool guides and Ways that make it easy. Have you got anything that you would recommend for listeners who are interested to try and learn further?" }, { "speaker": "justin_camarena", "time": "11:37", "start": 696.61, "text": "there's a Telegram channel called the Lightning Makers. LND has a, a Slack for, Lightning. I, we have a, a dev channel there where we, we help new, developers understand Lightning and understand the API and if they need any help or ask questions, we help them there. I think, there are several, Bitcoin focused- Classes, I believe. I think, his last name is Moon. Yep, Justin Moon. Hosts Justin Moon, Little Bootcamp. I think, there are a couple others that I would recommend, like those, maybe they can be linked in the chat. Yeah, sure, sure. There are a lot of opportunities compared to what we had. 2013, 2014, I believe." }, { "speaker": "stephan", "time": "12:20", "start": 739.62, "text": "Fantastic. And now, let's talk a little bit about Bitrefill. So, one cool thing that we're moving into now, and, I saw, John Cavallo comment about this as well. He spoke about this idea of gearing up for the Lightning Network paradigm, 'cause that just makes a lot more sense. So, how are you guys thinking about this at Bitrefill?" }, { "speaker": "justin_camarena", "time": "12:41", "start": 761.34, "text": "we're basically trying to integrate Lightning in everything that we, we think possible. We're trying to expand, services and sort of on ramp users and earners into this, this system because, we think that with Lightning we'll have a more of a user base that will be able to, will be able to cheaply, be able to earn money and send money at this point. So the paradigm of, expanding and helping push this. This, which is one of the reasons why we started accepting it early on, when many others didn't think it was ready at this point, we, we thought. If we show the world that we can accept Lightning and accept it well, and that users can use this to pay us, there's something to, to buy, you can sort of, jumpstart this." }, { "speaker": "stephan", "time": "13:27", "start": 806.67, "text": "Excellent. And also, one really cool thing with Lightning is just that it, it helps smooth the customer experience in certain ways, so maybe you can comment on that. So com- one example might be if- Comparing back to the old Bitcoin only model, where if a service is provided by that company and then they put up the invoice, but then that customer might pay after the invoice has expired, can you talk a little bit about that?" }, { "speaker": "justin_camarena", "time": "13:56", "start": 836.44, "text": "Yes, so with, accepting orders with Lightning, it's, it's a lot easier than with Bitcoin on chain or even any other altcoins, from what I've seen. as a merchant, I've, we've had experience with all the different issues we had with accepting Bitcoin payments, one of them of course being expiration of the invoice. usually a merchant wants a set timeframe to be able to accept money, because the rate of the, Bitcoin rate may change so much or fluctuate where they wouldn't, they may not want to accept it at that point. with Lightning There's an expiry, expiration time built in so that if, a user pays after a certain time, they're not able to pay the invoice, they just generate a new invoice. So from what we've seen, is it, it helps with, the payment UX, user experience entirely. we also had, had, have had a lot of issues with users, paying under the, the price, like partial orders or paying over the price. Or, most of this is probably due to exchanges, like if the user is in exchange, at times they check the fee from them not being sent out, which causes issues for merchants everywhere. I think that's one of the reasons, BitPay has implemented Taproot, but I think in my opinion, Lightning's a better, fix or sort of improvement to this. It's basically, other than some early bugs We haven't had many payment issues accepting." }, { "speaker": "stephan", "time": "15:18", "start": 918.33, "text": "Excellent. Okay, so let's maybe go now into talking about Bitrefill's Thor service. So can you tell us, first of all, what it is?" }, { "speaker": "justin_camarena", "time": "15:29", "start": 928.68, "text": "Yes. So, one of the biggest issues with Lightning would be, off the bat, being able to receive funds. So with, Lightning, if you have funds, you're able to create channels to different participants, merchants, nodes, but at that point, you would only be able to, to pay people. we thought like early on, we sort of thought of a solution to this, one of my coworkers actually built Thor, and what Thor is, is that, it provides sort of capacity or liquidity for users to be able to receive money over Lightning. So what we do is, we open a channel of a certain capacity, a certain Bitcoin amount. To your node, to your mobile wallet, and we make it easy to do so. Basically, you pay for, the amount you wanna receive. Say you wanna have a channel open to you, Three hundred thousand Satoshi's or even the max amount, point one six Bitcoin. So what we do is you would purchase this, there'd be a fee involved as we have to tie up funds, and it opens a channel to the node, to the user. So then at that point, when the channel is confirmed, they're able to receive money without having to spend from the channels they have. So if, we open the point one six Bitcoin channel, point one six Bitcoin channel to a user, they'll be able to receive that much off the bat." }, { "speaker": "stephan", "time": "16:48", "start": 1007.6, "text": "Right, and I suppose Maybe help the listeners follow along if they're not as familiar with channel management. Let's maybe talk through just an example. So let's say I open a channel to you, Justin, and I open that channel with, say, five million satoshis or zero point zero five BTC. Now, at that point, let's say I opened it to you, at that point I have no incoming or inbound capacity. I only have, you know, capacity to you. Once, let's say I have- Paid you, so let's say out of that five million, I made a payment for one million satoshis, only at that point now I have incoming, capacity, correct? Yes. Yeah. And so I suppose what your service is doing then is Giving people that incoming capacity without having to first spend." }, { "speaker": "justin_camarena", "time": "17:39", "start": 1059.21, "text": "Yes, so, if you are a new user to Bitcoin or even, buying it with Fiat, you wouldn't necessarily have Bitcoin to do this in the first place. And while you can receive after you've opened a channel and spent from it, we provide the service sort of so you can do this off the bat. So it's a, a really good example." }, { "speaker": "stephan", "time": "17:59", "start": 1078.67, "text": "Fantastic. Okay. And so, can you tell us a little bit about what the sort of fees are on this service?" }, { "speaker": "justin_camarena", "time": "18:07", "start": 1086.66, "text": "So, currently, it's a bit arbitrary, I, I, I would think, it's basically based on the fee rates to open and close a channel, and then there's the fee where we, Charge to lock up liquidity, because us opening a channel to any, any person would mean we're locking up a sort of, an amount of money that would only be available to them at that point. So I think for the, the lowest, channel capacity, three hundred thousand satoshis, we charge a dollar Dollar and fifty cents. And at the highest point, it's point one six Bitcoin, which is, I think, twenty dollars." }, { "speaker": "justin_camarena", "time": "18:47", "start": 1126.58, "text": "whether those are reasonable or not, we're always looking into changing the, the fee amounts. the bigger channels are more mostly focused toward maybe a new merchant wants to come online, they don't have any in, inbound capacity, and say they want to receive, they want to receive right away. Before launch, they could use us, and then over time with interest, other nodes would open channels to them automatically." }, { "speaker": "stephan", "time": "19:09", "start": 1149.14, "text": "I understand. And so is it just one upfront fee or any sort of ongoing to maintain the channel?" }, { "speaker": "justin_camarena", "time": "19:16", "start": 1155.87, "text": "So it's a one up upfront fee, but we, we sort of put a time limit, so thirty days now, we guarantee it'll be up. But after that point, we may close the channel, but in my opinion, I think we, we may just leave it open if there's activity in the channel as we earn fees. That are routed through that channel. But we're only hearing T30 days." }, { "speaker": "stephan", "time": "19:39", "start": 1178.67, "text": "I see. And then what are the main-- I guess, yeah, so we've sort of spoken about what are some of the main benefits for the users, but, I suppose one example is you could have capacity to accept the Lightning Torch." }, { "speaker": "justin_camarena", "time": "19:51", "start": 1190.99, "text": "Yes, so from what we saw was, many users weren't used to Lightning or even if they had, already were already using it, they didn't have the capacity to receive the Lightning Torch. So early, early on, we saw more than a couple of users using our service just to be able to receive the Torch and send it out. So that's one great use case." }, { "speaker": "stephan", "time": "20:10", "start": 1210.41, "text": "Well, there you go. Okay. now let's talk a little bit about potential improvements to that service, of the Thor service. Can you talk a little bit about what might be coming down in the, down the pipeline?" }, { "speaker": "justin_camarena", "time": "20:23", "start": 1223.28, "text": "Yeah, so, we really wanna focus on pushing out Lightning and sort of being able to have as much users as possible and for them to be able to earn and spend, spend money. So one improvement we thought of was sort of implementing what- This new scheme called turbo channels. So instead of just the channel being all our money pushed to a user, we can sort of sell a balance on that channel, so selling, selling Bitcoin. And this concept of a turbo channel would be the amount we pushed to you, you're able to use right away when the channel is still unconfirmed. If that makes sense." }, { "speaker": "stephan", "time": "21:01", "start": 1261.33, "text": "Okay, so in this example, let's say we did something like, \"Okay, Justin, you're selling me something for one million satoshis, and I want to open a channel to you for five million satoshis, and we would open-- like, say, I open that channel to you, it's a five million channel, but I push it to you with a one million balance on your side. Is that correct?\"" }, { "speaker": "justin_camarena", "time": "21:23", "start": 1282.62, "text": "No, so it'd be, we would, we would sell a channel to you. And we would open a channel to you, and then we would push the amount of Bitcoin bought on your end." }, { "speaker": "stephan", "time": "21:32", "start": 1292.21, "text": "Oh, sorry, it's the other way around, got you. Yeah, yeah. Yes. Yeah." }, { "speaker": "justin_camarena", "time": "21:35", "start": 1294.57, "text": "So, so, currently with Lightning, or the way the world works right now is we don't push any amount out to the user, 'cause we don't sell Bitcoin in the channel yet. But in the future, we would support this sort of scheme where we can open a channel to you Push like twenty dollars, say if you bought twenty dollars worth of Bitcoin, and you're immediately able to spend that twenty dollars in that channel to anywhere on the network. So this would provide, this would provide a way so you would be able to use Lightning instantly after buying it with, say, an Alcoin or with, with cash or anything, but only from exchanges or sort of these brokerages." }, { "speaker": "stephan", "time": "22:12", "start": 1332.39, "text": "Right. And what would be required to enable this concept of turbo channels? Would it require any sort of protocol? Protocol level change or is it more just something that can be implemented at an application level?" }, { "speaker": "justin_camarena", "time": "22:25", "start": 1344.78, "text": "It's more of a, the, the great thing about Lightning is that, all nodes don't need to be updated for sort of these sort of improvements to be made. It can sort of be, the wallet and the node in question upgrading to support this. It would be like a policy. I mean, I think the current proposal is on the protocol level, but it would be something that, that would be optional. Something that, if both no- If both nodes or wallets supported, they would be able to do it. So currently there's a proposal being proposed for this, and I believe, Bitcoin Lightning Wallet has already implemented this, and he's, and Anton, the developer of this wallet, has a fork of Eclair with this, functionality coming soon." }, { "speaker": "stephan", "time": "23:06", "start": 1386.49, "text": "Fascinating. And I, I'm just wondering then, what are the kind of, are there any other risks or trade-offs around this? Could that money get stolen?" }, { "speaker": "justin_camarena", "time": "23:15", "start": 1395.29, "text": "well, on the receiving side, they aren't really, well, they're, they're trusting us that we're gonna deliver the channel. So basically Thor right now is sort of not trustless, it's sort of like buying a gift card or RIFA from us, you have to trust that we'll deliver the, the money. So in this case, there's no extra risk for the user to buy a triple channel, because, when the money isn't confirmed and being pushed to them There's no, sort of incentive for them to cheat, they're gonna want to be able to push their money out. and for us, we are selling and fulfilling this, this order, so for us, we have a reputation and we don't want to, Sort of double spend this or anything, but there is the, the risk of us double spending a user, but we-- Otherwise, you would just wait for a confirmation. You're not forced to, use this feature." }, { "speaker": "stephan", "time": "24:06", "start": 1445.78, "text": "Right, I see, yeah. So I suppose really it's the same risk as any with a double spend, that if I were to purchase something from you and I thought, \"Oh, that you were untrustworthy,\" well then I would have to-- I would just wait for the confirmation before relying on that balance to make a- Lightning payment." }, { "speaker": "justin_camarena", "time": "24:25", "start": 1464.8, "text": "Right, but, there'd be no risk to you if you used it. Like, it, if it would send, it'd be great for you, if not, then you would wait until confirmation." }, { "speaker": "stephan", "time": "24:32", "start": 1471.69, "text": "Great. and then are there any other upcoming Lightning services that you wanna talk about?" }, { "speaker": "justin_camarena", "time": "24:38", "start": 1478.46, "text": "Right, so initially Thor was just a, a test for us. we didn't think it'd get as much attention, initially. We were maybe thinking maybe it'd be used for merchants' notes that wanted capacity right away or some, some, users that were wanting to receive a lot more. but it sort of- It exceeded our expectations. We have sold a lot of channels and we have a lot of other Lightning services in the works that I can't really talk about right now, but they should be coming out soon." }, { "speaker": "stephan", "time": "25:09", "start": 1509.0, "text": "Fantastic. Okay. Let's now talk, I think, I was quite fascinated about, about just channel management in general, and I was watching, one of Alex Blasworth's talks from the Chaincode Labs residency, and I thought it'd be just interesting to talk to you about How, you know, a couple of things around channel management at Bitrefill. So can you tell us a little bit about your experience on that?" }, { "speaker": "justin_camarena", "time": "25:36", "start": 1536.35, "text": "Right. So, I run one of the-- Bitrefill is one of the biggest nodes on the network currently. As for balancing, I haven't really put much time or thought into balancing out, out, out channels, I have done manual work where I, I open channels to nodes previously. That I would think would have routes that users want, would want to route to. So it was all, it was all done manually. I haven't scripted anything or I balance out channels because I think naturally, other nodes are opening channels to us and sending payments to us, so we sort of already have a naturally occurring balancing for some channels. But one of the issues we had was having funds swept off, 'cause we don't necessarily want to close channels on a node or a user of us, early on. So I think one of the biggest issues we're now we're facing is Sort of hoping that exchanges would adopt us, so maybe we'd be able to temporarily, push out funds and balance at the same time. But I know there's work being done by Alex Balsworth to sort of automate this or having scripts that would do this on its own on the software level." }, { "speaker": "stephan", "time": "26:46", "start": 1605.8, "text": "Can you tell us a little bit more about this concept of sweeping off and the need to sweep off?" }, { "speaker": "justin_camarena", "time": "26:52", "start": 1612.09, "text": "Right, so right now we're still technically in, in beta, so we don't necessarily want to have too much money on our node. So I believe right now, our node is, 21 Bitcoin capacity on the, there's some utility sites that show the balance, but it doesn't really show the real balance. I think on 1ML shows that we have 17.3. But our real capacity is like twenty, twenty-one Bitcoin. so around like a good chunk of that is balance we have tied up in a hot wallet. So one of the challenges is being able to sort of sweep some of that off. so we can sell it via our, for our providers or sort of not have so much tied up online. one of the big reasons why we don't wanna have so much money on a node was, is other than it being obviously a hot wallet, There's still work to be done on the node level to enable backups and safer recovery of funds. The current issue many nodes are that I'm worried about is if our node crashes or for some reason we have some corruption issue, we may lose some funds on our node. So that's why we have a need to sort of sweep or close out channels that have haven't been used in a, in a while or a long time." }, { "speaker": "stephan", "time": "28:06", "start": 1685.91, "text": "Right, and that's interesting as well, because then there's a decision around when you want to try and- Reduce the capacity a little bit, let's say it's gotten really big, and you wanna just try and take some of that out of a hot wallet into a cold wallet, you've gotta decide which channel it is you want to keep. W-h-how do you decide what are the factors around which channel to keep open and which ones to shut?" }, { "speaker": "justin_camarena", "time": "28:29", "start": 1709.3, "text": "Right, so there's, there have been, a lot of visualizers or services that are popping up that, that show channel activity. I've sort of been doing it manually, in the future I hope to automate this. I, I've been looking at channels that haven't been used often that, that have a big balance to us, so I'm manually have been, have been closing those out. Or say, one, one of the other big notes has, say, six channels with us, and they have three of them are completely, all on our side. I may close some of those out, but it's all been manual at this point. But in the future, I hope to have something automatic and I'm sure other notes, or maybe not even have to close out channels in the future. I think in the future with exchanges and more participants will be able to just sort of push out funds over like, that's the goal, at least. Or, there's sort of a tech called splicing, so we may splice out funds instead of completely closing the channel." }, { "speaker": "stephan", "time": "29:27", "start": 1766.57, "text": "Excellent. And do you use any sort of dashboards to monitor the channels, or do you literally just use CLI, character interface, and just list your channels and pick from that?" }, { "speaker": "justin_camarena", "time": "29:38", "start": 1777.86, "text": "There's one, software that's called LN Dash. So LN Dash, I think one of the slash employees or treasurer employees, contributed or somebody else made this. But I run that and I, I can visually see everything. I also run Zap, so Zap's a really good wallet to have on the desktop and on the mobile phone where I'm able to remotely connect to our server and visually see channels and close them out on my phone. So I don't necessarily need to do this on the Command line, and I don't. sometimes I don't. It may, it may depend." }, { "speaker": "stephan", "time": "30:13", "start": 1812.76, "text": "Fascinating. And can we talk a little bit about whether you've faced any sort of-- Are there any kind of attacks or people trying to maybe use up your cha-channel balance or execute some sort of funky attack on you using Lightning Network?" }, { "speaker": "justin_camarena", "time": "30:28", "start": 1828.43, "text": "So one of the cool things with, the Lightning Network is that if they try to use up your channels or sort of route- They're sort of paying us to do so, so there hasn't really been sort of a hacker or anything doing anything malicious in this regard, 'cause they'd be paying us. And I think we sort of raised the, the channel fees for routing just a bit, so we're not at the default, but we're not too expensive either. that sort of helped in allowing nodes to sort of just pick another route, not through us, so we don't become the main central node on the network. we did see some attacks early on when, when we were in beta, I think, there was sort of, not really a hacker, but it was more of a script kiddie sort of trying to ddos our nodes, but, for the most part, it was really easy to sort of mitigate this. on Linux and on servers, there's, I think it's called IP tables, you sort of limit connections or sort of mitigate this. So any, anything that we've had, people have thrown at us, we sort of mitigated it" }, { "speaker": "stephan", "time": "31:31", "start": 1891.3, "text": "Fascinating, okay. And let's talk a little bit about some of the guidelines that I saw Alex Blasworth mention. So he mentioned, now, in fairness, some of these guidelines might be more applicable to the individual who's playing at home as opposed to somebody who's setting up, let's say, a big, routing node like yourself, a Bitrefill. but he mentioned this idea of, you know, don't make too many channels, don't make them too small, don't make them too large, and you want to assign Assign more capacity to channels that get used and in the direction that they're being used, and you want to assign less to channels not being used. So how have you found those sort of guidelines?" }, { "speaker": "justin_camarena", "time": "32:13", "start": 1932.74, "text": "So I think many of those we actually employ ourselves. early on we saw many people creating really small channels to nodes, and where the cost of closing that channel would sort of eat up half of the, the channel. So early on we implemented a minimum channel. So LND has this configuration where you can sort of have a minimum channel needed or capacity needed to open to us directly. I've said that I think at the time was point zero zero one bitcoin, But nowadays, we have so many channels in, in inbound capacity to us, where we don't really have much of an issue with capacity, I've actually increased the, the minimum limit to something like point one five Bitcoin, because at this point, we really don't need more channels, the only new channels we're really being, that are really being opened at this point are big channels to us from other nodes that are interested in providing capacity to us, or, our channels to, to through Thor that are under that limit." }, { "speaker": "stephan", "time": "33:15", "start": 1995.07, "text": "Fantastic. And I think another comment that might be interesting to get your thoughts on is whether Bitrefill or any other large- Bitcoin companies who are taking, who are using Lightning Network, this idea of whether they should become a routing node or whether they shouldn't." }, { "speaker": "justin_camarena", "time": "33:34", "start": 2014.4, "text": "Yeah, so, in the future, I don't envision all merchants or exchanges being, a routing node. we sort of did this in the beginning where we were a hybrid, we were a routing node and a merchant node and all these kinds of things. Sort of because of necessity, like there were no, there was no way for us to get capacity or purchase capacity back then. So today we had, we had, I mean, at the time we had to sort of become a routing node just to be able to reliably accept payments. But in the future, I see sort of exchanges hiding behind other nodes and purchasing liquidity and not being on the, the network. So there's, there's different views on this. many think that merchants can be routing nodes or- They shouldn't be, the-- depending on who you ask, sort of the opinion changes there. But in our take, we sort of become a hybrid, we're a running node and public." }, { "speaker": "stephan", "time": "34:32", "start": 2072.44, "text": "Right, and also, around the, just another idea, around Lightning Network withdrawals that some- Service providers say, \"Oh, we'll do that for free for you if you wanna do a withdrawal via Lightning Network, and we'll only charge you if you're doing an on-chain, withdrawal with Bitcoin.\" Can you talk a little bit about any risks potentially that can arise from that, potentially the fee siphoning attack or any other things around how you're thinking about Lightning Network withdrawals for customers?" }, { "speaker": "justin_camarena", "time": "35:05", "start": 2104.7, "text": "Yeah, so early on I saw that if you allow free withdrawals over Lightning, there's sort of an attack where, you can sort of steal money from an exchange or a node by Having two nodes and, and, and, let's see, or you have two nodes and the route to that single node will go through another no-node that you own, and with the fees being increased, you can sort of attack an exchange that isn't charging for withdrawals, so to speak. So, in my opinion, withdrawals or any APIs shouldn't be doing free lightning withdrawals or at least rate-limiting, rate-limiting it. for us, we've integrated withdrawals and deposits into, Bitrefill accounts. So for withdrawal, we charge five Satoshi's, which may lower, But for Bitcoin withdrawals, we charge, I think, one thousand Satoshi's. So we, we, we wanna make it cheap, but you can't have it for free, otherwise you have this attack vector that can be," }, { "speaker": "stephan", "time": "36:08", "start": 2168.19, "text": "Exploited, yeah. Yes. Okay, so how about the Lightning Network reference rate? So some of Nick Batea's work. Has, has there been much in terms of fee revenue coming in for Bitrefill? And now obviously, I presume it's small, if any, but just curious to know your thoughts." }, { "speaker": "justin_camarena", "time": "36:27", "start": 2186.9, "text": "Yeah, so we've actually made a lot more money, for, from Thor, our, our channel opening service. it has way, I mean, the, the routing revenue so far right now is way minuscule compared to what we earn from Thor. But, in our opinion, it doesn't really matter so much right now because- Like, we, we get a use, use case and we sort of benefit from having a node already without having to, to route. So in our opinion, having a node that doesn't make it, nothing running fees, doesn't really so much matter to us. But in comparison to when Lightning wasn't there, and even the past few months, we've actually have seen an increased activity from fees earned. and maybe some of that is attributed to, us having channels directly to customers or having that service. But we have seen, an increase increase in, in, fee revenue and events and, this kind of area, and I think that may also be, attributed to just how many different wallets we have and services integrating The centralized wallets we've seen from Lightning." }, { "speaker": "stephan", "time": "37:32", "start": 2251.68, "text": "Right, and now, I think, I think those are most of the things I had around Bitrefill's channel management. Let's talk a little bit around, as you touched on there, some of the different Lightning wallets. Do you have any thoughts on just some of the different wallet, you know, what are the different, the different flavors available, and just your thoughts on which ones you might recommend for users, which ones aren't so good?" }, { "speaker": "justin_camarena", "time": "37:57", "start": 2277.44, "text": "Right. So in my opinion, I would rather use, a self-hosted wallet where I am in control of my own funds, but I believe having the choice is also, is also great. I know like every, every exchange that sort of implements or centralized service that integrates Lightning, they sort of in itself become a custodial wallet. But in my opinion, having more choices sort of benefits everyone. centralized wallets are paying fees in routing fees, they are participating, it allows easy, movement of funds between exchanges, between centralized entities, and it also, Allows you to sort of withdraw funds instantly from an exchange. Like say if you hold your own keys, you, you run your own wallet, you're able to withdraw money from an exchange instantly. You don't, you, you don't have the sort, sort of necessity to leave it on an exchange. So, one of the wallets I use, for example, is, Bitcoin Lightning Wallet. It's, sort of a mixed, mixed case. All the keys are in your control, but, and there's like a watchtower service around this. It's only on there's a server that, central server that passes along the routes for, for you. but there are other wallets too that don't have this. I think Eclair is one example, they're only on Android as well, and everything is on the device, keys, routing, And so on, but they don't currently receive funds, so there's sort of trade-offs between all the different wallets. But I, I think in the long term, we'll see most wallets have sending and receiving. There's also another, case of wallets where the centralized services, like custodial wallets, we have, Blue Wallet, have a lot of Satoshi. I'm not really so interested in those as much, but I do like that they're, they're sort of onboarding users to use this wallet initially, and then they can sort of become interested in using it. Sort of a wallet with that uses, that has control of their own keys, they can sort of jumpstart. I, I've been seeing a lot more interest in Lightning because of those, with those wallets, and a lot of more fee revenue for our node because of those wallets. as an example, I gave, Blue Wallet, they are constantly opening big channels to us and Quickly being depleted on our end. Like we've been getting so many users from them paying us, and I'm sure, some of those users have sort of looked at Lightning and found just how great it is in paying, and maybe they're using their own wallet at this point, maybe they became a Thor customer or and so on and so on." }, { "speaker": "stephan", "time": "40:38", "start": 2438.01, "text": "Yeah, it's an interesting point. Obviously, I'm all in favor of the self-sovereign approach, running your own node, holding your own keys, et cetera, but we do have to be cognizant that there are many people out there, it's early days, and there are many people out there who are just not willing to go to the same level that, say, you or I are willing to go to. And especially as we start seeing Bitcoin and Lightning become used more online, let's say, people like streamers who just use it to take payments. While they're on their Twitch stream, they might not necessarily have the technical competence to go and set up in a more self-sovereign way, but hopefully they might start with something like a Blue Wallet or a Wallet of Satoshi, and then over time Change to using, you know, more like a, a wallet where they are holding the keys and they are, running the node underlying that, lightning wallet as well." }, { "speaker": "justin_camarena", "time": "41:35", "start": 2494.61, "text": "Right, so, it's just different types of users. I, I'm all for having the choice. I personally prefer the self-sovereign one, but I mean, we see people have run their own routing nodes and also are users of the network. We have people on mobile wallets, centralized wallets. I think, I would prefer everyone would use, their own self-sovereign wallet, but I mean, I, I can't force anyone to do so, and I think having the choice is at least a good thing." }, { "speaker": "stephan", "time": "42:02", "start": 2522.3, "text": "Yeah, and the other interesting part is just with Bitcoin and Lightning, it's much more interoperable, right? So one, piece of feedback I noticed on Twitter, JP Koning made a comment saying, \"Oh, well, what's the big deal about Twitter if they potentially put in Bitcoin and Lightning? Couldn't they have just done this years ago with their own?\" In-game currency, quote, you know, to use the word. And the way I responded there was to say, actually, there's a slight difference here because Bitcoin and Lightning is interoperable, so if- A person, let's say Twitter someday down the line puts in Lightning payments in, in the DMs or whatever, then users could now withdraw that out into their own Bitcoin Lightning wallet, and that's different to having their own in-game currency." }, { "speaker": "justin_camarena", "time": "42:50", "start": 2569.9, "text": "Yeah, and, and at this point it would be an instant withdrawal, they'd be able to sort of take liquidity out or even move funds from another service that they may not trust as much into something else. Like it sort of allows the, the freedom, the flow of funds to be more instant between services and interoperable. Which I find great." }, { "speaker": "stephan", "time": "43:08", "start": 2587.65, "text": "Excellent. And now I suppose let's just finish up with a bit of talk around Lightning improvements that you're excited about. Are there any things coming down the pipeline or any other things you're excited for?" }, { "speaker": "justin_camarena", "time": "43:20", "start": 2599.82, "text": "Like, not lightning itself, but I'm, I'm, I'm excited to sort of have, people earning Bitcoin, that sort of people creating services sort of so that you can earn on Bitcoin and spend from it. So we have a lot of- Sort of marketplaces where you're able to spend Lightning at, but not so much where you can earn. there have been a few that I haven't interested in. There was one, there was a, I think it was a Monero miner where you would mine, you would CPU mine, and you would be able to withdraw satoshis into your wallet. So, a centralized wallet would probably be easier to start off, like you can sort of have a wallet run this service and withdraw a few satoshis, and then you can even try something sort of like Yaws or something, I'm excited about more use cases like around this earning. Hopefully we can work on some, some in this area, there's also in marketplaces for, Earning in that regard as well. I think there was another one called, Making Bitcoin. I, I forgot the name. It was, they had Lightning support, which I thought was great. as for Lightning itself, the improvements I'm excited about, I think, just sort of having like a node stability, being able to, sort of have channel backups, not having to, be worried about our node being corrupted and going offline and losing a lot of money, I'm excited about watchtowers Anything that makes it easier for us and, and safer for us to host a node with a lot of money in it. I think that would go a long ways in helping adoption with merchants. Other improvements that I'm excited about personally as well is, AMP, so being able to send money through multiple Lightning channels, split up. I think that would help a lot with adoption and sort of help avoid, a lot of the errors we see today with sending big amounts. So you may, you may not need to pay big refill to open a big channel just to receive the Lightning Torch. But you may be able to receive it through many channels. So those are, those are just a few, things I'm excited about, as well as the turbo channels, maybe enabling that in the near future," }, { "speaker": "stephan", "time": "45:31", "start": 2731.28, "text": "Yeah, fantastic. I love those comments. And I think just to touch on your comment earlier about adoption more broadly and people trying to earn in Bitcoin, and I think that's also a theme that, came up in my interview with Sergey as well, actually. and I like, I love that theme. I think because people are-- we are, ultimately, we are greedy and motivated to earn. And so one, one thing that I've noticed that's quite funny is oftentimes People got really excited by the whole idea of Lightning Network reference rate or earning, you know, money from Lightning Network, even though in practice the routing fees aren't that much, but perhaps- Right. I think, I" }, { "speaker": "justin_camarena", "time": "46:12", "start": 2772.16, "text": "think, I think in, in that regard, it's mostly because activity isn't as, as high yet, like there isn't as much adoption yet, but I, I'm sure the, the rates, the fees would, would be a lot more significant if we say have tens of thousands of users, even if with just that we would have a lot more, fees earned. So yeah, it may be viable in the future, right now it may not be, but maybe in the, in the future." }, { "speaker": "stephan", "time": "46:35", "start": 2794.95, "text": "Yeah, no, I, I agreed with you there. And I think it was just funny though, because I would notice how any sort of video talking about, Lightning fee revenue or any sort of content about that would have more views and more listeners and things, even though it wasn't necessarily a reality today. So I think, you know, your comments there about helping people earn in Bitcoin are quite Relevant. Yeah." }, { "speaker": "justin_camarena", "time": "47:01", "start": 2821.31, "text": "But I know there are some, node operators that actually earn a lot more in fees. I know Alex Bosworth earns a multitude of more fees than I do, I think maybe that's due to him better balancing out channels or better strategic opening of channels liquidity to certain nodes. But, there's a future definitely, I think. Like it's only becoming more and more promising. I'm seeing more traffic, more people pay with, Lightning. I'm not sure if I mentioned it, I think we have- More payments in Lightning than we do, than, than all coins in, in most days, I think at this point, it'd be Tetherium and, and Dash and all, all the other shit coins at this point, it's like second only to Bitcoin on chain at this point" }, { "speaker": "stephan", "time": "47:44", "start": 2864.16, "text": "Yeah, that's, that's a great trend to see, and I think we're gonna see more and more of it as the years go on. So, look, Justin, it's been fantastic talking with you. Justin, do you have any, closing thoughts or parting thoughts for the listeners?" }, { "speaker": "justin_camarena", "time": "47:59", "start": 2878.97, "text": "I think just, I hope there are more people interested in Lightning and more companies and developers and users that start to get onboarded because it's, it's exciting tech. I've been in this for a few years and it's, it's, it's amazing to see it finally taking off in some way at this point." }, { "speaker": "stephan", "time": "48:16", "start": 2896.06, "text": "Excellent, and also obviously just tell the listeners where they can find you and where they can find Bitrefill." }, { "speaker": "justin_camarena", "time": "48:22", "start": 2902.26, "text": "Yeah, so I'm on Twitter, j u s c a m e r e n a, just Camarena. We also have a website, Bitrefill is also on Twitter, on Bitrefill, that's Bitrefill the name. We're also available at bitrefill dot com. We sell vouchers, gift cards, services like that. And so on." }, { "speaker": "stephan", "time": "48:40", "start": 2919.55, "text": "Excellent, I'll put those notes in the show notes so, listeners, you can get the links for Justin and for Bitrefill there. Justin, look, I think it's been a really interesting conversation, and I trust that the listeners will find it interesting, and educational as well, just to understand a little bit more about Lightning and channel management. So thank you very much for coming on the show." }, { "speaker": "justin_camarena", "time": "48:59", "start": 2938.75, "text": "Great, thanks for having me on." }, { "speaker": "stephan", "time": "49:01", "start": 2941.05, "text": "So there you go guys, go and check out Justin and Bitrefill, which is a cool place to go and buy stuff with Bitcoin Lightning if you like. As usual, if you enjoyed the episode, make sure you tell a friend about it and you retweet and share it on social media. That's it from me. Thanks for listening, and I'll speak to you soon." } ] }