In Bitcoin, transaction fees play an important role in ensuring timely and reliable transaction confirmation. However, accurately estimating these fees can be a challenging task, given the ever-shifting market dynamics and competing transaction demands. This challenge is further compounded by the scarcity of blockspace, a finite resource that every transaction competes for.

With new blocks being mined approximately every 10 minutes, it's up to you, the transaction sender, to determine the fee you're willing to pay to secure your transaction's inclusion in the blockchain. A higher fee generally translates into a higher probability of a fast confirmation, while a lower fee may lead to delayed confirmation. But how can you accurately estimate the fee required based on your time preference?

In this post, we'll look at some of the different ways of estimating bitcoin transaction fees, and how you can combine these into an effective fee estimation strategy.

Bitcoin fee estimation types

Accurately predicting future fee trends remains a challenge. While wallets and nodes employ various fee estimation methods, each approach has its strengths and limitations.

There are two primary fee estimation methods that we’ll look at: mempool-based and history-based.

Mempool-based fee estimation

Mempool-based fee estimation focuses on analyzing the current state of the mempool - the collection of unconfirmed transactions vying for block space. By examining the fee rates of transactions recently added to the mempool, this method provides real-time insights into the current fee landscape. This approach is particularly useful for situations where rapid confirmation is desired, as it can provide up-to-date estimates based on the immediate fee dynamics of the actual content of the mempool.

A good example of mempool-based fee estimation is the the Mempool open source project. The team at Muun also use their own custom mempool-based fee estimator, which you can read more about here. They produce this chart on their website that shows how much of a difference a mempool-based approach can make:

As you can see, when it comes to accurately estimating the fee to get a transaction confirmed in the next block, mempool-based estimators like this do a good job; They produce estimates that are more optimal as they are more reactive to short term changes.

History-based fee estimation

In contrast to mempool-based estimation, history-based fee estimation uses historical data to predict future fee trends, analyzing past transaction data to identify patterns and trends in fee fluctuations. By understanding historical fee movements, history-based estimation can provide more stable and predictable fee estimates for transactions with more flexible confirmation targets.

Bitcoin Core - the most widely used Bitcoin node implementation, and the one we use at Strike - uses this history based approach. If you want to see exactly how it generates its fee estimates, you can read the Bitcoin Core fee estimation source code. But to break it down in simple terms, the estimation process involves:

• Grouping transactions into buckets based on similar fee rates
• Tracking how long it takes for transactions in these buckets to be mined
• Using different data sets to track this information over various time horizons (short, medium, and long)
• Applying a decay factor for each dataset to give more weight to recent transactions
• Using success thresholds for different time horizons
• Calculating estimates based on the number of transactions confirmed within the target number of blocks or less

Bitcoin Core's fee estimation uses a moving average and various parameters to provide fee rate estimates under different conditions. Within this, there are two estimation modes that you can select from: economical and conservative.

Economical Mode: This mode estimates fees based on recent blocks, aiming to optimize for lower fees. It is more responsive to the latest changes in the fee market, making it suitable for situations where you are willing to risk longer confirmation times for potentially lower fees.

Conservative Mode: This mode provides estimates that are more likely to ensure faster confirmation, as it considers a longer transaction history. However, it is less responsive to short-term drops in fee rates, often resulting in higher fee recommendations. This mode is better if you want to prioritize transaction confirmation time over fee cost.

Both modes aim to balance cost and confirmation time based on different user preferences and market conditions.

Striking the right balance

At Strike, we make a lot of Bitcoin transactions on behalf of our users. Depending on the scenario, we have differing requirements for our confirmation times. To look at a couple of brief examples:

Transaction batching: When we broadcast on-chain transactions on behalf of our users, we use transaction batching in order to optimize our block space usage. Generally speaking our batching algorithm aims to broadcast a batch of transactions every block. For this to be effective, we need strong guarantees that our transaction will actually make it into the next block, so using a mempool-based fee estimator works best for this use case.

Lightning channel opening/closing: Conversely, when we are opening and closing lightning channels, we may be able to wait longer, and so a history based estimate, like those from Bitcoin Core, can be more appropriate as it enables us to more effectively target blocks further into the future.

Blended fee estimation

Given the benefits and limitations of each individual approach, we use a blended fee estimation method that leverages the advantages of both mempool-based and history-based estimations.

For immediate transactions demanding prompt confirmation within the next few blocks, we prioritize mempool-based estimates for their real-time accuracy.

For transactions with more flexible confirmation targets stretching into the future, we rely on history-based estimates to provide a more stable and predictable fee range.

This blended approach strikes a balance between real-time precision and long-term predictability, enabling us to optimize transaction fees effectively across a wide spectrum of scenarios.

Improving LND's fee estimates

In the rest of this post, we’ll look at how we combined the two techniques - mempool-based and history-based fee estimation.

Given we primarily use the Lightning Network Daemon (LND) for our Lightning node needs, we’ll focus on that; however, a similar approach could be taken for other node implementations.

Custom fee estimates with LND

By default, an LND node that is connected to a Bitcoin Core instance will use the estimatesmartfee RPC from Bitcoin Core to provide its fee estimates. As we saw above, this is far less than ideal in some situations as its estimates are only based on historical data.

Wouldn’t it be better if we could plug a different fee estimator into LND? Well, we’re in luck because the engineers that work on LND thought of that and, of course, already made the fee service configurable.

It might not be immediately apparent, however if you look at the sample config file provided in the LND source code, you’ll find this code block:

; Optional URL for external fee estimation. If no URL is specified, the method
; for fee estimation will depend on the chosen backend and network. Must be set
; for neutrino on mainnet.
; Default:
;   feeurl=
; Example:
;   feeurl=https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json

As you can see, there is a feeurl config option, which can be set to a URL that provides fee estimates. And if we visit the example url from that same snippet, you’ll land on a page that serves some json that looks like this:

{
  "current_block_hash": "0000000000000000000075b86...6303e",
  "fee_by_block_target": {
    "2": 141428,
    "4": 96148,
    "85": 82688,
    "121": 79068
    "24": 171819,
    "25": 86680,
    "144": 58520,
    "504": 56192,
    "1008": 37896
  }
}

Examining this a little, what we notice is that it is providing a series of confirmation target estimates. From the example above, the estimated fee to have a transaction mined within the next 4 blocks is 96,148 (that is fee per kb - so that translates to a fee rate of ~96 sats/vbyte).

This is pretty nice. It means that we can point the feeurl setting to any url that returns json following this format, so in theory we could plug any fee estimation system into that, as long as it can return fee estimates in the expected format.

Using selective mempool-based and historical-based estimates

Our goal was to use mempool-based fee estimates for transactions that should be confirmed within the next few blocks, but continue to use history-based estimates for transactions that should be confirmed further into the future.

When it comes to mempool-based fee estimators, we think the Mempool open source project is one of the better tools out there, so we wanted to use that for our short term fee estimates. And for the longer term estimates we think Bitcoin Core does a pretty good job so we wanted to continue to use that.

To achieve this, we created a Bitcoin Blended Fee Estimator. This is a basic service that provides a single endpoint that serves json with fee estimates in the expected format for LND.

Estimates for a confirmation target of 1 through 6 are provided by the Mempool API, whilst estimates for confirmation targets further out into the future are provided by Bitcoin Core via the Esplora API.

By default it connects to the mempool.space and blockstream.info public APIs, however, it can be configured to connect directly to your own Mempool and Esplora instances to reduce the reliance on these hosted third party APIs.

Running the blended fee estimation service

You can run this service in your own infrastructure alongside your LND node. Or, you can use our hosted version to get up and running quickly (read on for more details).

If you have Docker setup on your machine, you can try it out by running the Docker image which we provide for convenience.

docker run -p 3000:3000 lnzap/bitcoin-blended-fee-estimator:latest

After the docker image has downloaded, you should see the following output indicating that the service is up and running.

---
Using port: 3000
Using base URL: http://localhost:3000
Using Esplora base URL: https://blockstream.info
Using Mempool base URL: https://mempool.space
Using Mempool estimation depth: 6
Using fee multiplier: 1
Using cache stdTTL: 15
Using cache checkperiod: 20
---
Fee Estimates available at http://localhost:3000/v1/fee-estimates

Then, simply navigate to http://localhost:3000/v1/fee-estimates where you should see the the current fee estimates, nicely formatted for LND.

You can set this URL in your LND config file and restart your LND node:

; URL for external fee estimation.
feeurl=http://localhost:3000/v1/fee-estimates

With this in place, LND will now use fee estimates from Mempool for next block fee estimates, and estimates from Esplora for estimates further into the future.

Leveraging the blended fee estimation service

Because we have plugged directly into LND, it will use this for all of its internal fee estimations that target a specific number of blocks into the future such as making on-chain transactions, opening channels, sweeps, etc. Additionally, anything that connects to or builds on LND will also benefit from the new fee estimation sources. So you’ll have a consistent approach to generating fee estimates throughout your stack.

For example, if you are using Lightning Loop to rebalance channels, the fee estimates provided by the Loop service will also be using our custom fee sources since it calls through to LND to get those estimates (we found this to drastically reduce our rebalancing costs by preventing Loop from overpaying onchain fees).

Or if you are building a wallet or other service on top of LND, that too will use the custom fee estimation sources.

Essentially, anything that calls LND’s EstimateFee RPC - either directly or indirectly - will use the new fee estimation source.

A Strike hosted fee estimation service for LND

Using a mix of mempool-based and history-based fee estimates in our tooling has provided us with better estimates for getting transactions mined. This helps keep costs down for our users as well as relieving fee pressure from the Bitcoin blockchain.

We figured it could be useful for other node operators, wallet service providers, or builders - so we have open sourced the code:

https://github.com/LN-Zap/bitcoin-blended-fee-estimator

Additionally, we wanted to make it super simple for you to try this out without needing to deploy an instance of the service to your own infrastructure, so we’ve setup a hosated version of the service for anyone to use. You can access it at the following URL:

https://bitcoinchainfees.strike.me

To use, simply add/update the feeurl setting in your lnd.conf file to point at the endpoint provided by the service:

feeurl=https://bitcoinchainfees.strike.me/v1/fee-estimates

We hope you find the blended fee estimation tool useful! Let us know if it works for you or if you have any questions or suggestions for improvements.

Future improvements

Issues with Bitcoin Core's current history-based fee estimations are fairly well known and understood, and there are multiple efforts and proposals to improve this.

One recent proposal is the idea of Cluster Fee Estimation, emerging from discussions in the broader Bitcoin community. This approach aims to address some of the complexities of the Child Pays For Parent (CPFP) mechanism by clustering linked transactions and evaluating their collective fee rates. To understand the CPFP mechanism's impact on fee estimation and the Cluster Fee Estimation proposal in more detail, this article from Bitcoin Optech is worth a read, or check out this more detailed summary of the proposal from Suhas Daftuar.

Though no fee estimation technique will ever be perfect given the impossible task of predicting the future, we can continue to implement more advanced, robust, and adaptive solutions that deliver overall improvements to the fee estimation problem.

The more we collectively optimize our on-chain transaction fees, the better for everyone; by reducing fee overpayment, we can help avoid the self-perpetuating cycle in which people paying unnecessarily high fees, can drive fee estimates further higher.