[PROPOSAL:CONSENSUS][WIP] Simple Extended Consensus Resolution

docs/SECOR.md

@@ -0,0 +1,196 @@
+# SECOR
+
+https://github.com/masari-project/research-corner/blob/master/secor/secor.pdf
+
+## Proposal Summary
+
+I would like to increase the speed of Nerva transactions by integrating the "Simple Extendend Consensus Resolution" (SECOR) protocol into Nerva.
+
+The goal is to better define how the network should respond to temporary chain forks when an orphan block is found and reduce deep chain reorganizations.
+
+### Consensus
+
+When two blocks are found for the top block height, all network participants should reorganize so that the block with Proof-of-Work which clears the highest difficulty should be on top.
+
+Miners should include the hash of the losing block in their next block via the `uncle_hash` block field.
+
+A miner will be rewarded an additional bonus for including that uncle block hash in their block, and the difficulty for the uncle block will be included in the chain cumulative difficulty, giving a miner who includes a valid uncle hash an advantage over miners who may choose not to.
+
+### Miner Transaction Format
+
+Miner transactions should include only 1 reward output if the uncle_hash is null.
+
+If an uncle block is included in the block, then the miner transaction must include a secondary transaction output.
+
+In order to generate a transaction on behalf of another miner in a non-interactive way without compromising privacy, we re-use the keys associated with the original uncle block miner transaction from the alt chain into the nephew block.
+
+Since output index is a part of the stealth address generation algorithm, `Hs(aR|i)*G + B = Hs(rA|i)G + B` (see cryptonote whitepaper section 4.3), the ordering of the output destinations in the transaction is important.
+
+If a block doesn't reference an uncle block, then there should be only one output in the reward transaction, which should be used to reward the person who mined the block. If that block is then included into a nephew block as an uncle block, then the nephew block must place the reward for the uncle miner, using the output key from the uncle block miner tx, into the nephew block miner transaction at index 0. The reward for the person who found the nephew block should be placed at index 1 in the reward transaction.
+
+If an uncle block references an uncle block itself, then the reward to the true uncle miner should be at index 1 in the uncle block. To accomodate this, the reward to the nephew miner in the main chain should be addressed to output index 0 so that the uncle reward from index 1 may be recycled.
+
+### Block Reward Bonuses
+
+I use the reward constants proposed in the original SECOR paper: +5% of the base reward amount to the nephew miner and 50% of the base reward amount to the uncle miner. The impact of this is detailed in the "Block Reward Rationale" section.
+
+The optimal reward for Nerva's implementation is still an active research question.
+
+### Block Time Target and Transaction Maturity
+
+Nerva currently has a target block time of 1 minute and requires a maturity of 20 blocks to spend non-miner transaction outputs. I propose to use a 15 second block time as proposed in the original SECOR paper. The first (and only known?) project to integrate SECOR with CryptoNote was Masari, which decreased from a 2 minute to a 1 minute block time. Therefore, real-world behavior with a 15 second block time is currently unknown. Further research should be done to select/confirm the optimal block time for Nerva.
+
+With a 20 block transaction maturity threshold and a 15 seconds block target, it should take about 5 minutes for a transaction to mature. If we succeed in preventing blockchain reorganizations > depth 1, then we may be able to decrease the threshold to only 5 blocks, or 75 seconds.
+
+## Block Reward Rationale
+Maximum block reward emission per minute, R, given base reward X:
+```
+R = X + (X/2) + (X/20)
+R = X + (10X/20) + (X/20)
+R = X + (11X/20)
+```
+
+If we keep the base emission reward the same, X=0.3, then...
+```
+R = 0.3 + ((11*0.3)/20) = 0.465
+```
+
+If we preserve the current emission target of R=0.3 then...
+```
+0.3 = (20X/20) + (11X/20) = 31X/20
+0.3 * 20 = 31X
+X = (0.3 * 20) / 31 = 0.19354838709
+```
+
+If you have an estimate of how frequently uncle blocks are included in the blockchain, Z, then you can scale the reward to produce the same estimated average emission.
+```
+R = X + (((X/2) + (X/20)) * Z)
+R = X + ((11X/20) * Z)
+```
+
+If we think that uncle blocks will be included in half of the main chain blocks then...
+```
+0.3 = X + ((11X/20) * 0.5)
+0.3 = X + (11X/40)
+0.3 = 51X / 40
+X = (0.3 * 40) / 51 = 0.23529411764
+```
+
+## Modeling Orphan/Uncle Block Frequencies
+
+Mining is essentially an [IID process](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables) as the output of the hashing function used for mining should be an IID random variable. Each time the miner chooses a nonce and produces a hash, the miner has either found a valid block for the given difficulty, or they have not and must attempt again with a successive independent trial. The probabilities for this kind of binary problem follow a [bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution). Therefore, a [binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution) where k=1, aka a [geometric distribution](https://en.wikipedia.org/wiki/Geometric_distribution), can be used to model the cumulative probability of finding a block after mining `n` hashes.
+
+A block difficulty target is used to control the rate at which blockchain can be written to. Nerva currently has a block time target of 1 block per minute. If there are 10 miners on the network, each mining at a rate of 5 hashes/second, then the target difficulty should eventualy fit to about `10*5*60`.
+
+#### Scenario A - Few miners, low difficulty.
+```
+Suppose:
+There is a target block time of 60 seconds.
+There are 2 miners, miner A and miner B, on the network, each mining at a rate of 1 H/s.
+
+Then:
+The network difficulty should be 1*2*60=120
+The probability that miner A finds a block after 1 second of hashing is 1/120.
+The probability that miner B has found a block after 1 second of hashing is also 1/120.
+
+The probability that miner A finds a block after 2 seconds of hashing is 2/120.
+The probability that miner B has found a block after 2 seconds of hashing is also 1/120.
+
+The probability that miner A and miner B both find a block after 1 second of mining (at the same time) is (1/120)*(1/120) per IID joint probability.
+
+The cumulative probability that miner A finds a block after 2 seconds of mining is 1 - (1-(1/120))^2 per geometric CDF
+The cumulative probability that miner B finds a block after 2 seconds of mining is 1 - (1-(1/120))^2 per geometric CDF
+
+The probability that miner A and miner B have both found a block after 2 seconds of mining is (1 - (1-(1/120))^2)^2
+
+The probability that miner A and miner B have both found a block after 10 seconds of mining is (1 - (1-(1/120))^10)^2
+
+The probability that a given miner finds a block after mining for 1 minute is 1 - ((1-(1/120))^60), roughly 40%.
+
+The probability that both miners have each found a block after 1 minute of mining is (1 - ((1-(1/120))^60))^2, which is roughly 15%.
+```
+
+#### Scenario B - Few miners, higher difficulty.
+```
+Suppose:
+There is a target block time of 60 seconds.
+There are 2 miners, miner A and miner B, on the network, each mining at a rate of 1,000 H/s.
+
+Then:
+The network difficulty should be 1000*2*60=120000
+The probability that miner A finds a block after mining a single hash is 1/120000.
+The probability that miner B has found a block after mining a single is also 1/120000.
+
+The probability that miner A and miner B both find a block at the same time after each mining a single hash is (1/120000)^2 per IID joint probability.
+
+The probability that a given miner finds a block after mining for 1 second is 1 - ((1-(1/120000))^1000) per geometric CDF.
+
+The probability that miner A and miner B have both found a block after mining 10 hashes is (1 - (1-(1/120000))^10)^2
+
+The probability that miner A and miner B have both found after 1 second of mining is (1 - (1-(1/120000))^1000)^2
+
+The probability that a given miner finds a block after mining for 1 minute is 1 - ((1-(1/120000))^60000), which is roughly 40%. This is the same as when the network hashrate is only 2 H/s.
+
+The probability that both miners have each found a block after mining for 1 minute is (1 - ((1-(1/120000))^60000))^2, which is roughly 15%.
+```
+
+#### Scenario C - Many miners
+```
+Suppose:
+There is a target block time of 60 seconds.
+There are 100 miners on the network, each mining at a rate of 1,000 H/s.
+
+Then:
+The network difficulty should be 1000*100*60=6000000
+The probability that a given miner finds a block after mining a single hash is 1/6000000.
+
+The probability that 2 miners both find a block at the same time after each mining a single hash is (1/6000000)^2 per IID joint probability.
+
+The probability that a given miner finds a block after mining for 1 second is 1 - ((1-(1/6000000))^1000) per geometric CDF.
+
+The probability that two miners have both found a block after mining 10 hashes is (1 - (1-(1/6000000))^10)^2
+
+The probability that two miners have both found after 1 second of mining is (1 - (1-(1/6000000))^1000)^2
+
+The cumulative probability that a given miner finds a block after mining for 1 minute is 1 - ((1-(1/6000000))^60000), which is just less than 1%.
+
+The probability that any two miners have each found a block after mining for 1 minute is (100*99/2)*((1-(1-(1/6000000))^(60000))^2)*(((1-(1/6000000))^(60000))^98), per binomial distribution (100, 2) which is about 18%
+```
+
+Overall, the network difficulty adjusts in a way that adding miners to the network doesnt affect the probability of chain splits significantly assuming that mining power is distributed evenly among miners. This model does not account for the fact that adding miners to the network increases the amount of work required for block propogation across all miners on the network.
+
+### Mining Difficulty Considerations
+
+The current difficulty algorithm adjusts based on the observed difference between block timestamps of recently found blocks compared against the target solve time. We will be lowering that target solve time from 60 seconds to 15 seconds. The algorithm should adjust quickly, but it will be a bit wonky for the first ~60 blocks after hard forking.
+
+Currently the LWMA algorithm uses the parameter N=60 so we are using the most recent 60 blocks when calculating the weighted average. Since we are lowering the target block time, that means we will only be using 15 minutes worth of block histories instead of the last 1 hour worth of blocks. Is this OK?
+
+The "future time limit" used in the difficulty algorithm is intended to be 500 seconds according to the comments in the code. Currently this is, in fact, hard coded to 300 seconds (`CRYPTONOTE_BLOCK_FUTURE_TIME_LIMIT_V6`). Since we are lowering the target solve time, the (FTL / T) ratio increases. Should this be adjusted?
+
+## Status
+
+This proposal is a work in progress. I am seeking community feedback to see if I should continue working on this idea, and to get an idea of what people want to do about the block reward.
+
+### TODOs
+
+#### Coding
+
+* Address the TODO comments
+
+* graceful database migration - currently a sync from scratch is required after upgrading
+
+#### Research
+
+* Confirm that the difficulty algorithm doesnt require further adjustment
+
+* How long does it currently take for a block to propogate throughout the network ?
+
+* Quantify frequency of orphans / chain re-organizations
+
+* Decide transaction maturity blocks #
+
+#### Testing
+
+* Unit test the whole thing
+
+* Deploy to testnet

scripts/analyze_alt_chains.py

@@ -0,0 +1,30 @@
+import json
+import datetime
+import requests
+
+DAEMON_HOST = 'http://127.0.0.1:17566/json_rpc'
+
+def get_block(block_hash:str) -> dict:
+    r = requests.post(DAEMON_HOST, json={'method': 'get_block', 'params': {'hash': block_hash}})
+    return json.loads(r.json()['result']['json'])
+
+def get_block_by_height(height:int) -> dict:
+    r = requests.post(DAEMON_HOST, json={'method': 'get_block', 'params': {'height': height}})
+    return json.loads(r.json()['result']['json'])
+
+def get_alt_chains():
+    r = requests.post(DAEMON_HOST, json={'method': 'get_alternate_chains'})
+    chains = r.json()['result']['chains']
+    sorted_chains = sorted(chains, key=lambda alt_chain: alt_chain['height'])
+    return sorted_chains
+
+if __name__ == '__main__':
+    prev_alt_chain_height = None
+    for alt_chain in get_alt_chains():
+        alt_block_timestamp = get_block(alt_chain['block_hashes'][-1])['timestamp']
+        main_chain_sibling_timestamp = get_block_by_height(alt_chain['height'])['timestamp']
+        print(f'Alt chain found at height {alt_chain["height"]} with head {alt_chain["block_hashes"][-1]}')
+        print(f'  Alt chain length: {alt_chain["length"]}')
+        print(f'  Timestamp: {datetime.datetime.fromtimestamp(alt_block_timestamp)}; Difference from mainchain: {abs(alt_block_timestamp - main_chain_sibling_timestamp)}s')
+        print(f'  Height difference from previous alt chain head: {None if not prev_alt_chain_height else alt_chain["height"] - prev_alt_chain_height}')
+        prev_alt_chain_height = alt_chain["height"]

scripts/analyze_tx_frequency.py

@@ -0,0 +1,55 @@
+import json
+import requests
+
+DAEMON_HOST = 'http://127.0.0.1:17566/json_rpc'
+
+class BlockchainRPCIterator:
+    '''
+    Iterate from the top of the chain to the origin block via RPC
+    '''
+    def __init__(self):
+        self.session = requests.Session()
+        self.next_block_hash, self.height = self.get_top_block()
+
+    def __iter__(self):
+        return self
+
+    def __next__(self):
+        block = self.get_block(self.next_block_hash)
+        if not block:
+            raise StopIteration
+        self.next_block_hash = block['prev_id']
+        self.height = block['miner_tx']['vin'][0]['gen']['height']
+        return block
+
+    def get_top_block(self) -> (str, int):
+        r = self.session.post(DAEMON_HOST, json={'method': 'get_info'})
+        return r.json()['result']['top_block_hash'], r.json()['result']['height']
+
+    def get_block(self, block_hash:str) -> dict:
+        try:
+            r = self.session.post(DAEMON_HOST, json={'method': 'get_block', 'params': {'hash': block_hash}})
+            return json.loads(r.json()['result']['json'])
+        except KeyError:
+            return
+
+if __name__ == '__main__':
+
+    empty_block_counter = 0
+    blocks_found = 0
+    total_blocks_scanned = 0
+
+    for block in BlockchainRPCIterator():
+        non_miner_txs = block['tx_hashes']
+        total_blocks_scanned += 1
+
+        if block['tx_hashes']:
+            blocks_found += 1
+            # print(block)
+            print(f'{empty_block_counter} block searched since last block')
+            print(f'{blocks_found} / {total_blocks_scanned} are not empty ({(float(blocks_found)/float(total_blocks_scanned))*100.0}%)')
+            empty_block_counter = 0 # reset counter to get gap until next non-empty block
+        else:
+            empty_block_counter += 1
+
+    print(f'{100.0 - ((float(blocks_found)/float(total_blocks_scanned))*100.0)}% blocks are empty')

src/blockchain_db/blockchain_db.cpp

@@ -167,6 +167,7 @@ uint64_t BlockchainDB::add_block( const std::pair<block, blobdata>& blck
                                 , const difficulty_type_128& cumulative_difficulty
                                 , const uint64_t& coins_generated
                                 , const std::vector<std::pair<transaction, blobdata>>& txs
+                                , const crypto::hash& uncle_blk_hash
                                 )
 {
   const block &blk = blck.first;
@@ -206,7 +207,7 @@ uint64_t BlockchainDB::add_block( const std::pair<block, blobdata>& blck
 
   // call out to subclass implementation to add the block & metadata
   time1 = epee::misc_utils::get_tick_count();
-  add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, num_rct_outs, blk_hash);
+  add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, num_rct_outs, blk_hash, uncle_blk_hash);
   TIME_MEASURE_FINISH(time1);
   time_add_block1 += time1;
 

src/blockchain_db/blockchain_db.h

@@ -384,6 +384,7 @@ class BlockchainDB
                 , const uint64_t& coins_generated
                 , uint64_t num_rct_outs
                 , const crypto::hash& blk_hash
+                , const crypto::hash& uncle_blk_hash
                 ) = 0;
 
   /**
@@ -837,6 +838,7 @@ class BlockchainDB
                             , const difficulty_type_128& cumulative_difficulty
                             , const uint64_t& coins_generated
                             , const std::vector<std::pair<transaction, blobdata>>& txs
+                            , const crypto::hash& uncle_blk_hash
                             );
 
   /**

src/blockchain_db/lmdb/db_lmdb.cpp

@@ -679,7 +679,7 @@ uint64_t BlockchainLMDB::get_estimated_batch_size(uint64_t batch_num_blocks, uin
 }
 
 void BlockchainLMDB::add_block(const block& blk, size_t block_weight, uint64_t long_term_block_weight, const difficulty_type_128& cumulative_difficulty, const uint64_t& coins_generated,
-    uint64_t num_rct_outs, const crypto::hash& blk_hash)
+    uint64_t num_rct_outs, const crypto::hash& blk_hash, const crypto::hash& uncle_blk_hash)
 {
   LOG_PRINT_L3("BlockchainLMDB::" << __func__);
   check_open();
@@ -743,6 +743,8 @@ void BlockchainLMDB::add_block(const block& blk, size_t block_weight, uint64_t l
   }
   bi.bi_long_term_block_weight = long_term_block_weight;
 
+  bi.uncle_blk_hash = uncle_blk_hash;
+
   MDB_val_set(val, bi);
   result = mdb_cursor_put(m_cur_block_info, (MDB_val *)&zerokval, &val, MDB_APPENDDUP);
   if (result)
@@ -4079,7 +4081,7 @@ void BlockchainLMDB::block_rtxn_abort() const
 }
 
 uint64_t BlockchainLMDB::add_block(const std::pair<block, blobdata>& blk, size_t block_weight, uint64_t long_term_block_weight, const difficulty_type_128& cumulative_difficulty, const uint64_t& coins_generated,
-    const std::vector<std::pair<transaction, blobdata>>& txs)
+    const std::vector<std::pair<transaction, blobdata>>& txs, const crypto::hash& uncle_blk_hash)
 {
   LOG_PRINT_L3("BlockchainLMDB::" << __func__);
   check_open();
@@ -4097,7 +4099,7 @@ uint64_t BlockchainLMDB::add_block(const std::pair<block, blobdata>& blk, size_t
 
   try
   {
-    BlockchainDB::add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs);
+    BlockchainDB::add_block(blk, block_weight, long_term_block_weight, cumulative_difficulty, coins_generated, txs, uncle_blk_hash);
   }
   catch (const DB_ERROR_TXN_START& e)
   {

src/blockchain_db/lmdb/db_lmdb.h

@@ -66,7 +66,21 @@ typedef struct mdb_block_info_4
   uint64_t bi_long_term_block_weight;
 } mdb_block_info_4;
 
-typedef mdb_block_info_4 mdb_block_info;
+typedef struct mdb_block_info_5
+{
+  uint64_t bi_height;
+  uint64_t bi_timestamp;
+  uint64_t bi_coins;
+  uint64_t bi_weight;
+  uint64_t bi_diff_hi;
+  uint64_t bi_diff_lo;
+  crypto::hash bi_hash;
+  uint64_t bi_cum_rct;
+  uint64_t bi_long_term_block_weight;
+  crypto::hash uncle_blk_hash;
+} mdb_block_info_5;
+
+typedef mdb_block_info_5 mdb_block_info;
 
 typedef struct txindex {
     crypto::hash key;
@@ -348,6 +362,7 @@ class BlockchainLMDB : public BlockchainDB
                             , const difficulty_type_128& cumulative_difficulty
                             , const uint64_t& coins_generated
                             , const std::vector<std::pair<transaction, blobdata>>& txs
+                            , const crypto::hash& uncle_blk_hash
                             );
 
   virtual void set_batch_transactions(bool batch_transactions);
@@ -402,6 +417,7 @@ class BlockchainLMDB : public BlockchainDB
                 , const uint64_t& coins_generated
                 , uint64_t num_rct_outs
                 , const crypto::hash& block_hash
+                , const crypto::hash& uncle_blk_hash
                 );
 
   virtual void remove_block();