standardizing decoder performance results

[qodesign]

One of the attractions of using qLDPC to measure code performance is that it integrates other commonly used decoders (MWPM, BP-OSD, ...). This will be even more attractive if circuit noise is added to work uniformly with different decoders. This is more useable than working with individual packages (pymatching, ldpc,...) and figure out the details for each.

It would be nice if code.get_logical_error_rate_func returns results that are consistent across different decoders (when possible).

• "logical error rate" would be split into block logical rate (if any of logicals is wrong) and average (number of logicals that are wrong)

• even though LER is the important metric, it helps if "word error rate" and "bit error rate" are also reported; this is using terminology
  from them classical side. The majority of codes in the literature are CSS codes and are decoded as classical codes.

Anyway, something to keep in mind if these are minor tweaks to the code.

[perlinm]

"logical error rate" would be split into block logical rate (if any of logicals is wrong) and average (number of logicals that are wrong)

We could add a per_logical: bool argument to get_logical_error_rate_func, which if True divides the logical error rate by the number of encoded qubits (or bits, in the case of a classical code). Would that be correct/sufficient?

UPDATE: on further thought, if it's just a matter of dividing the error by the logical dimension, I would rather not add a boolean argument for it, but rather add documentation (and possibly rename functions) appropriately, leading the simple division for the user to perform.

even though LER is the important metric, it helps if "word error rate" and "bit error rate" are also reported

What is the definition of "word error rate" and "bit error rate"?

[qodesign]

They're not related by a simple division. Also I would not select one over the other; I would collect both + the other statistics in a structure that's passed to the outputs.

Here's some python "pseudo code" that I pulled out of something I use. I tried to rename things so they're less cryptic so hopefully I didn't pick up any typos. I actually collect much more statistics which I found useful over the years but these four should be enough.

aBerCount=0
aWerCount=0
bBerCount=0
bWerCount=0
NumWrd=0

Hm,Hn=H.shape  # H matrix (Hx or Hz)
Lm,Ln=L.shape  # L matrix (Lx or Lz)

# p is error probability 
while True:  # for each p

 NumWrd+=1;
 Nx=(np.random.random(Hn)<p).astype(np.uint8) # "original data"
 syndrome=(H@Nx)%2                            #  syndrome (decoder input)
 Rx=dec.decode(syndrome)                      # "recovered data" (decoder output)
 NNx=(L@Nx)%2                                 # "original logical"
 RRx=(L@Rx)%2                                 # "recovered logcal"

 acount=np.count_nonzero(RRx != NNx);       # "logical errors" = difference between original and recovered logicals
 bcount=np.count_nonzero(Rx  != Nx);        # "bit errors"     = difference between original and recovered data
 
 aBerCount+=acount;    # counts all individual errors
 aWerCount+=(acount>0);  # incremented if at least one error, number of individual errors is masked over

 bBerCount+=bcount; 
 bWerCount+=(bcount>0);

 #(exit loop when certain criteria are met)

# after the loop, for each p, save these 4 values : 
logical block error rate = aWerCount/NumWrd;
logical error rate       = aBerCount/(NumWrd*Lm)  # note division by number of logicals
word error rate          = bWerCount/NumWrd;
bit error rate           = bBerCount/(NumWrd*Hn)  # note division by number of qubits

Let me know if this is still not clear.

[perlinm]

I'm not sure I fully understand. My main point of confusion is that I don't see how or why
bcount=np.count_nonzero(Rx != Nx);
is a meaningful quantity to consider for a quantum error correcting code. Generally speaking, the decoder cannot (and need not) distinguish two errors that are related by a stabilizer. It may be the case, for example, that Rx = Nx + S where S is a stabilizer -- this would be a decoding success! But bcount would be nonzero.

Having said that, (a) I think the distinction between block vs. logical error rate makes sense, and (b) word and bit error rates could still in principle be added for classical codes.

[qodesign]

it is meant as a measure of the decoding process even if it is not relevant for qecc;

this can be useful when experimenting with different codes/decoders/decoder parameters;
most times these are decoded as classical codes so this data lets you analyze them as classical codes with a large existing toolset.