byte serialization does not match go and Java versions.

[ieugen]

Hello,

While trying to implement nebula-certs in Clojure I encountered a bug with byte serialization.
Protojure reads the data ok but during byte serialization produces "wrong" byte arrays - different from what go and Java.
Hence my code to verify the signature in Clojure fails to work.

Discussion on Slack is here: https://clojurians.slack.com/archives/CNZQPJLD9/p1711152208441789 .

  (def a (file->bytes "ieugen.bytes"))
  (.formatHex hex a)

"0a0669657567656e12098180805080feffff0f2880d9f7af0630b8a5f7be063a200fba4efd592f08b540d0b396994fdfd7f0f123c2ed36fc5e717ff67d9952ae714a20037e9d6aa053411cd9c0c7ee315760b01b0384ff43f7144c4a4a77701e5df134"

  (def arr (protojure/->pb (:Details ieugen-crt)))
  (.formatHex hex arr)

"10818080501080feffff0f2880d9f7af064a20037e9d6aa053411cd9c0c7ee315760b01b0384ff43f7144c4a4a77701e5df13430b8a5f7be060a0669657567656e3a200fba4efd592f08b540d0b396994fdfd7f0f123c2ed36fc5e717ff67d9952ae71"


;; original bytes -> protojure map -> bytes
(.formatHex hex (protojure/->pb (cert/pb->RawNebulaCertificateDetails a)))

"10818080501080feffff0f2880d9f7af064a20037e9d6aa053411cd9c0c7ee315760b01b0384ff43f7144c4a4a77701e5df13430b8a5f7be060a0669657567656e3a200fba4efd592f08b540d0b396994fdfd7f0f123c2ed36fc5e717ff67d9952ae71"

(ns user
  (:require [clojure.java.io :as io]
            [protojure.protobuf :as protojure])
  (:import (java.time Instant)
           (org.bouncycastle.util.io.pem PemObject PemReader)
           (org.bouncycastle.crypto.signers Ed25519Signer)
           (org.bouncycastle.crypto.params AsymmetricKeyParameter
                                           Ed25519PublicKeyParameters)
           (java.util HexFormat)
           (nebula.cert Cert
                        Cert$RawNebulaCertificate
                        Cert$RawNebulaCertificateDetails)))

(defn read-pem
  "Read pem from file, url, etc.
   To pass String, wrap in InputStream."
  (^PemObject [pem]
   (let [pr (PemReader. (io/reader pem))]
     (.readPemObject pr))))


(def rnc (Cert$RawNebulaCertificate/parseFrom (.getContent (read-pem "ieugen.crt"))))

(->> rnc
      (.getDetails)
      (.toByteArray)
      (.formatHex hex))

Cert.java.txt
cert.proto.txt
ieugen.crt.txt
ieugen.bytes.txt
ieugen-crt.bytes.txt

[ieugen]

I got a reply on Slack:

Greg Haskins
10 hours ago
If you were interested in taking a stab at what I think the problem is, I suspect its here:

lib/modules/core/src/protojure/protobuf/serdes/complex.cljc

Line 49 in c841410

;; FIXME: Add support for optimizing packable types

complex.cljc
;; FIXME: Add support for optimizing packable types

[ieugen]

To go around this and still use protojure (clojure maps are nicer to work with) I do round-trip serialization.

I manage my code using protojure and when I need to serialize I do :

protojure -> bytes -> java protocol buffers -> bytes.

e.g.

(defn cert-details->bytes
  "Convert protojure a RawNebulaCertificateDetails map to bytes.
  Uses double serializations since protojure has a serialization bug with
  https://github.com/protojure/lib/issues/164
  Once it's fixed we can use only protojure."
  [details]
  (let [d-bytes ^bytes (protojure/->pb details)
        cert2 (Cert$RawNebulaCertificateDetails/parseFrom d-bytes)
        d-bytes2 (.toByteArray cert2)]
    d-bytes2))