Firmware update via SFTP

[brainstorm]

Instead of regular OTA update mechanisms, i.e:

• https://github.com/filipton/esp-hal-ota
• https://github.com/faern/esp-ota
• Simple ota example esp-rs/esp-hal#3629

We can leverage the existing SSH server AAA mechanisms to deploy new firmware securely, see the following for good examples (rough guidance):

https://github.com/azonenberg/staticnet/tree/master/sftp
esp-rs/esp-hal#3354

Ideally this process shouldn't have to involve the bootloader since this should happen at a "userspace + reboot" level, but perhaps there's some limitation I'm not aware of and this approach is not possible?

[jubeormk1]

I am on it

[brainstorm]

Worth building on top of https://github.com/mkj/sunset/commits/dev/sftp-start/

[jubeormk1]

We have made significant progress on the SFTP subsystem in Sunset It was a great experience.

Thanks to this, we have the required building pieces to implement our OTA-SFTP server:

• Upload files to a server
• List server elements
• Obtain properties for server elements
• Download server elements (with a known issue that it is being followed up)
• Trait to tailor our response for most common requests

I will integrate these new features in my ssh-stamp's fork.

[jubeormk1]

SSH-Stamp OTA structure

What follows is my take on the process of OTA verification with ideas borrowed from @mygnu article Building Secure OTA Updates for ESP32 Over BLE with Rust. I am sharing this considerations here as I value other people opinion and sharing them might expose early some flaws and save us all time.

Starting considerations

In his article @mygnu describes the implementation of commands using TLV over BLE. @mygnu defines a series of commands and responses, but the approach of SSH-Stamp where we decided using SFTP over SSH does not provide flexibility to add extra commands. To deal with this situation, there is an easy solution: Adding extra data to the uploaded file in order to capture required metadata.

This approach will add extra steps preparing a binary that can be used in an OTA, but provide us with the tools to add any validations. When I started deriving this idea I was thinking of serialising the required metadata as a monolitic single specified data structure including all the fields required for the validation. After reading the earlier mentioned article, I think that an OTA process would benefit from a Type(Tag)-Length-Value collection of data.

The TLV approach is powerful because older version of firmware may simply ignore, as @mygnu suggest, unknown fields that extend the verifications in newer software revisions.

Metadata to include

As far as I know, it is not required to provide any additional validation for the OTA process in the initial target platform ESP32. However, I believe that it is common sense to at least validate minimally the content of the data and its integrity. For that reason I propose the next minimum fields:

• OTA-Type (First Field): Magic number for SSH-Stamp to identify the file is a SSH-Stamp firmware.
• SHA2 Hash: a hash to validate the integrity of the firmware blob sent.
• Firmware Blob: Nested TLV with the next fields:
  • Length: The length in bytes for the firmware blob
  • Blob: The firmware blob itself

Additionally, and thanks to the extensibility of a TLV approach, we could later define more fields that earlier versions of firmware could ignore (while newer versions might require):

• Target hardware code: Would avoid targeting the wrong MCU target
• Firmware version: We could mandate rejecting old firmwares or previous firmware versions with known flaws?
• OTA Signature: Added as optional but this is open to discussion. An specially critical application of SSH-Stamp could require that the firmware to be loaded over the air (but by a SSH trusted channel) to be implemented by a particular team. Providing an optional compilation parameter with a public key would enable this practice.

When to mark the uploaded application as valid

The initial SSH-Stamp platform is Expressif, and the OTA process requires that after the reset following the OTA the "application" must be mark as valid manually to avoid an automatic app rollback. With that in mind the question of when is an application valid comes to my head.

The obvious point is that if the firmware blob is corrupted, the application is invalid. That being truth does not mean that this point is sufficient to guarantee that our application is useful to the point that we want to keep it.

Making an application valid on reset is merely calling a method, deciding where to call the method is way more complicated. I would argue that some user interaction would be desirable to validate the firmware update and to do so, I would require the user to obtain an SSH or SFTP connection to the device.

OTA Libraries

Last time I checked, the two most popular Espressif OTA libraries in crates.io were esp-hal-ota and esp-ota. I am inclined to use esp-ota since it does not include unsafe code and looks very simple to use.

Other Considerations

• Dependencies consolidation: Where possible, I am going to use dependencies already in use in SSH-Stamp. As an example, I will use the crate SHA2 both for OTA validation and the opaque file handle in the SFTP server implementation.
• Action Feedback: We are using SFTP to PUT firmware in the microcontroller. I have not defined any feedback to the user in case that something goes wrong that would help troubleshooting potential problems. I am going to explore sending different SSH_FXP_STATUS messages, using the error message field as a way to pass feedback back to the client on OTA verification status.
• OTA file preparation: As I mentioned before, I am going to include metadata together with the binary blob. To do so I will add a module that will make an OTA file packing the metadata with the blob.

[mygnu]

Hey there author of the said post on TLV here. Thanks for checking it out, happy to brainstorm and help if I can.

[jubeormk1]

Hi there @mygnu thanks for sharing your thoughts on OTA updates in the article and your minimal BLE OTA repository.

I have noticed in the esp32-ble-ota-tlv that you are not doing app-rollback. Are you doing that for simplicity or because with the hash and signature correct there is a strong certainty that everything is OK?

[mygnu]

It's mostly for simplicity, it's really hard to determine if the firmware is broken in functionality (at least in my case). Hash verification makes sure it's the exact byte by byte firmware released by you. In my case that is good enough.

[jubeormk1]

First version of ota tests in my fork

just testing the encoding/decoding of ota tlv variants. I am sharing it because I made a mayor change in the crate, I introduced a workspace and I am happy to refactor that if it is not ok.

[brainstorm]

First version of ota tests in my fork

just testing the encoding/decoding of ota tlv variants. I am sharing it because I made a mayor change in the crate, I introduced a workspace and I am happy to refactor that if it is not ok.

Why wouldn't it not be ok? Does it interfere with anything else? If not, it'd say go ahead, it's generally a good practice to separate concerns if possible or needed for clarity?

[jubeormk1]

It does not interfere with anything else, except that obviously I have modified the Cargo.toml moving the dependencies used in ota to workspace.dependencies.

I will keep posting. Soon I will have a pull request

[jubeormk1]

As part of my work for the update over SFTP I have prepared a std utility, ota-packer, to include metadata for the given binary.

At the moment this utility is target agnostic, and does not care about the binary that is packing and simply adds a checksum, binary size and application type.

The usage is simple and can be seen in action in the function pack_ota() of the ota test-hil script:

pack_ota(){
    mkdir -p $OUTPUT_DIR
    echo "saving app binary to app.bin"
    espflash save-image --chip esp32c6 $SSH_STAMP_ELF $OUTPUT_DIR/app.bin
    echo "saving app binary to app.ota"
    cargo ota-packer -- $OUTPUT_DIR/app.bin
}

I believe that its functionality could be extended to make it easier to the user to pack a ssh-stamp binary for a given target. My idea is that a future, and more complex ota-packer would:

• Receive a parameter with the target SoC, such as esp32c6
• Read the ssh-stamp version from the Cargo.toml file
• Make sure that all the tools required to build the target are installed
• Run the right build alias
• call espflash save-image to obtain the binary
• pack the current metadata plus target SoC
• write an ota file with a meaninful name like ssh-stamp-1.0.42-esp32c6.ota

I don't necessary believe that we need to make this a priority, but it would make ssh-stamp mass deployment ready.

Another option would be to keep it as a small internal tool and wrap it with a script to create multiple ota packets whenever we release a new version. This could make our life easier too.

What do you think?