Increase SD card speed (and file transfer) by roughly 10x for Tanmatsu and Konsool

ProtocolV2.md

@@ -0,0 +1,146 @@
+# BadgeLink Protocol Changes
+
+## Protocol Version Negotiation (Version 2)
+
+This update adds protocol version negotiation to BadgeLink. The previous protocol without version negotiation is considered version 1.
+
+### New Message Types
+
+#### VersionReq (Request tag 7)
+
+Sent by the client to negotiate the protocol version.
+
+| Field | Tag | Type | Description |
+|-------|-----|------|-------------|
+| client_version | 1 | uint32 | Highest protocol version supported by the client |
+
+#### VersionResp (Response tag 6)
+
+Sent by the server in response to a VersionReq.
+
+| Field | Tag | Type | Description |
+|-------|-----|------|-------------|
+| server_version | 1 | uint32 | Highest protocol version supported by the server |
+| negotiated_version | 2 | uint32 | Protocol version to use for this session |
+
+### Negotiation Algorithm
+
+The server calculates the negotiated version as:
+```
+negotiated_version = min(client_version, server_version)
+```
+
+The server stores this negotiated version and uses it for the remainder of the session.
+
+**Important**: The server resets the negotiated version to 1 when a sync packet is received. This ensures each new connection starts fresh with v1 behavior until version negotiation occurs.
+
+### Backwards Compatibility
+
+- **Old server (v1) + New client**: The server responds with `StatusNotSupported`. The client should fall back to version 1 behavior.
+- **New server (v2) + Old client**: The client never sends a VersionReq. The server defaults to version 1.
+- **New server + New client**: Full version negotiation occurs.
+
+### Client Implementation
+
+Recommended flow for new clients:
+
+```
+1. After sync, send VersionReq with client_version = 2
+2. If response is StatusNotSupported:
+   - Server is version 1, use legacy behavior
+3. If response is VersionResp:
+   - Use negotiated_version for session behavior
+   - server_version indicates what features the server supports
+```
+
+### Server API
+
+The server exposes `badgelink_get_protocol_version()` which returns the currently negotiated protocol version (defaults to 1 if no VersionReq was received).
+
+---
+
+## Streaming CRC for Downloads (Version 2)
+
+Version 2 changes how file downloads work to improve performance for large files.
+
+### Version 1 Behavior (Legacy)
+
+1. Client sends download request
+2. Server reads **entire file** to calculate CRC32
+3. Server responds with `size` and `crc32`
+4. Client requests chunks with `XferContinue`
+5. Server sends `download_chunk` responses
+6. Client sends `XferFinish`
+7. Server responds with `StatusOk`
+
+**Problem**: For large files, the server must read the entire file before sending the first byte. This causes significant delays.
+
+### Version 2 Behavior (Streaming CRC)
+
+1. Client sends download request
+2. Server uses `stat()` to get file size (no file read)
+3. Server responds with `size` and `crc32 = 0`
+4. Client requests chunks with `XferContinue`
+5. Server sends `download_chunk` responses, computing CRC incrementally
+6. Client sends `XferFinish`
+7. Server responds with `FsActionResp` or `AppfsActionResp` containing the final `crc32`
+
+**Benefit**: Download starts immediately without reading the entire file first.
+
+### Response Differences
+
+| Event | Version 1 | Version 2 |
+|-------|-----------|-----------|
+| Download start | `crc32` = actual CRC | `crc32` = 0 |
+| XferFinish (FS) | `StatusOk` | `FsActionResp` with `crc32` |
+| XferFinish (AppFS) | `StatusOk` | `AppfsActionResp` with `crc32` |
+
+---
+
+## Python Client Updates
+
+The Python client (`badgelink.py`) has been updated to support protocol version 2.
+
+### New Features
+
+- **Automatic version negotiation**: The client automatically negotiates the protocol version with the server on connection.
+- **Streaming CRC for downloads**: Downloads use streaming CRC verification for v2 servers.
+
+### Command Line Options
+
+```
+--version1    Force protocol version 1 (legacy mode, skip version negotiation)
+```
+
+Use `--version1` when you need to connect using the legacy protocol, for example when testing v1 compatibility or connecting to a known v1 server.
+
+### Example Usage
+
+```bash
+# Normal usage (auto-negotiates version)
+./badgelink.sh fs download /sd/file.bin local_file.bin
+
+# Force version 1 protocol
+./badgelink.sh --version1 fs download /sd/file.bin local_file.bin
+```
+
+---
+
+## Revision History
+
+### 2025-12-16: Field Type Change (uint16 to uint32)
+
+The version fields in `VersionReq` and `VersionResp` were changed from `uint16` to `uint32`.
+
+**Reason**: Protocol Buffers (proto3) does not have a native `uint16` type. The smallest unsigned integer type available is `uint32`. While nanopb can generate C code with `uint16_t` fields using options, this creates inconsistency between the C implementation and other language bindings (Python, etc.) which use `uint32`.
+
+**Changes made**:
+
+| File | Change |
+|------|--------|
+| `badgelink.proto` | Added `VersionReq` and `VersionResp` messages with `uint32` fields |
+| `badgelink.pb.h` | Changed struct fields from `uint16_t` to `uint32_t` |
+| `tools/libraries/badgelink_pb2.py` | Regenerated from updated proto file |
+| `tools/badgelink.py` | Simplified to use generated protobuf classes instead of manual encoding |
+
+**Wire compatibility**: This change is wire-compatible. Protobuf varints encode small values (like version numbers 1, 2, etc.) identically regardless of whether the field is declared as `uint16` or `uint32`. Existing implementations will continue to work.

badgelink.c

@@ -49,6 +49,11 @@ static uint32_t next_serial = 0;
 // Frame refers here to the networking term, not the computer graphics term.
 static uint8_t  frame_buffer[BADGELINK_BUF_CAP];
 
+// Protocol version constants.
+#define BADGELINK_PROTOCOL_VERSION 2
+// Negotiated protocol version (defaults to 1 for backwards compatibility).
+static uint16_t negotiated_version = 1;
+
 // Queue that sends received data over to the BadgeLink thread.
 static QueueHandle_t rxqueue;
 // Handle to the BadgeLink thread.
@@ -58,7 +63,7 @@ static void          badgelink_thread_main(void*);
 
 // Prepare the data for the BadgeLink service to start.
 void badgelink_init() {
-    rxqueue = xQueueCreate(16, sizeof(fragment_t));
+    rxqueue = xQueueCreate(256, sizeof(fragment_t));
 }
 
 // Start the badgelink service.
@@ -128,6 +133,32 @@ void badgelink_send_status(badgelink_StatusCode code) {
     badgelink_send_packet();
 }
 
+// Get the negotiated protocol version.
+uint16_t badgelink_get_protocol_version() {
+    return negotiated_version;
+}
+
+// Handle a version negotiation request.
+static void handle_version_req() {
+    badgelink_VersionReq* req = &badgelink_packet.packet.request.req.version_req;
+    uint16_t client_version   = req->client_version;
+
+    // Negotiate: use the lower of client and server versions.
+    uint16_t negotiated = client_version < BADGELINK_PROTOCOL_VERSION ? client_version : BADGELINK_PROTOCOL_VERSION;
+    negotiated_version  = negotiated;
+
+    ESP_LOGI(TAG, "Version negotiation: client=%u, server=%u, negotiated=%u", client_version,
+             BADGELINK_PROTOCOL_VERSION, negotiated);
+
+    // Send response with server version and negotiated version.
+    badgelink_packet.which_packet                                        = badgelink_Packet_response_tag;
+    badgelink_packet.packet.response.status_code                         = badgelink_StatusCode_StatusOk;
+    badgelink_packet.packet.response.which_resp                          = badgelink_Response_version_resp_tag;
+    badgelink_packet.packet.response.resp.version_resp.server_version    = BADGELINK_PROTOCOL_VERSION;
+    badgelink_packet.packet.response.resp.version_resp.negotiated_version = negotiated;
+    badgelink_send_packet();
+}
+
 // Abort / finish a transfer.
 static void xfer_stop(bool abnormal) {
     switch (badgelink_xfer_type) {
@@ -237,7 +268,9 @@ static void handle_packet() {
             return;
         }
         // Sync packet received; set next expected serial number and respond with the same sync packet.
-        next_serial = badgelink_packet.serial + 1;
+        // Reset negotiated version to 1 for new connections.
+        next_serial        = badgelink_packet.serial + 1;
+        negotiated_version = 1;
         badgelink_send_packet();
         return;
     } else if (badgelink_packet.which_packet != badgelink_Packet_request_tag) {
@@ -291,6 +324,9 @@ static void handle_packet() {
         case badgelink_Request_fs_action_tag:
             badgelink_fs_handle();
             break;
+        case badgelink_Request_version_req_tag:
+            handle_version_req();
+            break;
         default:
             badgelink_status_unsupported();
             break;

badgelink.h

@@ -17,3 +17,6 @@ void badgelink_start(usb_callback_t usb_callback);
 
 // Handle received data.
 void badgelink_rxdata_cb(uint8_t const* data, size_t len);
+
+// Get the negotiated protocol version.
+uint16_t badgelink_get_protocol_version();

badgelink.pb.c

@@ -18,6 +18,12 @@ PB_BIND(badgelink_Response, badgelink_Response, 4)
 PB_BIND(badgelink_StartAppReq, badgelink_StartAppReq, AUTO)
 
 
+PB_BIND(badgelink_VersionReq, badgelink_VersionReq, AUTO)
+
+
+PB_BIND(badgelink_VersionResp, badgelink_VersionResp, AUTO)
+
+
 PB_BIND(badgelink_Chunk, badgelink_Chunk, 4)
 
 

badgelink.pb.h

@@ -100,6 +100,20 @@ typedef enum _badgelink_NvsValueType {
 } badgelink_NvsValueType;
 
 /* Struct definitions */
+/* Protocol version request. */
+typedef struct _badgelink_VersionReq {
+    /* Highest protocol version supported by client. */
+    uint32_t client_version;
+} badgelink_VersionReq;
+
+/* Protocol version response. */
+typedef struct _badgelink_VersionResp {
+    /* Highest protocol version supported by server. */
+    uint32_t server_version;
+    /* Negotiated protocol version (min of client and server). */
+    uint32_t negotiated_version;
+} badgelink_VersionResp;
+
 typedef struct _badgelink_StartAppReq {
     /* App slug. */
     char slug[48];
@@ -287,6 +301,8 @@ typedef struct _badgelink_Request {
         badgelink_StartAppReq start_app;
         /* Transfer control request. */
         badgelink_XferReq xfer_ctrl;
+        /* Protocol version request. */
+        badgelink_VersionReq version_req;
     } req;
 } badgelink_Request;
 
@@ -321,6 +337,8 @@ typedef struct _badgelink_Response {
         badgelink_FsActionResp fs_resp;
         /* NVS action response. */
         badgelink_NvsActionResp nvs_resp;
+        /* Protocol version response. */
+        badgelink_VersionResp version_resp;
     } resp;
 } badgelink_Response;
 
@@ -496,6 +514,10 @@ extern "C" {
 #define badgelink_Request_nvs_action_tag         4
 #define badgelink_Request_start_app_tag          5
 #define badgelink_Request_xfer_ctrl_tag          6
+#define badgelink_Request_version_req_tag        7
+#define badgelink_VersionReq_client_version_tag  1
+#define badgelink_VersionResp_server_version_tag 1
+#define badgelink_VersionResp_negotiated_version_tag 2
 #define badgelink_NvsEntriesList_entries_tag     1
 #define badgelink_NvsEntriesList_total_entries_tag 2
 #define badgelink_NvsActionResp_rdata_tag        1
@@ -505,6 +527,7 @@ extern "C" {
 #define badgelink_Response_appfs_resp_tag        3
 #define badgelink_Response_fs_resp_tag           4
 #define badgelink_Response_nvs_resp_tag          5
+#define badgelink_Response_version_resp_tag      6
 #define badgelink_Packet_serial_tag              1
 #define badgelink_Packet_request_tag             2
 #define badgelink_Packet_response_tag            3
@@ -527,27 +550,42 @@ X(a, STATIC,   ONEOF,    MESSAGE,  (req,appfs_action,req.appfs_action),   2) \
 X(a, STATIC,   ONEOF,    MESSAGE,  (req,fs_action,req.fs_action),   3) \
 X(a, STATIC,   ONEOF,    MESSAGE,  (req,nvs_action,req.nvs_action),   4) \
 X(a, STATIC,   ONEOF,    MESSAGE,  (req,start_app,req.start_app),   5) \
-X(a, STATIC,   ONEOF,    UENUM,    (req,xfer_ctrl,req.xfer_ctrl),   6)
+X(a, STATIC,   ONEOF,    UENUM,    (req,xfer_ctrl,req.xfer_ctrl),   6) \
+X(a, STATIC,   ONEOF,    MESSAGE,  (req,version_req,req.version_req),   7)
 #define badgelink_Request_CALLBACK NULL
 #define badgelink_Request_DEFAULT NULL
 #define badgelink_Request_req_upload_chunk_MSGTYPE badgelink_Chunk
 #define badgelink_Request_req_appfs_action_MSGTYPE badgelink_AppfsActionReq
 #define badgelink_Request_req_fs_action_MSGTYPE badgelink_FsActionReq
 #define badgelink_Request_req_nvs_action_MSGTYPE badgelink_NvsActionReq
 #define badgelink_Request_req_start_app_MSGTYPE badgelink_StartAppReq
+#define badgelink_Request_req_version_req_MSGTYPE badgelink_VersionReq
 
 #define badgelink_Response_FIELDLIST(X, a) \
 X(a, STATIC,   SINGULAR, UENUM,    status_code,       1) \
 X(a, STATIC,   ONEOF,    MESSAGE,  (resp,download_chunk,resp.download_chunk),   2) \
 X(a, STATIC,   ONEOF,    MESSAGE,  (resp,appfs_resp,resp.appfs_resp),   3) \
 X(a, STATIC,   ONEOF,    MESSAGE,  (resp,fs_resp,resp.fs_resp),   4) \
-X(a, STATIC,   ONEOF,    MESSAGE,  (resp,nvs_resp,resp.nvs_resp),   5)
+X(a, STATIC,   ONEOF,    MESSAGE,  (resp,nvs_resp,resp.nvs_resp),   5) \
+X(a, STATIC,   ONEOF,    MESSAGE,  (resp,version_resp,resp.version_resp),   6)
 #define badgelink_Response_CALLBACK NULL
 #define badgelink_Response_DEFAULT NULL
 #define badgelink_Response_resp_download_chunk_MSGTYPE badgelink_Chunk
 #define badgelink_Response_resp_appfs_resp_MSGTYPE badgelink_AppfsActionResp
 #define badgelink_Response_resp_fs_resp_MSGTYPE badgelink_FsActionResp
 #define badgelink_Response_resp_nvs_resp_MSGTYPE badgelink_NvsActionResp
+#define badgelink_Response_resp_version_resp_MSGTYPE badgelink_VersionResp
+
+#define badgelink_VersionReq_FIELDLIST(X, a) \
+X(a, STATIC,   SINGULAR, UINT32,   client_version,    1)
+#define badgelink_VersionReq_CALLBACK NULL
+#define badgelink_VersionReq_DEFAULT NULL
+
+#define badgelink_VersionResp_FIELDLIST(X, a) \
+X(a, STATIC,   SINGULAR, UINT32,   server_version,    1) \
+X(a, STATIC,   SINGULAR, UINT32,   negotiated_version, 2)
+#define badgelink_VersionResp_CALLBACK NULL
+#define badgelink_VersionResp_DEFAULT NULL
 
 #define badgelink_StartAppReq_FIELDLIST(X, a) \
 X(a, STATIC,   SINGULAR, STRING,   slug,              1) \
@@ -692,6 +730,8 @@ extern const pb_msgdesc_t badgelink_Packet_msg;
 extern const pb_msgdesc_t badgelink_Request_msg;
 extern const pb_msgdesc_t badgelink_Response_msg;
 extern const pb_msgdesc_t badgelink_StartAppReq_msg;
+extern const pb_msgdesc_t badgelink_VersionReq_msg;
+extern const pb_msgdesc_t badgelink_VersionResp_msg;
 extern const pb_msgdesc_t badgelink_Chunk_msg;
 extern const pb_msgdesc_t badgelink_FsUsage_msg;
 extern const pb_msgdesc_t badgelink_AppfsMetadata_msg;
@@ -714,6 +754,8 @@ extern const pb_msgdesc_t badgelink_NvsActionResp_msg;
 #define badgelink_Request_fields &badgelink_Request_msg
 #define badgelink_Response_fields &badgelink_Response_msg
 #define badgelink_StartAppReq_fields &badgelink_StartAppReq_msg
+#define badgelink_VersionReq_fields &badgelink_VersionReq_msg
+#define badgelink_VersionResp_fields &badgelink_VersionResp_msg
 #define badgelink_Chunk_fields &badgelink_Chunk_msg
 #define badgelink_FsUsage_fields &badgelink_FsUsage_msg
 #define badgelink_AppfsMetadata_fields &badgelink_AppfsMetadata_msg
@@ -753,6 +795,8 @@ extern const pb_msgdesc_t badgelink_NvsActionResp_msg;
 #define badgelink_Request_size                   4153
 #define badgelink_Response_size                  5134
 #define badgelink_StartAppReq_size               179
+#define badgelink_VersionReq_size                6
+#define badgelink_VersionResp_size               12
 
 #ifdef __cplusplus
 } /* extern "C" */