Base64 decode error with compressed payloads - 'utf-8' codec can't decode byte 0xb5

[pheuter]

Description

The _decode_base64 function in upstash_workflow/utils.py fails when QStash sends compressed/binary payloads, resulting in UTF-8 and ASCII decode errors.

Error Message

Upstash Qstash: Failed while decoding base64 'KLUv/QBIZQMAwocXGYC5OfhG1KSglKWqu1HS75rcuD2dgzyRFQ0LS9yLoT6yyQfKozST5Rx6q8Zo+Va+sZBDcuj397smUOWumj4y9kBp1csmJ8gZmDEbB9GKAez3C9IPgXgKXgQz9vttigMAg+kq9ZbQrzID'. Falling back to standard base64 decoding. 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte

Environment

• upstash-workflow: 0.1.3
• qstash: 2.0.5
• Python: 3.13.8 (production), 3.13.7 (local)
• Framework: FastAPI
• Workflow: Multi-step workflow with browser automation tasks

Root Cause

The issue is in upstash_workflow/utils.py:15-24:

def _decode_base64(base64_str: str) -> str:
    try:
        decoded_bytes = base64.b64decode(base64_str)
        return decoded_bytes.decode("utf-8")  # ❌ Fails here
    except Exception as error:
        _logger.error(
            f"Upstash Qstash: Failed while decoding base64 '{base64_str}'."
            f" Falling back to standard base64 decoding. {error}"
        )
        return base64.b64decode(base64_str).decode("ascii")  # ❌ Also fails

The base64-decoded bytes contain binary/compressed data (byte 0xb5 at position 1), which cannot be decoded as UTF-8 or ASCII text.

Analysis

Analyzing the problematic base64 string:

import base64
data = base64.b64decode('KLUv/QBIZQMAwocXGYC5OfhG1KSglKWqu1HS75rcuD2dgzyRFQ0LS9yLoT6yyQfKozST5Rx6q8Zo+Va+sZBDcuj397smUOWumj4y9kBp1csmJ8gZmDEbB9GKAez3C9IPgXgKXgQz9vttigMAg+kq9ZbQrzID')
print(data[:10].hex())  # Output: 28b52ffd0048650300c2

The byte sequence suggests compressed data (possibly zlib or another compression format).

Possible Causes

1. Large payloads - QStash may automatically compress large workflow payloads or step results
2. QStash service changes - Recent changes to QStash compression behavior
3. Missing compression handling - The library doesn't handle compressed payloads

Suggested Fix

The _decode_base64 function should detect and decompress compressed payloads:

import base64
import gzip
import zlib
import logging

_logger = logging.getLogger(__name__)

def _decode_base64(base64_str: str) -> str:
    decoded_bytes = base64.b64decode(base64_str)
    
    # Try UTF-8 decoding first (uncompressed)
    try:
        return decoded_bytes.decode("utf-8")
    except UnicodeDecodeError:
        pass
    
    # Try gzip decompression
    if decoded_bytes[:2] == b'\x1f\x8b':
        try:
            return gzip.decompress(decoded_bytes).decode("utf-8")
        except Exception:
            pass
    
    # Try zlib decompression (raw deflate)
    try:
        return zlib.decompress(decoded_bytes, -zlib.MAX_WBITS).decode("utf-8")
    except Exception:
        pass
    
    # Try standard zlib
    try:
        return zlib.decompress(decoded_bytes).decode("utf-8")
    except Exception:
        pass
    
    # Fall back to ASCII
    _logger.warning(f"Failed to decompress base64 payload, falling back to ASCII")
    return decoded_bytes.decode("ascii", errors="ignore")

Impact

This prevents workflows from completing successfully when QStash sends compressed payloads, affecting production reliability.

Request

Could you please:

1. Confirm if QStash can send compressed payloads
2. Update _decode_base64 to handle compressed data
3. Document payload size thresholds or compression triggers
4. Add tests for compressed payload handling

Thank you!