Skip to content

strace-hack.md

Latest commit

 

History

History
180 lines (126 loc) · 6.02 KB

strace-hack.md

File metadata and controls

180 lines (126 loc) · 6.02 KB

Tracking Custom Data with strace via Unknown System Calls

This document describes a lightweight technique for exporting runtime data from an application and observing it externally using strace.

The approach relies on invoking an unused (unknown) system call number with carefully populated arguments. Although the syscall is not implemented by the kernel and fails with ENOSYS, strace still records the attempted invocation and its arguments, which can then be decoded from the trace output.

Rationale

strace intercepts and logs system call invocations along with their arguments and return values.

By deliberately issuing a system call that:

  • uses an unused syscall number, and
  • passes structured data via syscall arguments,

it is possible to repurpose the syscall interface as a debug channel without introducing additional logging support.

This is particularly useful when:

  • instrumentation overhead must be minimal,
  • modifying the runtime environment is undesirable, or
  • attaching a debugger is impractical.

High-Level Approach

  1. Choose an unused syscall number

    • Select a syscall number that is not assigned on the target architecture. A searchable table of Linux syscalls can be found here.
    • On x86_64 Linux, large values (e.g., 10000 / 0x2710) reliably return -ENOSYS
    • The syscall will appear in strace output as syscall_<number> or syscall_0x<number>

  2. Populate syscall arguments with structured data

    • Use integer arguments directly for scalar values
    • Use pointers to user-space buffers for structured payloads
    • All arguments must reference valid user-space memory

  3. Invoke the syscall intentionally

    • Load the unused syscall number into the syscall register
    • Execute the syscall instruction directly
    • The syscall is expected to fail; the return value is not used

  4. Trace execution with strace

    • Run or attach strace to the process
    • Filter for the unknown syscall using a regular-expression filter
    • Decode arguments from the trace output

Calling Convention (x86_64 Linux)

When issuing a syscall directly on x86_64 Linux, registers are used as follows:

Register Purpose
RAX Syscall number
RDI Argument 1
RSI Argument 2
RDX Argument 3
R10 (not RCX) Argument 4
R8 Argument 5
R9 Argument 6

This allows up to six arguments to be passed directly via registers and observed verbatim by strace.

Clobbered Registers

The syscall instruction always clobbers the following registers:

  • RCX
  • R11

These registers must be treated as volatile across the syscall boundary. Inline assembly invoking syscall must declare them as clobbered.

Additionally:

  • RAX is overwritten with the return value
  • On failure, RAX contains -errno (e.g. -ENOSYS)

Tracing with strace

Because the syscall number is unknown to the kernel and to strace, filtering must be done using a regular expression:

strace -f ./program

Notes:

  • strace renders unknown syscalls as syscall_<number> or syscall_0x<number>
  • The syscall will appear even though it fails with ENOSYS

Example

The following code provides an (artificial) example of a syscall with crafted register inputs.

// unused_syscall.c
// x86_64 Linux example: issue an unused syscall number (10000) via the syscall instruction.

#define _GNU_SOURCE
#include <errno.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>

static long do_unused_syscall(uint64_t a1, uint64_t a2, uint64_t a3,
                              uint64_t a4, uint64_t a5, uint64_t a6)
{
    long ret;
    register uint64_t rax asm("rax") = 10000; // unused syscall number (example)
    register uint64_t rdi asm("rdi") = a1;    // arg1
    register uint64_t rsi asm("rsi") = a2;    // arg2
    register uint64_t rdx asm("rdx") = a3;    // arg3
    register uint64_t r10 asm("r10") = a4;    // arg4 (note: r10, not rcx)
    register uint64_t r8  asm("r8")  = a5;    // arg5
    register uint64_t r9  asm("r9")  = a6;    // arg6

    asm volatile (
        "syscall"
        : "+r"(rax)                         // rax in/out (return value)
        : "r"(rdi), "r"(rsi), "r"(rdx),
          "r"(r10), "r"(r8),  "r"(r9)
        : "rcx", "r11", "memory"            // clobbers per syscall ABI
    );

    ret = (long)rax;
    return ret;
}

int main(void)
{
    // Encode some examples in the registers. These will show up in strace as arguments.
    uint64_t a1 = 0x1111111111111111ULL;
    uint64_t a2 = 0x2222222222222222ULL;
    uint64_t a3 = 0x3333333333333333ULL;
    uint64_t a4 = 0x4444444444444444ULL;
    uint64_t a5 = 0x5555555555555555ULL;
    uint64_t a6 = 0x6666666666666666ULL;

    long ret = do_unused_syscall(a1, a2, a3, a4, a5, a6);

    // Linux syscalls return negative errno in rax on failure.
    if (ret < 0) {
        errno = (int)(-ret);
        fprintf(stderr, "unused syscall returned %ld (errno=%d: %s)\n",
                ret, errno, strerror(errno));
    } else {
        fprintf(stderr, "unused syscall returned %ld\n", ret);
    }

    return 0;
}

The example can be compiled with gcc -O0 -g -Wall -Wextra -o unused_syscall unused_syscall.c and then tested with strace with strace -f ./unused_syscall. The relevant output in this case is syscall_0x2710(0x1111111111111111, 0x2222222222222222, 0x3333333333333333, 0x4444444444444444, 0x5555555555555555, 0x6666666666666666) = -1 ENOSYS (Function not implemented). More information on Linux system call conventions can be found on StackOverflow.

Limitations

  • This technique is Linux- and architecture-specific
  • strace output is textual and not suitable for high-throughput data
  • Kernel or strace version differences may affect formatting
  • Not suitable for performance-critical hot paths