Skip to content

Win64 LLP64 ABI support with cross-compiler tooling#630

Open
xdqi wants to merge 3 commits intolkl:masterfrom
xdqi:lkl-win64
Open

Win64 LLP64 ABI support with cross-compiler tooling#630
xdqi wants to merge 3 commits intolkl:masterfrom
xdqi:lkl-win64

Conversation

@xdqi
Copy link
Copy Markdown

@xdqi xdqi commented May 4, 2026

lkl: Win64 LLP64 ABI support with cross-compiler tooling

Summary

This PR enables LKL to be compiled for Win64 (PE) targets using a hybrid LP64/LLP64 toolchain. The core challenge is that Win64 follows the LLP64 data model (long=32-bit) while the LKL kernel internally uses LP64 (long=64-bit). These patches bridge that gap with compile-time typedef remapping, cross-compiler shims, and a custom setjmp/longjmp for the MS ABI.

Changes

1. __lkl_long_t typedef for cross-ABI compatibility

  • New header arch/lkl/include/uapi/asm/lkl_long.h defines __lkl_long_t / __lkl_ulong_t:
    • LLP64 (MinGW-w64 x64): expands to long long (64-bit)
    • LP64 (kernel, Linux, Cygwin, macOS): expands to long (unchanged)
  • headers_install.py auto-replaces long with __lkl_long_t in installed UAPI headers and inserts the required include
  • All host_ops implementations, syscall parameters, ioctl casts, and tests updated accordingly

2. Cross-compiler wrapper scripts (tools/lkl/bin/)

  • x86_64-w64-mingw32-gcc: routes kernel code (-D__KERNEL__) to x86_64-pc-cygwin-gcc (LP64), user-space to real MinGW-w64 gcc (LLP64), with generated specs to a patched linker
  • x86_64-w64-mingw32-cc: symlink to the gcc wrapper
  • Patched x86_64-w64-mingw32-ld: supports PE --image-base=0x10000
  • Patched x86_64-w64-mingw32-objcopy: PE format support

3. Custom setjmp/longjmp for x86_64 MS ABI

  • Replaces libc setjmp.h with raw register save/restore inline assembly
  • Bypasses Windows SEH RtlUnwindEx which cannot handle cross-stack jumps used by LKL's cooperative context switching
  • Falls back to libc setjmp/longjmp on non-Win64 platforms

4. Patched binutils: fix weak externals in PE COFF after ld -r

When ld -r creates a relocatable object (e.g. vmlinux), the tagndx field in PE COFF weak external aux entries is not relocated — it retains stale symbol indices from the original object. All 352 C_NT_WEAK symbols in the LKL vmlinux have incorrect tagndx values: some point to non-code sections (e.g. .data), others to completely unrelated code symbols (e.g. bpf_arena_get_user_vm_startcmd_line_append).

The existing binutils code only fell back to name-based lookup when tagndx pointed to a non-code section, missing the case where it points to a valid-but-wrong code symbol. This caused objcopy -G to resolve weak externals to incorrect addresses, leading to runtime crashes (e.g. ktime_get dereferencing NULL through tk_core->clock).

The fix in coff_nt_weak_to_local() (lkl-binutils@64f5155) always searches for the .weak.<name>.<tag> implementation by name first — PE COFF weak external naming is deterministic and reliable. Only falls back to tagndx when no name match is found.

Repositoriy: xdqi/lkl-binutils (lkl branch, based on binutils-2.25.1)

Toolchain

The build environment is Debian 13 (trixie):

Component Source
MinGW-w64 cross-compiler Debian 13 system packages (gcc-mingw-w64-x86-64, binutils-mingw-w64-x86-64)
Cygwin cross-compiler deb-cygwin snapshotx86_64-pc-cygwin-gcc produces LP64 PE/COFF output directly
Patched binutils Custom ld/objcopy with PE --image-base support (lkl-binutils)

The kernel is compiled with Cygwin GCC (LP64, direct PE output), while user-space libs and tests use MinGW-w64 GCC (LLP64). The __lkl_long_t typedef bridges the ABI between these two worlds.

Disclaimer

This pull request is mostly done by Claude Code, but I audited every line of code :).

xdqi added 3 commits May 4, 2026 17:52
Cross-compiler wrapper (tools/lkl/bin/):
- x86_64-w64-mingw32-gcc: routes kernel code (-D__KERNEL__) to
  x86_64-pc-cygwin-gcc (LP64), user-space to real mingw-gcc (LLP64)
  with a specs file to override linker path
- x86_64-w64-mingw32-cc: symlink to gcc wrapper
- x86_64-w64-mingw32-ld: patched ld supporting PE --image-base
- x86_64-w64-mingw32-objcopy: patched objcopy for PE format

Custom runtime:
- lib/jmp_buf.c: setjmp/longjmp for x86_64 MS ABI (10 callee-saved
  XMM registers + different GPR set vs SysV)
PE64 default image base is 0x140000000. Weak symbols resolve to 0,
causing rel32 relocation overflow (displacement > 2GB). Setting
--image-base=0x10000 keeps all addresses in the low 32-bit range.
On Win64 (MinGW-w64 LLP64), 'long' is 32-bit while the LKL kernel
(compiled with Cygwin LP64 GCC) uses 64-bit 'long'. This causes ABI
mismatches in syscall parameters, UAPI struct layouts, and pointer
casts between user-space and kernel.

Introduce __lkl_long_t / __lkl_ulong_t conditional typedefs
(arch/lkl/include/uapi/asm/lkl_long.h):
- LLP64 (MinGW-w64 x64): expands to 'long long' (64-bit)
- LP64 (Linux/Cygwin/macOS): expands to 'long' (no change)

headers_install.py replaces 'long' with __lkl_long_t in installed
UAPI headers and auto-inserts #include <lkl/asm/lkl_long.h>.

Source files updated to use __lkl_long_t for syscall parameter
arrays, ioctl pointer casts, and return values. Also fix
case-sensitive includes in virtio_net_wintap.c for Linux
cross-compilation.
Copy link
Copy Markdown
Member

@tavip tavip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @xdqi ! Could you please expand a little bit about the benefits of switching to LLP64?

#define _ASM_UAPI_LKL_LONG_H

/*
* __lkl_long_t / __lkl_ulong_t: pointer-width integer types for the
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not lkl_long_t instead of __lkl_long_t?


define nt_host
$(call set_autoconf_var,NT,y)
$(call set_kernel_config,INIT_STACK_ALL_ZERO,n)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the weak externals fix in xdqi/lkl-binutils@64f5155!

I think it is time to move away from comitting binaries. We could use a script that downloads, patches and builds the tools similar with tools/lkl/scripts/dpdk-sdk-build.sh.

@tavip tavip requested a review from thehajime May 4, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants