syzbot: ci-upstream-gce-arm64 is broken

[a-nogikh]

The instance broke early-to-mid January 2026.

https://syzkaller.appspot.com/bug?extid=71d58a4b4bc91a412638

failed to run ["ssh" "-p" "22" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "IdentitiesOnly=yes" "-o" "BatchMode=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "root@10.128.1.17" "pwd"]: exit status 255
ssh: connect to host 10.128.1.17 port 22: Connection timed out
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '[us-central1-ssh-serialport.googleapis.com]:9600' (ECDSA) to the list of known hosts.
UEFI firmware (version  built at 09:00:00 on Jan 10 2025)
EMU Variable FVB Started
EMU Variable invalid PCD sizes
Found PL031 RTC @ 0x9010000
InitializeRealTimeClock: using default timezone/daylight settings
�[2J�[01;01H�[=3h�[2J�[01;01H�[2J�[01;01H�[=3h�[2J�[01;01HBdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)

UEFI: Attempting to start image.
Description: UEFI Misc Device
FilePath: PciRoot(0x0)/Pci(0x2,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)
OptionNumber: 1.

  Booting `syzkaller'


serialport: Connected to syzkaller.us-central1-a.ci-upstream-gce-arm64-test-1 port 1 (session ID: ab976aff211ac777ede4703290659ba7bac7803aac8e4770619ed8a1d1047230, active connections: 1).

Some observations:

• Not caused by kernel revision alone? v6.18 (Nov 2025) + the current config we use still fails.
• Not caused just by the config alone? Latest commit + config from Oct 2025: still fails.
• v6.18 + config from Oct 2025: still fails.
• Let's take a crash from Dec 2025: on kernel commit 8f0b4cce4481fb22653697cced8d0d04027cb1e8 (v6.19-rc1) and with the config from that time: still fails

[a-nogikh]

It seems to work well on an older Docker revision, e.g. sha256:73918cd051ff404391e9f171e5879c301768659ab7011728f3996fd84f6eef35 pushed on Dec 3, 2025.

Still a mystery why ci2-linux-6-1-kasan-arm64 works totally fine even though it's in the same container version.

[a-nogikh]

FTR

• The base image: https://storage.googleapis.com/syzkaller/images/buildroot-arm64-img-2025.05.gz
• Docker command: docker run -it --privileged --user root -v ~/linux:/linux -v ~/syzkaller:/syzkaller -v ~/data:/data -v /dev:/dev gcr.io/syzkaller/syzbot@sha256:73918cd051ff404391e9f171e5879c301768659ab7011728f3996fd84f6eef35 /bin/bash
• syz-build command: ./syz-build -arch arm64 -compiler clang -kernel_src /linux -userspace /data/buildroot-arm64-img-2025.05 -config /data/arm64.config

[a-nogikh]

tools/docker/syzbot revisions:

• bee60a8 - the instance boots
• 7cea5f2 - doesn't boot anymore
• c97f8d0 - doesn't boot

[a-nogikh]

Switching to clang-22 doesn't help :(

diff --git a/tools/docker/syzbot/Dockerfile b/tools/docker/syzbot/Dockerfile
index 4f09959ad..dd026699d 100644
--- a/tools/docker/syzbot/Dockerfile
+++ b/tools/docker/syzbot/Dockerfile
@@ -39,23 +39,23 @@ ENV PATH /usr/local/go/bin:$PATH
 RUN apt-get install -y -q gnupg apt-transport-https
 RUN curl https://apt.llvm.org/llvm-snapshot.gpg.key | tee /usr/share/keyrings/llvm-snapshot.gpg
 # It uses SHA-1 signatures which are deprecated since 2026, so we have to use trusted=yes.
-RUN echo "deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/llvm-snapshot.gpg trusted=yes] http://apt.llvm.org/trixie/ llvm-toolchain-trixie-21 main" | tee /etc/apt/>
+RUN echo "deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/llvm-snapshot.gpg trusted=yes] http://apt.llvm.org/trixie/ llvm-toolchain-trixie-22 main" | tee /etc/apt/>
 RUN apt-get update --allow-releaseinfo-change
-RUN apt-get install -y -q --no-install-recommends llvm-21 clang-21 clang-format-21 clang-tidy-21 lld-21 llvm-21-dev libclang-cpp21-dev libclang-21-dev
+RUN apt-get install -y -q --no-install-recommends llvm-22 clang-22 clang-format-22 clang-tidy-22 lld-22 llvm-22-dev libclang-cpp22-dev libclang-22-dev
 RUN apt autoremove -y -q
-RUN sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-21 100
-RUN sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-21 100
-RUN sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-21 100
-RUN sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-21 100
-RUN sudo update-alternatives --install /usr/bin/ld.lld ld.lld /usr/bin/lld-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-nm llvm-nm /usr/bin/llvm-nm-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-ar llvm-ar /usr/bin/llvm-ar-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-objcopy llvm-objcopy /usr/bin/llvm-objcopy-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-objdump llvm-objdump /usr/bin/llvm-objdump-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-addr2line llvm-addr2line /usr/bin/llvm-addr2line-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-readelf llvm-readelf /usr/bin/llvm-readelf-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-strip llvm-strip /usr/bin/llvm-strip-21 100
-RUN sudo update-alternatives --install /usr/bin/llvm-link llvm-link /usr/bin/llvm-link-21 100
+RUN sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-22 100
+RUN sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-22 100
+RUN sudo update-alternatives --install /usr/bin/clang-tidy clang-tidy /usr/bin/clang-tidy-22 100
+RUN sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-22 100
+RUN sudo update-alternatives --install /usr/bin/ld.lld ld.lld /usr/bin/lld-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-nm llvm-nm /usr/bin/llvm-nm-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-ar llvm-ar /usr/bin/llvm-ar-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-objcopy llvm-objcopy /usr/bin/llvm-objcopy-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-objdump llvm-objdump /usr/bin/llvm-objdump-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-addr2line llvm-addr2line /usr/bin/llvm-addr2line-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-readelf llvm-readelf /usr/bin/llvm-readelf-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-strip llvm-strip /usr/bin/llvm-strip-22 100
+RUN sudo update-alternatives --install /usr/bin/llvm-link llvm-link /usr/bin/llvm-link-22 100

[pimyn-girgis]

I know we already ruled out that clang-15 might be responsible, but it seems that it's now available as a trixie package. So we don't need to use the bookworm package instead.

Maybe it's worth a try?

[a-nogikh]

Could you please prepare a PR that installs clang-15 directly? Would be nice to do in any case.

[pimyn-girgis]

I was mistaken. Even though the entry exists now, the package files for amd64 and arm64 are empty.

[a-nogikh]

Seems to work on newer clang versions without CONFIG_KASAN.