syz-ci: syzkaller update failures

[a-nogikh]

Observed today on syzbot

Instance: ci-upstream-kasan-gce-root

Log line:
aborting RPC server: mismatching manager/executor git revisions: 50cdcaa1adff11dc8afdfa7ec9e390b9c5b72c3f vs 766b643402ed0591709000628ef6968f365d8630

[pimyn-girgis]

Commit ee48975 is the most suspicious, since it's related to git revisions, but I'm not sure what specifically is happening.

[pimyn-girgis]

@tarasmadan to continue our offline discussion,

Looking at the logs, I noticed that in some cases the manager and executor had revisions A, B respectively, and in other cases it was B, A respectively. As such, I don't think the conclusion that the manager was somehow connecting to an already existing VM with an old executor is a right one.

[a-nogikh]

A case observed on syzbot:

We try to build b5ceaad20a1c1f911045a12770d98c0d99300b1b, but fail:

Mar 28 14:14:48 ci docker[63825]: 2026/03/28 14:14:48 ci: syzkaller: poll: b5ceaad20a1c1f911045a12770d98c0d99300b1b (sys/linux: increase coverage with modified dynamic i
Mar 28 14:14:48 ci docker[63825]: 2026/03/28 14:14:48 ci: syzkaller: building ...
< .. >
Mar 28 14:37:06 ci docker[63825]: 2026/03/28 14:37:06 [ERROR] ci: syzkaller: failed to build: testing failed: failed to run ["go" "test" "-short" "./..."]: exit status 1

Some time later, ci-upstream-kasan-gce had already been running for 15-20 minutes.

2026/03/29 06:03:53 ci-upstream-kasan-gce: candidates=36796 corpus=29571 coverage=154709 exec total=127184 (149/sec) pending=0 reproducing=0
2026/03/29 06:03:53 ci-upstream-kasan-gce: deleting instance: ci-upstream-kasan-gce-4
2026/03/29 06:03:55 ci-upstream-kasan-gce: VM 4: crash: INFO: task hung in usbdev_open
2026/03/29 06:03:55 ci-upstream-kasan-gce: VM 4: crash(tail0): SYZFAIL: failed to recv rpc
2026/03/29 06:04:03 ci-upstream-kasan-gce: candidates=36467 corpus=29908 coverage=155387 exec total=128631 (149/sec) pending=0 reproducing=0
2026/03/29 06:04:13 ci-upstream-kasan-gce: candidates=36138 corpus=30239 coverage=155944 exec total=130098 (149/sec) pending=0 reproducing=0
2026/03/29 06:04:23 ci-upstream-kasan-gce: candidates=35795 corpus=30579 coverage=156623 exec total=131596 (149/sec) pending=0 reproducing=0
2026/03/29 06:04:33 ci-upstream-kasan-gce: candidates=35418 corpus=30962 coverage=157303 exec total=133206 (149/sec) pending=0 reproducing=0
2026/03/29 06:04:43 ci-upstream-kasan-gce: candidates=35079 corpus=31291 coverage=157832 exec total=134645 (149/sec) pending=0 reproducing=0
2026/03/29 06:04:46 ci-upstream-kasan-gce: creating instance: ci-upstream-kasan-gce-4
2026/03/29 06:04:53 ci-upstream-kasan-gce: candidates=34732 corpus=31643 coverage=158363 exec total=136139 (149/sec) pending=0 reproducing=0
2026/03/29 06:04:57 ci-upstream-kasan-gce: wait instance to boot: ci-upstream-kasan-gce-4 (10.128.0.235)
2026/03/29 06:05:03 ci-upstream-kasan-gce: candidates=34408 corpus=31971 coverage=158827 exec total=137546 (149/sec) pending=0 reproducing=0
2026/03/29 06:05:13 ci-upstream-kasan-gce: candidates=34115 corpus=32248 coverage=159382 exec total=138805 (148/sec) pending=0 reproducing=0
2026/03/29 06:05:23 ci-upstream-kasan-gce: candidates=33914 corpus=32448 coverage=161137 exec total=139780 (148/sec) pending=0 reproducing=0
2026/03/29 06:05:33 ci-upstream-kasan-gce: candidates=33680 corpus=32686 coverage=162640 exec total=140864 (147/sec) pending=0 reproducing=0
2026/03/29 06:05:43 ci-upstream-kasan-gce: candidates=33436 corpus=32921 coverage=163836 exec total=141977 (147/sec) pending=0 reproducing=0
2026/03/29 06:05:53 ci-upstream-kasan-gce: candidates=33138 corpus=33220 coverage=165941 exec total=143302 (147/sec) pending=0 reproducing=0
2026/03/29 06:06:03 ci-upstream-kasan-gce: candidates=32891 corpus=33450 coverage=166697 exec total=144404 (146/sec) pending=0 reproducing=0
2026/03/29 06:06:13 ci-upstream-kasan-gce: candidates=32663 corpus=33692 coverage=167646 exec total=145470 (146/sec) pending=0 reproducing=0
2026/03/29 06:06:16 ci-upstream-kasan-gce: runner 4 connected
2026/03/29 06:06:16 [FATAL] ci-upstream-kasan-gce: aborting RPC server: mismatching manager/executor git revisions: 356bdfc961dd4d1eeb96ca7d9071058dc92d2dde vs b5ceaad20a1c1f911045a12770d98c0d99300b1b

On the syz-ci side, the crash was noticed, but I see absolutely nothing interesting around that time.

Mar 29 06:06:16 ci docker[63825]: 2026/03/29 06:06:16 [ERROR] ci: ci-upstream-kasan-gce: manager exited unexpectedly: exit status 1

The only somewhat weird part is that, right before this restart of ci-upstream-kasan-gce, syz-ci was rebuilding its kernel image and it was restarted like this

Mar 29 05:22:03 ci docker[63825]: 2026/03/29 05:22:03 ci: ci-upstream-kasan-gce: build successful
Mar 29 05:22:05 ci docker[63825]: 2026/03/29 05:22:05 ci: ci-upstream-kasan-gce: [re]starting manager
Mar 29 05:22:05 ci docker[63825]: 2026/03/29 05:22:05 ci: ci-upstream-kasan-gce-root: building kernel...
Mar 29 05:22:05 ci docker[63825]: 2026/03/29 05:22:05 ci: uploading 2026-03-28_16h.json to gs://syzkaller/bench/ci-upstream-kasan-gce/2026-03-28_16h.json
Mar 29 05:22:05 ci docker[63825]: 2026/03/29 05:22:05 ci: ci-upstream-kasan-gce: stopping manager
Mar 29 05:23:08 ci docker[63825]: 2026/03/29 05:23:08 ci: ci-upstream-kasan-gce: killing manager
Mar 29 05:23:09 ci docker[63825]: 2026/03/29 05:23:09 ci: ci-upstream-kasan-gce: manager exited with signal: killed
< ... >
Mar 29 05:49:39 ci docker[63825]: 2026/03/29 05:49:39 ci: ci-upstream-kasan-gce: started manager

So

• The git revision mismatch didn't happen at the very start of syz-manager, but rather long after.
• The git revision of the syz-executor (b5ceaad) was never than that of syz-manager (356bdfc).

[pimyn-girgis]

A potential flow is:

1. syz-ci crashes
2. syz-ci starts
3. manager and executor binaries are updated
4. an already running manager creates a new VM instance that gets the new executor
5. mismatch and crash

Although, that only explains the cases where the executor is newer than the manager. What about the times where the manager is newer than the executor? Maybe managers are allowed to restart to update and reuse existing VMs? In that case they should be copying new executors, but we've also been getting "failed to copy executor binary" errors, do we crash managers afterwards?

I'll have to check all of this.

The main question is, why did it only start occurring recently? I couldn't see any recent changes to syz-ci or pkg/updater.
As you pointed out offline, it happened around the time we introduced, and later reverted and re-introduced, the spot VMs for GCE. Are these events related? How?

[a-nogikh]

Did you observe syz-ci crashes in the logs?
In the case I've described in #7014 (comment), it appears that syz-ci crash may not (necessarily) be involved here.

[pimyn-girgis]

I see. I may have misunderstood your initial comment then.

[a-nogikh]

It gets more wild:

Instance: ci2-android-6-1-perf

Log line:
aborting RPC server: mismatching manager/executor arches: amd64 vs 386 (full request: &flatrpc.ConnectRequestRawT{Cookie:0x311419435176f08f, Id:2, Arch:"386", GitRevision:"628666c62893a7468bcf1a68fc1129bbce02f74a", SyzRevision:"3a63314988ee4fd767ee4f010ec427b0543fae2a"})

[pimyn-girgis]

Curiously, I don't think we build 386 binaries on ci2. How did this happen?

[a-nogikh]

Hmm, that is true, there are no 386 instances there