migration_completion_precopy() reports its error through the Error
**errp argument, so its int return value carries no information beyond
success/failure. Convert it to return a bool, matching the common
convention.
Convert its underlying helper qemu_savevm_state_complete_precopy()
likewise, and in turn qemu_savevm_state_non_iterable(), which it calls.
Adjust all callers accordingly.
Refactor and clean migration_completion() code too, which no longer
needs to track int return values.
Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20260706085211.13905-16-avihaih@nvidia.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Switchover ACK is checked only during precopy while the guest is still
running. The last migration_can_switchover() decision and guest stop are
not atomic, so a device may want to request another switchover ACK in
the gap after switchover decision has been made but before the guest is
stopped. Migration would then miss that request, which can increase
downtime.
Cover this case by failing the migration if a switchover-ack was
requested during that time.
Ideally, precopy iterations should be resumed in this case, however,
VFIO doesn't support going back to precopy after being stopped, so
implementing such logic would require non-trivial changes to the guest
start/stop flow. Given the above and that this case should be rare,
failing the migration seems reasonable.
Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Link: https://lore.kernel.org/qemu-devel/20260706085211.13905-9-avihaih@nvidia.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Switchover-ack is a mechanism to synchronize between source and
destination QEMU during migration to prevent the source from switching
over prematurely.
VFIO uses switchover-ack to ensure switchover happens only after
destination side has loaded the precopy initial bytes. This is important
for VFIO, as otherwise downtime could be impacted and be higher.
In its current state, switchover-ack is a one-time mechanism, meaning
that switchover is acked only once and past that another ACK cannot be
requested again. This was sufficient until now, as VFIO precopy initial
bytes was defined to be monotonically decreasing. Thus, when precopy
initial bytes reached zero for all VFIO devices, a single ACK would be
sent and its validity would hold.
However, now the new VFIO_PRECOPY_INFO_REINIT feature allows precopy
initial bytes to be re-initialized during precopy. Specifically, it
means that initial bytes can grow after reaching zero, which would
invalidate a previously sent switchover ACK.
To solve this, make switchover-ack reusable and allow devices to request
switchover ACKs when needed via the save_query_pending SaveVMHandler.
Since now switchover ACK can be requested for a specific device and in
different times, make switchover ACK per-device (instead of a single ACK
for all devices) and let source side do the pending ACKs accounting.
Keep the legacy switchover-ack mechanism for backward compatibility and
turn it on by a compatibility property for older machines. Enable the
property until VFIO implements the new switchover-ack.
Acked-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Link: https://lore.kernel.org/qemu-devel/20260706085211.13905-8-avihaih@nvidia.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
A new switchover-ack mechanism will be added in the following patches.
However, the old mechanism must still be kept for backward
compatibility.
Rename existing code that will be used only for old switchover-ack
mechanism as legacy. This will help to distinguish legacy code from new
code and make it more readable and easier for removal later when no
longer needed.
No functional change intended.
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Link: https://lore.kernel.org/qemu-devel/20260706085211.13905-7-avihaih@nvidia.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
A new switchover-ack mechanism that will replace the existing one will
be added in the following patches. The new mechanism will not use
switchover_ack_needed SaveVMHandler, however, the old mechanism must
still be kept for backward compatibility.
To keep things clear and decrease API surface of old code, replace
switchover_ack_needed SaveVMHandler with a regular function
migration_request_switchover_ack().
No functional changes intended.
Acked-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Link: https://lore.kernel.org/qemu-devel/20260706085211.13905-6-avihaih@nvidia.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Before switchover, the source needs one last exact pending query so
modules can flush dirty state. This is currently done ad hoc in modules
handlers. For example, RAM syncs its dirty bitmap in its save_complete
handler.
This should be a general concept relevant for any module, so extract it
to migration core instead by running a final save_query_pending before
switchover.
The final query requires special handling by modules (e.g., it's called
with BQL locked, during VM stop), so extend save_query_pending
SaveVMHandlers callback and qemu_savevm_query_pending() with a "final"
flag so migration modules can tell the last pending query during
switchover from periodic iteration queries.
Call final pending query also in COLO checkpoint, which needs to flush
dirty state before the checkpoint's live state is saved. Unlike a
regular switchover, COLO reaches completion repeatedly for every
checkpoint, so this must be done on each one.
Signed-off-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20260706085211.13905-4-avihaih@nvidia.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Operations that read guest RAM (dump-guest-memory, memsave, pmemsave)
must refuse to run while the destination of a migration is still
receiving that RAM: during precopy it is incomplete, and during postcopy
a read faults the page in from the source. Provide a single predicate
they can share instead of open-coding the runstate and postcopy checks.
Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20260619101834.228432-2-den@openvz.org>
Replaces the direct accesses to global variable `current_migration`
with `migrate_get_current()` to ensure consistency across systems.
Note: Following this only direct access to `current_migration` will be
* `migrate_get_current()` itself
* `migration_object_init()` initializes `current_migration`
* `migration_shutdown()` to pair up with initialization
* `migration_is_running()`, as there might be a case where this function
is called by a thread before object initialization
Signed-off-by: Aadeshveer Singh <aadeshveer07@gmail.com>
Link: https://lore.kernel.org/r/20260513063513.250911-1-aadeshveer07@gmail.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Commit dd4fe8844b changed the reporting of expected downtime behavior, so
that the value will be calculated on-demand. One side effect on the change
is QEMU will allow the calculation to happen anytime even if there's no
transfer happening for a short while.
PeterM reported an ubsan report from clang when running migration-test with
aarch64 binary on x86_64 hosts. I can also reproduce if I run the test
concurrently so some of the src QEMU may not get chance to push any data,
causing mbps to be 0:
../migration/migration.c:1051:12: runtime error: -nan is outside the range of representable values of type 'long'
Fix it by properly handle both Inf and Nan to return INT64_MAX.
Add a rich comment, per PeterM's suggestion.
Link: https://lore.kernel.org/r/CAFEAcA-MYH6C39xO0OLx4-M5pKurJpurwRsMqZe9q=W-NShAbw@mail.gmail.com
Reported-by: Peter Maydell <peter.maydell@linaro.org>
Fixes: dd4fe8844b ("migration: Calculate expected downtime on demand")
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Link: https://lore.kernel.org/r/20260511182432.1333467-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Marc-André reported an issue on QEMU crash when retrying a cancelled
migration during early setup phase, see "Link:" for more information, and
also easy way to reproduce.
This patch is a replacement of the prior fix proposed by not only switching
to migration_cleanup(), but also fixing it from CPR side, so that we track
hup_source properly to know if src QEMU is waiting or the HUP signal.
To put it simple: this chunk of special casing in migration_cancel() should
not affect normal migration, but only cpr-transfer migration to cover the
small window when the src QEMU is waiting for a HUP signal on cpr
channel (so that src QEMU can continue the migration on the main channel).
To achieve that, we'll also need to remember to detach the hup_source
whenenver invoked: after that point, we should always be able to cleanup
the migration.
It's not a generic operation to explicitly detach a gsource from its
context while in its dispatch() function. But it should be safe, because
gsource disptch() will only happen with a boosted refcount for the
dispatcher so that the gsource will not be freed until the callback
completes. It's also safe to return G_SOURCE_REMOVE after the gsource is
detached, as glib will simply ignore the G_SOURCE_REMOVE.
One can refer to latest 2.86.5 glib code in g_main_dispatch() for that:
https://github.com/GNOME/glib/blob/2.86.5/glib/gmain.c#L3592
When at this, add a bunch of assertions to make sure nothing surprises us.
After this patch applied, the 2nd migration will not crash QEMU, instead
it'll be in CANCELLING until the socket connection times out (it will take
~2min on my Fedora default kernel). During this process no 2nd migration
will be allowed, and after it timed out migration can be restarted.
It's because so far we don't have control over socket_connect_outgoing(),
or anything yet managed by a task executed in qio_task_run_in_thread().
Speeding up the cancellation to be left for future.
I also tested cpr-transfer by only providing cpr channel not the main
channel (with -incoming defer), kickoff migration on source, then cancel it
on source directly without providing the main channel. It keeps working.
I wanted to add an unit test for that but it'll need to refactor current
cpr-transfer tests first; let's leave it for later.
Link: https://lore.kernel.org/r/20260417184742.293061-1-marcandre.lureau@redhat.com
Reported-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Tested-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20260421175820.302795-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Currently, mgmt can only query for remaining RAM using QMP command
"query-migrate" and monitor the "ram" section. There's no way to report
system-wide remaining data including VFIO devices. It was not a problem
before, because for a very long time RAM was the only part that matters.
After VFIO migrations landed upstream, it may not be enough. There can be
GPU devices that contain GBs of device states. Mgmt may want to know how
much remaining for special devices like VFIO, because all of them will be
accounted as VM data to migrate and will contribute to downtime in the
switchover phase.
Add a new "remaining" field in query-migrate results on the top level,
reflecting system-wide remaining data, which will include everything like
VFIO devices.
This information will be useful for mgmt to implement generic way of stall
detection that covers all system resources. For example, when system-wide
remaining data (especially, if sampled right after each migration
iteration) does not decrease anymore for a relatively long period of time,
then it may imply there is a challenge of converging, mgmt can react based
on how this value changes over time.
Before this patch, "expected_downtime" almost played this role. For
example, by monitoring "expected_downtime" at the beginning of each
iteration can in most cases also reflect the progress of migration
system-wide.
Said that, "expected_downtime" was always calculated based on a bandwidth
value that can fluctuate if avail-switchover-bandwidth is not used. This
new "remaining" field will remove that part of uncertainty for mgmt no
matter if avail-switchover-bandwidth is used by the mgmt.
With the new field, HMP "info migrate" now reports this:
(qemu) info migrate
Status: active
Time (ms): total=12080, setup=14, exp_down=300
Remaining: 1.36 GiB <--- this is the new line
RAM info:
Throughput (Mbps): 840.50
Sizes: pagesize=4 KiB, total=4.02 GiB
Transfers: transferred=1.18 GiB, remain=1.36 GiB
Channels: precopy=1.18 GiB, multifd=0 B, postcopy=0 B
Page Types: normal=307923, zero=388148
Page Rates (pps): transfer=25660
Others: dirty_syncs=1
When VFIO is not involved, the value reported in the new field should be
approximately the same as reported in the "remaining" field of the RAM
section. It is only approximately because the system-wide remaining data
is a cached value, which gets frequently updated by migration core. OTOH,
the RAM's remaining data is accurate.
When VFIO is involved, the new value reported should normally be larger,
because it will include the size of VFIO remaining data too.
Cc: Aseef Imran <aimran@redhat.com>
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dave@treblig.org>
Acked-by: Markus Armbruster <armbru@redhat.com> # QAPI schema
Link: https://lore.kernel.org/r/20260421202110.306051-15-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
QEMU will provide an expected downtime for the whole system during
migration, by remembering the total dirty RAM that we synced the last time,
divides the estimated switchover bandwidth.
That was flawed when VFIO is taking into account: consider there is a VFIO
GPU device that contains GBs of data to migrate during stop phase. Those
will not be accounted in this math.
Fix it by updating dirty_bytes_last_sync properly only when we go to the
next iteration, rather than hide this update in the RAM code. Meanwhile,
fetch the total (rather than RAM-only) portion of dirty bytes, so as to
include GPU device states too.
Update the comment of the field to reflect its new meaning.
Now after this change, the expected-downtime to be read from query-migrate
should be very accurate even with VFIO devices involved.
Tested-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20260421202110.306051-13-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
This value does not need to be calculated as frequent. Only calculate it
on demand when query-migrate happened. With that we can remove the
variable in MigrationState.
This paves way for fixing this value to include all modules (not only RAM
but others too).
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20260421202110.306051-12-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Add a helper migration_get_switchover_bw() to return an estimate of
switchover bandwidth. Use it to simplify the current code.
This will be used in later to remove expected_downtime.
When at it, remove two qatomic_read() to shrink the lines because atomic
ops are not needed when it's always the same thread who does the updates.
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20260421202110.306051-11-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
It used to hide in RAM dirty sync path. Now with more modules being able
to slow sync on dirty information, keeping it there may not be good anymore
because it's not RAM's own concept for iterations: all modules should
follow.
More importantly, mgmt may try to query dirty info (to make policy
decisions like adjusting downtime) by listening to iteration count changes
via QMP events. So we must make sure the boost of iterations only happens
_after_ the dirty sync operations with whatever form (RAM's dirty bitmap
sync, or VFIO's different ioctls to fetch latest dirty info from kernel).
Move this to core migration path to manage, together with the event
generation, so that it can be well ordered with the sync operations for all
modules.
This brings a good side effect that we should have an old issue regarding
to cpu_throttle_dirty_sync_timer_tick() which can randomly boost iteration
counts (because it invokes sync ops). Now it won't, which is actually the
right behavior.
Said that, we have code (not only QEMU, but likely mgmt too) assuming the
1st iteration will always shows dirty count to 1. Make it initialized with
1 this time, because we'll miss the dirty sync for setup() on boosting this
counter now.
Reviewed-by: Hyman Huang <yong.huang@smartx.com>
Reviewed-by: Prasad Pandit <pjp@fedoraproject.org>
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20260421202110.306051-10-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Allow modules to report data that can only be migrated after VM is stopped.
When this concept is introduced, we will need to account stopcopy size to
be part of pending_size as before.
However, when there're data only can be migrated in stopcopy phase, it
means the old "pending_size" may not always be able to reach low enough to
kickoff an slow version of query sync.
It used to be almost guaranteed to happen as all prior iterative modules
doesn't have stopcopy only data. VFIO may change that fact by having some
data that must be copied during stop phase.
So we need to make sure QEMU will kickoff a synchronized version of query
pending when all precopy data is migrated. This might be important to VFIO
to keep making progress even if the downtime cannot yet be satisfied.
So far, this patch should introduce no functional change, as no module yet
report stopcopy size.
This paves way for VFIO to properly report its pending data sizes, which
will start to include stop-only data.
Reviewed-by: Avihai Horon <avihaih@nvidia.com>
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20260421202110.306051-8-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
When QEMU queried the estimated version of pending data and thinks it's
ready to converge, it'll send another accurate query to make sure of it.
It is needed to make sure we collect the latest reports and that equation
still holds true.
However we missed one tiny little difference here on "<" v.s. "<=" when
comparing pending_size (A) to threshold_size (B)..
QEMU src only re-query if A<B, but will kickoff switchover if A<=B.
I think it means it is possible to happen if A (as an estimate only so far)
accidentally equals to B, then re-query won't happen and switchover will
proceed without considering new dirtied data.
It turns out it was an accident in my commit 7aaa1fc072 when refactoring
the code around. Fix this by using the same equation in both places.
Fixes: 7aaa1fc072 ("migration: Rewrite the migration complete detect logic")
Cc: qemu-stable@nongnu.org
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20260421202110.306051-3-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
The package_loaded event is not set in case MIG_RP_MSG_PONG does not
arrive on the source from the destination in the return path thread. The
migration thread would then be blocked waiting for package_loaded event
indefinitely in POSTCOPY_DEVICE state. Where as, in such a condition the
source VM can safely resume as the destination has not yet started. The
pong message can get lost in case of a network failure or destination
crash before sending the pong.
This patch removes the package_loaded event and uses rp_sem, instead of
kicking multiple events. The error is detected in case of network
failure or destination crash and rp_sem is set in the out path of the
return path thread. This will kick the migration thread out from a
condition of indefinitely waiting for rp_sem. The migration thread then
fails early and breaks from the migration loop to resume the vm on the
source side.
Fixes: 7b842fe354 ("migration: Introduce POSTCOPY_DEVICE state")
Signed-off-by: Pranav Tyagi <prtyagi@redhat.com>
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Link: https://lore.kernel.org/r/20260423094438.43556-1-prtyagi@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Some tests can cause QEMU to exit(1) too early while the incoming
coroutine has not yielded for a first time yet. This trips ASAN
because resources related to dispatching the incoming process will
still be allocated in the io/channel.c layer without a
straight-forward way for the migration code to clean them up.
As an example of one such issue, the UUID validation happens early
enough that the temporary socket from qio_net_listener_channel_func()
still has an elevated refcount. If it fails, the listener dispatch
code never gets to free the resource:
Direct leak of 400 byte(s) in 1 object(s) allocated from:
#0 0x55e668890a07 in malloc asan_malloc_linux.cpp:68:3
#1 0x7f3c7e2b6648 in g_malloc ../glib/gmem.c:130
#2 0x55e66a8ef05f in object_new_with_type ../qom/object.c:767:15
#3 0x55e66a8ef178 in object_new ../qom/object.c:789:12
#4 0x55e66a93bcc6 in qio_channel_socket_new ../io/channel-socket.c:70:31
#5 0x55e66a93f34f in qio_channel_socket_accept ../io/channel-socket.c:401:12
#6 0x55e66a96752a in qio_net_listener_channel_func ../io/net-listener.c:64:12
#7 0x55e66a94bdac in qio_channel_fd_source_dispatch ../io/channel-watch.c:84:12
#8 0x7f3c7e2adf4b in g_main_dispatch ../glib/gmain.c:3476
#9 0x7f3c7e2adf4b in g_main_context_dispatch_unlocked ../glib/gmain.c:4284
#10 0x7f3c7e2b00c8 in g_main_context_dispatch ../glib/gmain.c:4272
The exit(1) also requires some tests to setup qtest to expect a return
code of 1 from the QEMU process. Although we can check migration
status changes to be fairly certain where the failure happened, there
is always the possibility of QEMU exiting for another reason and the
test passing. This happens frequently with sanitizers enabled, but
also risks masking issues in the regular build.
Stop allowing the incoming migration to exit and instead require the
tests to wait for the FAILED state and end QEMU gracefully with
qtest_quit.
In practice this means setting exit-on-error=false for every incoming
migration, changing MIG_TEST_FAIL_DEST_QUIT_ERR to MIG_TEST_FAIL and
waiting for a change of state where necessary.
With this, the MIG_TEST_FAIL_DEST_QUIT_ERR error result is now unused,
remove it.
The affected tests are:
validate_uuid_error
multifd_tcp_cancel
dirty_limit
precopy_unix_tls_x509_default_host
precopy_tcp_tls_no_hostname
tcp_tls_x509_mismatch_host
dbus_vmstate_missing_src
dbus_vmstate_missing_dst
Also add a comment to QEMU source explaining that the incoming
coroutine might block for a while until it yields as this is the
actual root cause of the issue.
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Prasad Pandit <pjp@fedoraproject.org>
Link: https://lore.kernel.org/qemu-devel/20260311213418.16951-6-farosas@suse.de
[assert that key doesn't already exists]
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Since
121ccedc2b migration: block incoming colo when capability is disabled
x-colo capability needs to be always enabled on the incoming side.
So migration_incoming_colo_enabled() and migrate_colo() are equivalent
with migrate_colo() being easier to reason about since it is always true
during the whole migration.
Use migrate_colo() to initialize the ram cache in the normal migration path.
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
Link: https://lore.kernel.org/qemu-devel/20260302-colo_unit_test_multifd-v11-3-d653fb3b1d80@web.de
Signed-off-by: Fabiano Rosas <farosas@suse.de>
When migration connection is broken, the QEMU and libvirtd(8)
process on the source side receive TCP connection reset
notification. QEMU sets the migration status to FAILED and
proceeds to migration_cleanup(). Meanwhile, Libvirtd(8) sends
a QMP command to migrate_set_capabilities().
The migration_cleanup() and qmp_migrate_set_capabilities()
calls race with each other. When the latter is invoked first,
since the migration is not running (FAILED), migration
capabilities are reset to false, so during migration_cleanup()
the QEMU process crashes with assertion failure.
Introduce a new migration status FAILING and use it as an
interim status when an error occurs. Once migration_cleanup()
is done, it sets the migration status to FAILED. This helps
to avoid the above race condition and ensuing failure.
Interim status FAILING is set wherever the execution moves
towards migration_cleanup():
- postcopy_start()
- migration_thread()
- migration_cleanup()
- multifd_send_setup()
- bg_migration_thread()
- migration_completion()
- migration_detect_error()
- bg_migration_completion()
- multifd_send_error_propagate()
- migration_connect_error_propagate()
The migration status finally moves to FAILED and reports an
appropriate error to the user.
Interim status FAILING is _NOT_ set in the following routines
because they do not follow the migration_cleanup() path to the
FAILED state:
- cpr_exec_cb()
- qemu_savevm_state()
- postcopy_listen_thread()
- process_incoming_migration_co()
- multifd_recv_terminate_threads()
- migration_channel_process_incoming()
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
Link: https://lore.kernel.org/qemu-devel/20260224102547.226087-1-ppandit@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Cleanup bg_migration_thread() function on error handling. First of all,
early_fail is almost only used to say if BQL is taken. Since we already
have separate jumping labels, we don't really need it, hence removed.
Also, since local_err is around, making sure every failure path will set a
proper error string for the failure, then propagate to MigrationState.error.
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Tested-by: Lukas Straub <lukasstraub2@web.de>
Link: https://lore.kernel.org/qemu-devel/20260127185254.3954634-22-peterx@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Split it into two smaller chunks:
- Dump of early_setup VMSDs
- Dump of save_setup() sections
They're mutual exclusive, hence we can run two loops and do them
sequentially. This will cause migration thread to loop one more time, but
it should be fine when migration just started and only do it once. It's
needed because we will need to reuse the early_vmsd helper later to
deduplicate code elsewhere.
QEMU almost sticks with qemu_savevm_state_XXX() to represent the dump of
vmstates's section XXX. With that in mind, this patch renamed the original
qemu_savevm_state_setup() to qemu_savevm_state_do_setup() instead.
So after this patch:
- qemu_savevm_state_non_iterable_early() dumps early_vmsds only,
- qemu_savevm_state_setup() dumps save_setup() sections only,
- qemu_savevm_state_do_setup() does all things needed during setup
phase (including migration SETUP notifies)
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Tested-by: Lukas Straub <lukasstraub2@web.de>
Link: https://lore.kernel.org/qemu-devel/20260127185254.3954634-20-peterx@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Split the function, making itself to be the helper to dump all non-iterable
device states (early_vmsd excluded). Move the precopy end logic out to the
two callers that need it.
With it, we can remove the in_postcopy parameter. Meanwhile, renaming the
function to be qemu_savevm_state_non_iterable(): we don't need the keyword
"complete" because non-iterable doesn't iterate anyway, and we don't need
precopy because we moved precopy specialties out.
NOTE: this patch introduced one new migrate_get_current() user; will be
removed in follow up patch.
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Tested-by: Lukas Straub <lukasstraub2@web.de>
Link: https://lore.kernel.org/qemu-devel/20260127185254.3954634-17-peterx@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
All three events are shared between precopy and postcopy, rather than
precopy specific.
For example, both precopy and postcopy will go through a SETUP process.
Meanwhile, both FAILED and DONE notifiers will be notified for either
precopy or postcopy on completions / failures.
Rename them to make them match what they do, and shorter.
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20260126213614.3815900-6-peterx@redhat.com
[fixed-up entry in scsi-disk.c that got merged first]
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Postcopy (in failure path) should share with precopy on disk reactivations.
Explicit activiation should used to be fine even if called twice, but after
26f65c01ed ("migration: Do not try to start VM if disk activation fails")
we may want to avoid it and always capture failure when reactivation
happens (even if we do not expect the failure to happen). Remove this
redundant call.
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20260126213614.3815900-5-peterx@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Devices may opt-in migration FAILED notifiers to be invoked when migration
fails. Currently, the notifications happen in migration_cleanup(). It is
normally fine, but maybe not ideal if there's dependency of the fallback
v.s. VM starts.
This patch moves the FAILED notification earlier, so that if the failure
happened during switchover, it'll notify before VM restart.
After walking over all existing FAILED notifier users, I got the conclusion
that this should also be a cleaner approach at least from design POV.
We have these notifier users, where the first two do not need to trap
FAILED:
|----------------------------+-------------------------------------+---------------------|
| device | handler | events needed |
|----------------------------+-------------------------------------+---------------------|
| gicv3 | kvm_arm_gicv3_notifier | DONE |
| vfio_iommufd / vfio_legacy | vfio_cpr_reboot_notifier | SETUP |
| cpr-exec | cpr_exec_notifier | FAILED, DONE |
| virtio-net | virtio_net_migration_state_notifier | SETUP, FAILED |
| vfio | vfio_migration_state_notifier | FAILED |
| vdpa | vdpa_net_migration_state_notifier | SETUP, FAILED |
| spice [*] | migration_state_notifier | SETUP, FAILED, DONE |
|----------------------------+-------------------------------------+---------------------|
For cpr-exec, it tries to cleanup some cpr-exec specific fd or env
variables. This should be fine either way, as long as before
migration_cleanup().
For virtio-net, we need to re-plug the primary device back to guest in the
failover mode. Likely benign.
VFIO needs to re-start the device if FAILED. IIUC it should do it before
vm_start(), if the VFIO device can be put into a STOPed state due to
migration, we should logically make it running again before vCPUs run.
VDPA will disable SVQ when migration is FAILED. Likely benign too, but
looks better if we can do it before resuming vCPUs.
For spice, we should rely on "spice_server_migrate_end(false)" to retake
the ownership. Benign, but looks more reasonable if the spice client does
it before VM runs again.
Note that this change may introduce slightly more downtime, if the
migration failed exactly at the switchover phase. But that's very rare,
and even if it happens, none of above expects a long delay, but a short
one, likely will be buried in the total downtime even if failed.
Cc: Cédric Le Goater <clg@redhat.com>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Signed-off-by: Peter Xu <peterx@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20260126213614.3815900-4-peterx@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Migration notifiers will notify at any of three places: (1) SETUP
phase, (2) migration completes, (3) migration fails.
There's actually a special case for spice: one can refer to
b82fc321bf ("Postcopy+spice: Pass spice migration data earlier"). It
doesn't need another 4th event because in commit 9d9babf78d ("migration:
MigrationEvent for notifiers") we merged it together with the DONE event.
The merge makes some sense if we treat "switchover" of postcopy as "DONE",
however that also means for postcopy we'll notify DONE twice.. The other
one at the end of postcopy when migration_cleanup().
In reality, the current code base will also notify FAILED for postcopy
twice. It's because an (maybe accidental) change in commit
4af667f87c ("migration: notifier error checking").
First of all, we still need that notification when switchover as stated in
Dave's commit, however that's only needed for spice. To fix it, introduce
POSTCOPY_START event to differenciate it from DONE. Use that instead in
postcopy_start(). Then spice will need to capture this event too.
Then we remove the extra FAILED notification in postcopy_start().
If one wonder if other DONE users should also monitor POSTCOPY_START
event.. We have two more DONE users:
- kvm_arm_gicv3_notifier
- cpr_exec_notifier
Both of them do not need a notification for POSTCOPY_START, but only when
migration completed. Actually, both of them are used in CPR, which doesn't
support postcopy.
When at this, update the notifier transition graph in the comment, and move
it from migration_add_notifier() to be closer to where the enum is defined.
I didn't attach Fixes: because I am not aware of any real bug on such
double reporting. I'm wildly guessing the 2nd notify might be silently
ignored in many cases. However this is still worth fixing.
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
Cc: Dr. David Alan Gilbert <dave@treblig.org>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/qemu-devel/20260126213614.3815900-3-peterx@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
After cleanups, the qmp_migrate_finish function is now just a call to
migration_connect_outgoing(). Remove qmp_migrate_finish() and rename
the qmp_migrate_finish_cb callback.
This also allows the function's error handling to be removed as it now
receives &local_err like the rest of the callees of qmp_migrate().
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Prasad Pandit <pjp@fedoraproject.org>
Link: https://lore.kernel.org/qemu-devel/20260123141656.6765-25-farosas@suse.de
Signed-off-by: Fabiano Rosas <farosas@suse.de>
When setting a callback on a Glib source and giving it a data pointer,
it's natural to also provide the destructor for the data in question.
Since migrate_hup_add() already needs to clone the MigrationAddress
when setting the qmp_migrate_finish_cb callback, also pass the
qapi_free_MigrationAddress as the GDestroyNotify callback.
With this the address doesn't need to be freed at the callback body,
making the management of that memory slightly simpler.
Cc: Mark Kanda <mark.kanda@oracle.com>
Cc: Ben Chaney <bchaney@akamai.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Prasad Pandit <pjp@fedoraproject.org>
Link: https://lore.kernel.org/qemu-devel/20260123141656.6765-23-farosas@suse.de
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Encapsulate the MigrationChannelList parsing in a new
migrate_channels_parse() located at channel.c.
This also makes the memory management of the MigrationAddress more
uniform. Previously, half the parsing code (uri parsing) would
allocate memory for the address while the other half (channel parsing)
would instead pass the original QAPI object along. After this patch,
the MigrationAddress is always QAPI_CLONEd, so the callers can use
g_autoptr(MigrationAddress) in all cases.
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Prasad Pandit <pjp@fedoraproject.org>
Link: https://lore.kernel.org/qemu-devel/20260123141656.6765-21-farosas@suse.de
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Move the code responsible for the various channels connection into
channel.c. This is all executed before the migration_thread and
process_incoming_migration_co are running, so it helps the reasoning
to have them out of migration.c.
migration_ioc_process_incoming becomes migration_channel_identify
which is more in line with what the function does.
Reviewed-by: Peter Xu <peterx@redhat.com>
Link: https://lore.kernel.org/qemu-devel/20260123141656.6765-19-farosas@suse.de
Signed-off-by: Fabiano Rosas <farosas@suse.de>
