Skip to content
vJailbreak

Troubleshooting vJailbreak

Common issues

vJailbreak is deployed on Kubernetes running on Ubuntu 22.04.5, and distributed as a QCOW2 image. The Kubernetes namespace migration-system contains the vJailbreak UI and migration controller pods. Each VM migration will spawn a migration object. The status field contains a high level view of the progress of the migration of the VM. For more details about the migration, check the logs of the pod specified in the Migration object.

Getting logs

List all pods in the migration namespace

Terminal window
kubectl -n migration-system get pod

Find a specific VM migration pod

Terminal window
kubectl -n migration-system get pod | grep <source VM name>

Get details & events for a v2v-helper pod. This is helpful if a migration is stuck in a pending state, or to track the progress of a migration without the UI.

Terminal window
kubectl -n migration-system describe pod <v2v-helper-pod-name>

Get logs for a specific migration pod. This shows more detail than describe pod.

Terminal window
kubectl logs <pod> -n migration-system

Get logs for the migration-controller-manager

Terminal window
kubectl logs -n migration-system deploy/migration-controller-manager

Turn on Debug Mode

Terminal window
kubectl patch configmap -n migration-system migration-config-<vm-name> --type merge -p '{"data":{"DEBUG":"true"}}'

A migration is stuck in pending

If the migration was set to Retry on Failure, then delete the v2v-helper pod for that VM and collect the logs of the pod that comes up.

Terminal window
kubectl delete pod -n migration-system v2v-helper-<vm-name>

If the v2v-helper pod doesn’t come back up, and you can’t delete the migration in the UI, then delete the associated migrationplan.

  • First, get the migrationplan object name UUID for the associated VMs:
Terminal window
kubectl get migrationplans -n migration-system -o yaml
  • Then delete the migrationplan object, which should remove it from the UI.
Terminal window
kubectl delete migrationplan <UUID> -n migration-system

Get all vJailbreak custom resource definitions (CRDs)

Terminal window
kubectl get migrationplans,migrations,migrationtemplates,networkmappings,openstackcreds,storagemappings,vmwarecreds,secrets -n migration-system -o yaml

virt-v2v fails: rename /sysroot/etc/resolv.conf Operation not permitted

  • Symptom

    virt-v2v or virt-v2v-in-place fails with an error similar to:

    renaming /sysroot/etc/resolv.conf to /sysroot/etc/6vvk9gzd
    guestfsd: error: rename: /sysroot/etc/resolv.conf to /sysroot/etc/6vvk9gzd: Operation not permitted
    commandrvf: stdout=n stderr=n flags=0x0
    commandrvf: umount /sysroot/sys
    virt-v2v-in-place: error: libguestfs error: sh_out: rename: /sysroot/etc/resolv.conf to /sysroot/etc/6vvk9gzd: Operation not permitted

  • Cause

    On some Linux VMs, /etc/resolv.conf is marked immutable. When virt-v2v tries to rename or replace this file inside the guest filesystem during conversion, the immutable attribute prevents the operation and conversion fails.

    You can confirm the immutable bit inside the source VM with:

    Terminal window
    lsattr /etc/resolv.conf
    ----i----------------- /etc/resolv.conf

    The i flag indicates the file is immutable.

  • Resolution

    1. Remove the immutable attribute inside the source VM before migration:

      Terminal window
      chattr -i /etc/resolv.conf

    2. Verify the attribute is gone:

      Terminal window
      lsattr /etc/resolv.conf
      ---------------------- /etc/resolv.conf

    3. Re-run the migration.

  • Notes

    • This is a known and documented virt-v2v issue. See upstream documentation.
    • If configuration management or security hardening marks /etc/resolv.conf immutable, ensure this is unset before conversion, or adjust your automation so VMs intended for conversion do not have /etc/resolv.conf marked immutable.

Disk attach fails during migration: No more available PCI slots

  • Symptom

    During a migration, attaching a target volume to the vJailbreak VM (or an agent VM) fails. The nova-compute log on the OpenStack side shows an error similar to:

    TRACE nova.virt.libvirt.driver [instance: <uuid>] libvirt.libvirtError: internal error: No more available PCI slots

  • Cause

    During conversion, vJailbreak attaches the target Cinder volumes to the vJailbreak VM (or its agent VMs) to copy and convert the disk data. If the vJailbreak image was uploaded without a disk bus setting, OpenStack attaches these volumes using the default virtio-blk bus, where every attached volume is a separate PCI device and consumes its own PCI slot.

    The virtual PCI bus has a limited number of slots, several of which are already used by essential devices (network interfaces, video, memory balloon, and so on). Migrating VMs with many disks — or running many migrations in parallel on one agent — exhausts the available PCI slots, and the volume attach fails with the error above.

  • Resolution

    Configure the vJailbreak image to use the virtio-scsi disk bus. With virtio-scsi, all attached volumes share a single SCSI controller that consumes only one PCI slot and supports up to 256 devices.

    1. Set the following properties on the vJailbreak image before creating the vJailbreak VM:

      Terminal window
      openstack image set \
        --property hw_disk_bus=scsi \
        --property hw_scsi_model=virtio-scsi \
        <vjailbreak-image-name-or-ID>

    2. Deploy the vJailbreak VM from the updated image, then re-run the migration.

  • Notes

    • The disk bus is fixed when the VM is created. If your vJailbreak VM is already deployed, setting the properties on the image is not enough — you must recreate the vJailbreak VM from the updated image.
    • Agent VMs created during scale up use the same image, so set these properties before scaling up agents.
    • See also: Known Limitations.