Unable to launch a vm with a template which is direct download template and having nfs metalink

[kiranchavala]

problem

Unable to launch a vm with a template which is direct download template and having nfs metalink

versions

ACS 4.22.1

The steps to reproduce the bug

1. Create a metalink file

Example

<?xml version="1.0" encoding="UTF-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/">
  <files>
    <file name="ubuntu24-cloud-image-root-password-with-tools.qcow2">
      <size>10485760</size>
      <resources>
        <url type="nfs" preference="100">nfs://10.0.35.201/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2</url>
      </resources>
    </file>
  </files>
</metalink>

2. Expose it via http server

python3 -m http.server 8081 --bind 0.0.0.0

3. Register it a direct download template in cloudstack

4. Launch a vm

5. Exception in agent.log

2026-08-19 04:36:53,965 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Request:Seq 1-6884877930342650195:  { Cmd , MgmtId: 32985734710148, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.agent.directdownload.MetalinkDirectDownloadCommand":{"url":"http://10.0.35.201:8081/good-nfs.metalink","templateId":"202","destData":{"origUrl":"http://10.0.35.201:8081/good-nfs.metalink","uuid":"e50a9618-71f4-4b68-881a-54a2527d3e02","id":"202","format":"QCOW2","accountId":"2","hvm":"true","displayText":"test-nfs-metalink","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":"false"}},"name":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","size":"(6580590.5000 TB) 7235435666033759572","hypervisorType":"KVM","bootable":"false","uniqueName":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","directDownload":"true","deployAsIs":"false","followRedirects":"false"},"destPool":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":"false"},"headers":{},"connectTimeout":"5000","soTimeout":"5000","templateSize":"(6580590.5000 TB) 7235435666033759572","format":"QCOW2","followRedirects":"false","wait":"10800","bypassHostMaintenance":"false"}}] }
2026-08-19 04:36:53,965 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Processing command: org.apache.cloudstack.agent.directdownload.MetalinkDirectDownloadCommand

2026-08-19 04:36:53,965 DEBUG [storage.resource.StorageSubsystemCommandHandlerBase] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command MetalinkDirectDownloadCommand: [{"url":"http://10.0.35.201:8081/good-nfs.metalink","templateId":202,"destData":{"origUrl":"http://10.0.35.201:8081/good-nfs.metalink","uuid":"e50a9618-71f4-4b68-881a-54a2527d3e02","id":202,"format":"QCOW2","accountId":2,"hvm":true,"displayText":"test-nfs-metalink","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":false}},"name":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","size":7235435666033759572,"hypervisorType":"KVM","bootable":false,"uniqueName":"202-2-7772abb0-fd86-32ff-add7-ef855cd70293","directDownload":true,"deployAsIs":false,"followRedirects":false},"destPool":{"uuid":"24518d40-97cd-3433-bafa-dcb0291c03f9","name":"ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-12221-k-Mol8-kiran-chavala/ref-trl-12221-k-Mol8-kiran-chavala-kvm-pri1/?ROLE=Primary&STOREUUID=24518d40-97cd-3433-bafa-dcb0291c03f9","isManaged":false},"headers":{},"connectTimeout":5000,"soTimeout":5000,"templateSize":7235435666033759572,"format":"QCOW2","followRedirects":false,"wait":10800,"bypassHostMaintenance":false}].

2026-08-19 04:36:53,965 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Verifying temporary location for downloading the template exists on the host
2026-08-19 04:36:53,967 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Checking for free space on the host for downloading the template with physical size: 413 and virtual size: 7235435666033759572
2026-08-19 04:36:53,967 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c df --output=avail /var/lib/libvirt/images -B 1 | tail -1 ].
2026-08-19 04:36:53,973 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Successfully executed process [68470] for command [/bin/bash -c df --output=avail /var/lib/libvirt/images -B 1 | tail -1 ].
2026-08-19 04:36:53,973 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Trying to fetch storage pool 24518d40-97cd-3433-bafa-dcb0291c03f9 from libvirt
2026-08-19 04:36:53,974 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Looking for libvirtd connection at: qemu:///system
2026-08-19 04:36:53,978 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Successfully refreshed pool 24518d40-97cd-3433-bafa-dcb0291c03f9 Capacity: (3.4170 TB) 3757022642176 Used: (2.7297 TB) 3001299238912 Available: (703.82 GB) 755723403264
2026-08-19 04:36:53,987 ERROR [cloud.utils.UriUtils] (AgentRequest-Handler-3:[]) (logid:fe7c1979) no hash tag in XML response...
2026-08-19 04:36:53,987 INFO  [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Metalink downloader created, metalink URL: http://10.0.35.201:8081/good-nfs.metalink parsed - 1 urls and 0 checksums found
2026-08-19 04:36:53,987 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Trying to download template
2026-08-19 04:36:53,987 INFO  [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Trying to download Template from URL: nfs://10.0.35.201/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2
2026-08-19 04:36:53,987 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ].
2026-08-19 04:36:53,999 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Execution of process [68473] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] failed.
2026-08-19 04:36:53,999 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Exit value of process [68473] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] is [32].
2026-08-19 04:36:53,999 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Process [68473] for command [/bin/bash -c mount -t nfs 10.0.35.201:/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] encountered the error: [mount.nfs: mount point /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 does not exist].
2026-08-19 04:36:53,999 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ].
2026-08-19 04:36:54,005 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Execution of process [68475] for command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ] failed.
2026-08-19 04:36:54,006 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Exit value of process [68475] for command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ] is [1].
2026-08-19 04:36:54,006 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Process [68475] for command [/bin/bash -c cp /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2 null/null/template/n/null/96fa2884-a398-48f9-acb6-7d00e894520f.qcow2 ] encountered the error: [cp: cannot stat '/mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2': No such file or directory].
2026-08-19 04:36:54,006 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Executing command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ].
2026-08-19 04:36:54,011 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Execution of process [68476] for command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] failed.
2026-08-19 04:36:54,011 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Exit value of process [68476] for command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] is [32].
2026-08-19 04:36:54,011 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Process [68476] for command [/bin/bash -c umount /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04 ] encountered the error: [umount: /mnt/e460cef8-0a57-4080-878f-ddcb1abf4f04: no mount point specified.].
2026-08-19 04:36:54,011 INFO  [direct.download.MetalinkDirectTemplateDownloader] (AgentRequest-Handler-3:[]) (logid:fe7c1979) Successfully downloaded Template from URL: nfs://10.0.35.201/export/templates/ubuntu24-cloud-image-root-password-with-tools.qcow2

What to do about it?

Should be able to successfully launch a vm from metalink url which has a nfs file

https://docs.cloudstack.apache.org/en/4.22.1.0/adminguide/templates/_bypass-secondary-storage-kvm.html