🛠 VmWare & Hardware version debug

Jenkins runs a powershell script and it creates a VM. Normally it works, but this morning I got this:

 

Virtual Machine [xxx] does not exist
Importing new Virtual Machine
New-VM : 4/4/2025 9:55:27 AM New-VM  An invalid argument "configSpec.guestId" was specified. 
At C:\...\powershell\commonScripts\Initialize-VM.ps1:247 char:14
+ ...    $newVM = New-VM -Name $VMName -ContentLibraryItem $templateObj -Da ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-VM], ViError
    + FullyQualifiedErrorId : ViCore_VMServiceImpl_DeployFromLibraryItem_ViNetException,VMware.VimAutomation.ViCore.Cm    dlets.Commands.NewVM

The first idea was that something has been changed in the powershell wrapper. But with the terraform I got more a less the same error. It was complaning about unsupported guest ID. In packer we explicitly specify rhel8_64Guest, but just in case I also decided to specify it in terraform https://registry.terraform.io/providers/hashicorp/vsphere/latest/docs/resources/virtual_machine#guest_id-1 and got different errors:

rhel8_64Guest:

│ Error: deploy error: An invalid argument "configSpec.guestId" was specified.
│ 
│   with module.Personal.vsphere_virtual_machine.ld-lgonchar-2,
│   on modules/Personal/ld-lgonchar-2.example.local.tf line 2, in resource "vsphere_virtual_machine" "ld-lgonchar-2":
│    2: resource "vsphere_virtual_machine" "ld-lgonchar-2" {

rockylinux_64Guest:

│ Error: cannot find OS family for guest ID "rockylinux_64Guest": could not find guest ID "rockylinux_64Guest"
│ 
│   with module.Personal.vsphere_virtual_machine.ld-lgonchar-2,
│   on modules/Personal/ld-lgonchar-2.example.local.tf line 2, in resource "vsphere_virtual_machine" "ld-lgonchar-2":
│    2: resource "vsphere_virtual_machine" "ld-lgonchar-2" {

However in the docs https://knowledge.broadcom.com/external/article/321876/determine-the-guest-os-from-a-vm-configu.html both of them exist. The next idea was to epoxrt list of guest IDs:

[VMware.Vim.VirtualMachineGuestOsIdentifier].GetEnumValues()

...
rhel8_64Guest
...
rockylinux_64Guest
almalinux_64Guest

and again... both of them exist. Also the windows VM were created without any problems. However, my workmate spotted that the different hardware version is available for customezation in web UI. I also found it in the doc https://registry.terraform.io/providers/hashicorp/vsphere/latest/docs/resources/virtual_machine#hardware_version-1. So, If I set the hardware_version param to 21 it works because rockylinux_64Guest is introduced.

What happens?
1. We used to have the VM hardware compatibility set to version 19. And the templates we created used to be detected as RHEL8. And compatibility for version 19 supported RHEL 8.
2. Weekly template rebuild happens
3. what's happening right now?
- The template is still defined as RHEL 8
- During deployment VMWare automatically detects it as Rocky Linux 8
- Vmware sets the Guest OS to rockylinux_64Guest.
- rockylinux_64Guest is not compatible with VM version 19
- When the VM is startted it get the error The guest operating system 'rockylinux_64Guest' is not supported.
4. The compat level in packer template was increased to 21 and rebuilt.

Why it happens?

1. The first idea was that VMWare tools was updated(we are installing the latest one). but it's the same
2. The Vcenter version is also the same.
3. The code is also the same.
4. Something else has been updated inside the VM template(we also install all updates) like firmware but i've not find the diff yet. also the kernel https://oraclelinux.pkgs.org/8/ol8-baseos-latest-x86_64/kernel-4.18.0-553.46.1.el8_10.x86_64.rpm.html was suspicious but I found nothing interesting in the release notes
5. i also upgraded rocky 8.6->8.10 at a test VM just to exclude the changes in the /etc/redhat-release

so.. I don't know why.. wip..

#vmware #debug #terraform

🛠 Ansible doesn't support python path with spaces

1️⃣ Jenkins job issue
I have jenkins job to test ansible roles. The job was was moved to the jenkins folder with the space & the error appeared:

<127.0.0.1> EXEC /bin/sh -c 'ANSIBLE_ASYNC_DIR=/home/jenkins_slave/.ansible_async/ /home/jenkins_slave/workspace/Ansible Guild/Snapshot/.venv/bin/python3.12 /home/jenkins_slave/...
failed: [localhost] (item=postgresql-instance) => {
    "ansible_loop_var": "item",
...
    },
    "module_stderr": "/bin/sh: /home/jenkins_slave/workspace/Ansible: No such file or directory\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 127
}

I found that it caused by
1. calling community.docker.docker_container https://github.com/ansible-community/molecule-docker/blob/main/src/molecule_docker/playbooks/destroy.yml which executing
2. the python code https://github.com/ansible-collections/community.docker/blob/main/plugins/module_utils/module_container/module.py which called
3. the python code https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/async_wrapper.py

2️⃣ Reproduce
so it wasn't about the molecule. I decided to reproduce it:

1. Prepare the venv

 mkdir "test dir"
 cd test\ dir/
 export PYTHONPATH=/usr/lib64/python3.6/site-packages
 python3.12 -m venv .venv --system-site-packages
 . .venv/bin/activate
 pip3 install --upgrade pip
 pip3 install -r ../example/requirements.txt 

2. Run debug

ANSIBLE_KEEP_REMOTE_FILES=1 ansible  -vvvv -m stat -c local -i 'asd,' asd
...
python3 ./AnsiballZ_stat.py explode

3. check internals

 echo PLATFORM; uname; echo FOUND; command -v 'python3.12'; command -v 'python3.11'; command -v 'python3.10'; command -v 'python3.9'; command -v 'python3.8'; command -v 'python3.7'; command -v 'python3.6'; command -v '/usr/bin/python3'; command -v '/usr/libexec/platform-python'; command -v 'python2.7'; command -v '/usr/bin/python'; command -v 'python'; echo ENDFOUND
PLATFORM
Linux
FOUND
/home/username/test dir/.venv/bin/python3.12
/home/username/.pyenv/shims/python3.9
/usr/bin/python3.8
/usr/bin/python3.6
/usr/bin/python3
/usr/libexec/platform-python
/home/ms/lgonchar/test dir/.venv/bin/python
ENDFOUND

the module detected symlink and caused the issue.

4️⃣ What is the source?
1. .venv/lib64/python3.12/site-packages/ansible/executor/interpreter_discovery.py:60

try:
...
    shell_bootstrap = "echo PLATFORM; uname; echo FOUND; {0}; echo ENDFOUND".format('; '.join(command_list))

2. .venv/lib64/python3.12/site-packages/ansible/plugins/action/__init__.py

found_interpreters = [interp.strip() for interp in match.groups()[1].splitlines() if interp.startswith('/')]
quoted_interpreters = []
for interpreter in found_interpreters:
    if ' ' in interpreter and not (interpreter.startswith('"') or interpreter.startswith("'")):
        quoted_interpreters.append('"%s"' % interpreter)
    else:
        quoted_interpreters.append(interpreter)
found_interpreters = quoted_interpreters

so the idea was to modify the .venv/lib64/python3.12/site-packages/ansible/plugins/action/__init__.py

if ' ' in interpreter:
    interpreter = shlex.quote(interpreter)
async_cmd = [interpreter, remote_async_module_path, async_jid, async_limit, remote_module_path]

It fixed the python detection but also there are some other places were _low_level_execute_command is called and much work to do.

5️⃣ Final
1. Rename the jenkins folder without space
2. Propose to use the first interpreter without spaces https://github.com/ansible/ansible/issues/82535#issuecomment-2910076878

#ansible #debug #jenkins