VM network issue in VMM 2012R2 with IPAM integration

This week two colleagues (“Pitchou” and “Loulou”) and me stumbled on an VM Network issue in VMM 2012R2 with IPAM integration. All VM Subnets associated with IP Pools indicated “VM Subnet has been marked for deletion by the network administrator“.

We realized that when we wanted to live migrate a virtual machine. When we were on rating host’s screen on live-migration wizard, we had “port not available” error. Moreover, on Virtual Machine settings, we were not able to modify the VM subnet as below screen. The drop-down menu was grey out.

• Before show you the resolution, I would like that you understand the action timeline on VMM. To understand the rest of this topic I will follow this naming convention:
• IPAM server: VMIPM01
• VMM server 1: VMVMM01
• VMM server 2: VMVMM02
• VMM resource cluster name: CLVMM01

Action timeline on VMM

The first action was realized two weeks ago. I have installed an IPAM server (VMIPM01) and I have integrated it to Virtual Machine Manager 2012R2 on VMVMM01. At this time, all worked perfectly. IPAM collected VMM information as VM Subnet, VM Network or IP Pool. In IPAM all objects (VM subnet, network etc.) were marked to belong to VMVMM01. When I say to belong, I talk about the service instance field of each object.

The second action was realized by “Pitchou” last week. This action was the implementation of Failover Cluster for VMM. This Technet article was followed to implement this solution. So he has installed VMVMM02 and after many actions, VMM was reachable from CLVMM01 connector. And this is the drama… Let me explain.

VM Network issues and resolution

After that each schedule task of IPAM has run, every VM subnets associated with an IP Pool were in error (VM Subnet has been marked for deletion by the network administrator). At this time it was impossible to live-migrate a VM or create one, on the impacted VM networks.

After 2 days of searching we have found a field called NetworkEntityAccess. If you run Get-SCVMSubnet command in a powershell you will see this setting. The right value for this setting is VMMManaged. In our case this setting indicated ExternallyManagedMarkedForDeletion.

/!\ I assume that the below resolution is not approved and not supported by Microsoft …

So to resolve this issue “Loulou” has opened the SQL Management Studio to edit the VMM database. We have found the NetworkEntityAccess field in tbl_NetMan_VMSubnet table.

As you can see on the above screen, the NetworkEntityAccess of the impacted VM subnet is set to 2. This value means ExternallyManagerMarkedForDeletion. So we stop the VMM service and we modify this value to 0.

After that we restart the service. We come back to the VM network configuration to view the related VM subnet state and tada:

So we go verify in VM the settings:

So after many tests all seem alright.

My understanding of the problem

After resolving the issue, we tried to remove the IPAM connector and to re-add it. The result was spectacular: all VM Subnets were marked again for deletion …

So before Failover Cluster integration, IPAM and VMM worked perfectly and troubles came when Failover Cluster were implemented. All objects in IPAM were associated on VMVMM01 because IPAM was installed before cluster installation. After Failover Cluster implementation, IPAM is associated to CLVMM01. I think that IPAM tries to delete VM networks configuration associated to VMVMM01 to re-create and associate them to CLVMM01. But the VM Networks configuration can’t be deleted because there are a lot of VMM dependencies. I’m not sure because the behavior of that is pretty weird.