Well it seems that the gateway should be already installed and configured by the scripts, so here the full issue ...

• Operating System Version: 6.7.0 Update 3 (Build 14320388)
• Provider (VirtualBox/VMWare): VMWare ESXi
• Terraform Version: 0.12.25 + provider.esxi v1.6.4
• Ansible Version: 2.9.9
• Packer Version: 1.5.6
• Are you using stock boxes (downloaded) or were they built from scratch using Packer? ESXi standard behavior (using packer automatic downloading iso etc)
• Is the issue reproducible or intermittent?
  Dont't know yet. Will try again.

Please verify that you are building from an updated Master branch before filing an issue.

Output of the failing part in ansible is attached.
ansible_output.log

@certrik Thanks for this report. It's really strange, if it's happening in ESXi, I imagine it should be happening everywhere since it's just calling the script. I don't have a spare ESXi to test from (I'm working on getting something for this) but looking through your output, I don't immediately understand why it's throwing that Powershell error

As a wild guess, I think something goes wrong at

 $req = [System.Net.WebRequest]::CreateHttp("https://wef")
    try
    {
        $req.GetResponse()
    }
    catch
    {
        # we don't care about errors here, we just want to get the cert ;)
    }
    $ThumbPrint = $req.ServicePoint.Certificate.GetCertHashString()
    $body = $body -replace "{{THUMBPRINT}}", $ThumbPrint

    Invoke-RestMethod -uri https://localhost/api/management/systemProfiles/center -body $body -Method Post -UseBasicParsing -UseDefaultCredentials -ContentType "application/json"

in Vagrant/scripts/install-microsoft-ata.ps1.
So that $req.GetResponse() doesn't return anything therfore $body is somehow messed up.

This would explain

"stderr_lines": [
        "You cannot call a method on a null-valued expression.",
        "At C:\\vagrant\\scripts\\install-microsoft-ata.ps1:79 char:5",
        "+     $ThumbPrint = $req.ServicePoint.Certificate.GetCertHashString()",
        "+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~",
        "    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException",
        "    + FullyQualifiedErrorId : InvokeMethodOnNull",
        " ",

in the attached logfile.
I am trying now with some wait time (60s) before trying $req = [System.Net.WebRequest]::CreateHttp("https://wef") .

I will post my results.

Didn't change anything. Don't know. Could it be a problem with this variable?

I figured that

If (-not (Test-Path "$env:temp\gatewaysetup.zip"))
    {
        Invoke-WebRequest -uri https://wef/api/management/softwareUpdates/gateways/deploymentPackage -UseBasicParsing -OutFile "$env:temp\gatewaysetup.zip" -Credential (new-object pscredential("wef\vagrant",(convertto-securestring -AsPlainText -Force -String "vagrant")))
        Expand-Archive -Path "$env:temp\gatewaysetup.zip" -DestinationPath "$env:temp\gatewaysetup" -Force
    }
    else
    {
        Write-Host "[$env:computername] Gateway setup already downloaded. Moving On."
    }
    if (-not (Test-Path "C:\Program Files\Microsoft Advanced Threat Analytics"))
    {
        Set-Location "$env:temp\gatewaysetup"
        Start-Process -Wait -FilePath ".\Microsoft ATA Gateway Setup.exe" -ArgumentList "/q NetFrameworkCommandLineArguments=`"/q`" ConsoleAccountName=`"wef\vagrant`" ConsoleAccountPassword=`"vagrant`""
    }
    else
    {
        Write-Host "[$env:computername] ATA Gateway already installed. Moving On."
    }   

is all working. At least kind of. Running
Start-Process -Wait -FilePath ".\Microsoft ATA Gateway Setup.exe" -ArgumentList "/q NetFrameworkCommandLineArguments="/q" ConsoleAccountName="wef\vagrant" ConsoleAccountPassword="vagrant""
just creates an xml file in C:\Program Files\Microsoft Advanced Threat Analytics on the dc.
Now I am wondering if this really could be it. Shouldn't there be more than just an xml file?
The main error, I think, is that it cannot find the running service.
In my opinion something isn't installed correctly but I couldn't find an error.

I'm rebuilding DetectionLab on my ESXi instance at the moment and will let you know if I'm able to reproduce this!

@certrik - I wasn't able to reproduce that exact error, but ATA is definitely not configured correctly after the installation:

I'm going to see if I can figure out why that's the case

Scratch that, I'm totally able to reproduce this, error and everything. Digging into root cause now.

The DC is throwing this error when the WEF host tries to remotely install the lightweight gateway:

The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID 
{D63B10C5-BB46-4990-A94F-E40B9D520160}
 and APPID 
{9CA88EE3-ACB7-47C8-AFC4-AB702511C276}
 to the user WINDOMAIN\vagrant SID (S-1-5-21-3016237423-2057753171-3108155389-1000) from address LocalHost (Using LRPC) running in the application container Unavailable SID (Unavailable). This security permission can be modified using the Component Services administrative tool.

Not sure if it's the root cause yet, but just noticed it was happening

Changing:
Invoke-Command -computername dc -Credential (new-object pscredential("windomain\vagrant",(ConvertTo-SecureString -AsPlainText -Force -String "vagrant"))) -ScriptBlock {

to

Invoke-Command -computername dc -Credential (new-object pscredential("windomain\Administrator",(ConvertTo-SecureString -AsPlainText -Force -String "vagrant"))) -ScriptBlock {

seems to resolve the installation problem, but now the DC is failing to start the gateway due to memory problems. I'm going to bump that and see if it resolves everything.

Alrighty, got this working!

Root cause: Not enough memory on the DC. The gateway service gets stuck in a reboot loop because it keeps hitting OOM errors.

The error message I was seeing here was a red herring.

Simply upping the RAM to DC to 4096mb and re-provisioning the WEF and DC hosts solved the problem completely:

TASK [wef : Installing Microsoft Advanced Threat Analytics] ********************************************************************************************************************************************************************************
task path: /DetectionLab/ESXi/ansible/roles/wef/tasks/main.yml:109
changed: [192.168.3.203] => {"changed": true, "cmd": ".\\install-microsoft-ata.ps1", "delta": "0:04:21.432620", "end": "2020-06-02 05:24:56.243184", "failed_when_result": false, "rc": 0, "start": "2020-06-02 05:20:34.810563", "stderr": "", "stderr_lines": [], "stdout": "Installing Microsoft ATA 1.9\n\r\nHandles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName                                                  \r\n-------  ------    -----      -----     ------     --  -- -----------                                                  \r\n              0        0          0       0.09   1016                                                                  \r\n\r\n[05:23] [DC] Installing the ATA Lightweight gateway on DC...\n\r\n\r\n\r\n", "stdout_lines": ["Installing Microsoft ATA 1.9", "", "Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName                                                  ", "-------  ------    -----      -----     ------     --  -- -----------                                                  ", "              0        0          0       0.09   1016                                                                  ", "", "[05:23] [DC] Installing the ATA Lightweight gateway on DC...", "", "", ""]}

TASK [wef : debug] *************************************************************************************************************************************************************************************************************************
task path: DetectionLab/ESXi/ansible/roles/wef/tasks/main.yml:116
ok: [192.168.3.203] => {
    "msg": [
        "Installing Microsoft ATA 1.9",
        "",
        "Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName                                                  ",
        "-------  ------    -----      -----     ------     --  -- -----------                                                  ",
        "              0        0          0       0.09   1016                                                                  ",
        "",
        "[05:23] [DC] Installing the ATA Lightweight gateway on DC...",
        "",
        "",
        ""
    ]
}

It is a pitty but it still does not work for me. Installation of the Lightweight ATA Gateway still fails on the DC.
I even upped the RAM to 8 and added 2 vCPUs after trying your new configuration unsuccessfully.

@certrik can you take a look at the error logs for ATA on the DC? They're in C:\Program Files\Microsoft Advanced Threat Analytics\Gateway\Logs

Maybe it can provide some insight into why the DC is having issues

I will have a look.

Well ... the only file I can find in C:\Program Files\Microsoft Advanced Threat Analytics\Gateway\ on the DC is DataCollectorSet.xml . There is no folder or any other file.

Can you try:

1. Remove the C:\Program Files\Microsoft Advanced Threat Analytics folder on DC
2. From WEF, run . c:\DetectionLab\Vagrant\scripts\install-microsoft-ata.ps1 from Powershell and let me know what happens after that? There should definitely be more than an xml file on the DC

I get this on the WEF ...

and on the DC the MS ATA folder was created again with DataCollectorSet.xml.

Wanted to let you know that the new script from 2708f4f fails at the ansible task but at least it was possible to install the Gateway manually via this script invoking it from WEF.

I did some further testing and the mentioned script only worked once if executed from the WEF. I never experienced positive behavior in ansible.
The installation process gives error 1603 on the DC. Here is a screenshot from the extendend install log from the DC.

Wanted to let you know that the new script from 2708f4f fails at the ansible task but at least it was possible to install the Gateway manually via this script invoking it from WEF.

If I remember correctly I executed the script from the WEF with administrative privileges (and did it again just now => it worked). Maybe thats the overall problem. I am no powershell specialist but maybe UAC does something fishy.

I think I actually got it working but I had to seperate the installation of the MS ATA Center on the WEF and the installation of the MS ATA Lightweight Gateway on the DC into two ansible tasks with a reboot of the WEF machine in between.
Don't know why it fixed it but at least it worked for the first time on my ESXi setup.
Probably it is also sufficient to just restart IIS before trying to install ATA LGW on the DC. I will try to test this.

Hey @certrik - thanks for continuing to investigate this! I plan to dig into this and try to make the remote command process a little more robust and understand why it doesn't seem to work correctly when used with Ansible. Let me know if the IIS restart helps at all.

Hi @clong I tried a lot of different stuff with no luck. It seems the only thing that is working is a reboot of the WEF between the installation of ATA Center on the WEF and the ATA Lightweight Gateway on the DC.

Hey @certrik - I just pushed an update that fixed this for me. Would you be willing to try it out?

Hi @clong - I have tested it but I am very sorry to report that it didn't work.

Closing due to inactivity and my inability to reproduce this (multiple recent ESXi builds showed ATA connecting without issue after provisioning)