Quantcast
Channel: Ivanti User Community : All Content - Patch Manager
Viewing all articles
Browse latest Browse all 1121

How to troubleshoot a Patch and Compliance (vulnerability) scan

$
0
0

 

This document illustrates the files, registry, settings and other information necessary to effectively troubleshoot a vulnerability scan.

In addition, this document walks through the steps that a basic Patch and Compliance scan (otherwise known as a vulnerability scan) takes.

 

This article will not describe every single step that the Vulnerability Scanner takes, but those steps where a failure can occur.

 

For the purposes of this document a simple scan is performed by running the following at the client command line:

vulscan /scan=0 /showui

 

This command tells the vulnerability scanner to scan Windows vulnerabilities (type 0) and to show a user interface.

 

The name "LDMS2016" and "LDMS2016_v###" will be seen throughout this document.   This refers to the Core Server name of "LDMS2016" which is the name of the core server that the author had when creating the document.

 

 

Settings

The settings that control how the vulnerability scanner will behave are stored in the Distribution and Patch Settings within the Agent Settings tool.

 

These settings control behaviors such as user input options, Cloud Services Appliances patch options, scanner CPU utilization, etc.

 

These settings are stored in the Ivanti EPM Database in the AgentSettings table and physically on the core server in the

\Program Files\LANDESK\ManagementSuite\ldlogon\AgentBehaviors folder.  The Distribution and Patch Settings are stored in the AgentBehavior_(Corename)_v###.xml file within this folder.

 

AgentBehaviorsXML.jpg

                                                                            Example

 

 

Product Licensing

The categories available to scan and repair are controlled by the Product license that has been purchased and activated on the core server.

 

The following graphic shows the categories available within the Download Updates function within the Patch and Compliance tool for those with a license for all categories.

   DownloadUpdatesCategories.jpg

Click for full size

For Product Licensing support Contact Ivanti supportand select the Product Licensing option.

 

Registry Keys

 

Core

HLKM\Software\LANDESK\ManagementSuite\PatchManagement\WebServiceMaxThreads

This key does not exist by default and should only be created with an understanding of how this key works and the full ramifications of creating this key and changing the default value.  This changes the number of default threads

 

This key is documented here: https://community.landesk.com/docs/DOC-36027#jive_content_id_Increasing_the_Number_of_Web_Process_to_Database_Threads

 

Client

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\landesk\managementsuite\WinClient\Vulscan

 

This registry key contains the following information:

 

NameTypeData
Description
AgentBehaviorREG_SZLDMS2016_v495The agent behavior that vulscan will use when operating
AlternateRebootBehavior

/rebootIfNeeded is called from 3 possible locations during a client configuration.  It is hard for the task-handler version of the caller to know that a one-time-only (client-config-only) reboot override has been specified.  So all installers just call vulscan with the /UseAlternateRebootBehavior.  If vulscan can find the string value of "AlternateRebootBehavior" in the vulscanreg key, it'll act as if the behavior was passed by the command line.

CommandLineREG_SZThe command line that was used to launch vulscan
ComputerIdn.LDMS2016REG_DWORD0x00000006When running in a /showui mode the ComputerIDN is accessed locally from the registry rather than needing a separate GetSystemIDN for the UI through a second web service call to the core.  This value matches the ComputerIDN identifier in the Ivanti Endpoint Manager database.
KLBehaviorREG_SZLDMS2016_v517This refers to the Ivanti Antivirus behavior.  This will exist even if Ivanti Antivirus is not installed on the client.
LastReportedReboot.LDMS2016REG_DWORD0x00000001
trustedfilelistREG_SZLDMS2016_v861Trusted file list used for Ivanti Endpoint Security.  This will be present even if EPS is not installed or trusted file lists are not configured for this client.

Note: The populated "Data' entries are provided as an example.  Yours will differ.

 

The VulscanReboot key should NOT be modified, deleted or created.   This is a volatile registry key used by the vulnerability scanner.  Creating this key manually will create a persistent registry key that does not go away and will cause reboot loops and/or other undesirable behavior.

 

 

Gathering information for Ivanti Support

 

The vulnerability scan log files are located in the C:\ProgramData\LANDESK\log folder.

 

When in doubt just .ZIP up the entire folder and send it.

 

Otherwise, the following logs should be gathered:

 

  • vulscan*.log
  • statusdlg*.log

 

It is very useful to turn on Xtrace with the following enabled from the registry key prior to duplicating the problem and gathering logs:

 

From HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\landesk\managementsuite\LogOptions:

2016-07-18_11-23-22.jpg

How To: Enable XTrace Diagnostic Logging for the Ivanti Core and Clients

 

Tips and Tricks

 

Vulscan can be used as a shortcut to open various folders.  The following should be run on the client command line:

 

Vulscan e - Open the folder where Vulscan resides

Vulscan c - Open the LDClient folder,

Vulscan log - Open the ProgramData\LANDESK\Log folder

 

Issue: Cannot open vulscan logs folder from the command line using "vulscan e"

 

Ivanti Patch and Compliance (vulnerability) Scan Process Flow

 

It is important to note that the following must all be able to take place:  Client contact to core through IIS and several web services.  Core contact to Database Server  Core Contact to client  Correct permissions on core ManagementSuite\Incoming directory and \ManagementSuite\LDLOGON\VulnerabilityData and VulscanResults folders.

 

Note that issues can come and go during a vulscan.  This would indicate intermittent issues.  Most of the time this occurs when the server or database has connectivity issues or are too overwhelmed to respond to requests.

 

Step 1  - Contact the Ivanti Core Server

The vulscan engine attempts to contact the core server by checking the HKLM\SOFTWARE\Intel\LANDESK\LDWM registry key.  The client tries to contact vulcore.asmx through the WSVulnerabilityCore web service.  Thus the client needs to be able to contact the core, IIS needs to be available, the app pool needs to be running, and the database needs to be able to contact the core. 

 

Good Vulscan.log entry

Fri, 15 Jul 2016 11:38:20 Core server name found in HKLM\SOFTWARE\Intel\LANDesk\LDWM: LDMS2016

Fri, 15 Jul 2016 11:38:20 File C:\Program Files (x86)\LANDesk\Shared Files\ProxyHost.exe version within specified

Fri, 15 Jul 2016 11:38:20 Attempting to connect to proxyhost

Fri, 15 Jul 2016 11:38:20 connect to proxy result: 0

Fri, 15 Jul 2016 11:38:20 Using proxyhost to communicate with the core

What could go wrong?

Certificate-Based Authentication - New Secure Client information

 

Ivanti Endpoint Manager Enhanced Security Mode

 

If core has been upgraded and you have copied the .CRT, .KEY and

Client unable to connect to the core server

 

Error: "Host not found. Retrying"

Bad vulscan.log entry:

Fri, 15 Jul 2016 13:50:16 In SendRequest: Action = SOAPAction: "http://tempuri.org/GetHashForFile"

 

Fri, 15 Jul 2016 13:50:16 SendRequest: SOAPAction: "http://tempuri.org/GetHashForFile"

 

Fri, 15 Jul 2016 13:50:16 Action SOAPAction: "http://tempuri.org/GetHashForFile" failed, socket error: 0, SOAPCLIENT_ERROR: 5.  Status code: 503, fault string:

Fri, 15 Jul 2016 13:50:16   Retrying in 0 seconds...

Fri, 15 Jul 2016 13:50:16 Action SOAPAction: "http://tempuri.org/GetHashForFile" failed, socket error: 0, SOAPCLIENT_ERROR: 5.  Status code: 503, fault string:

Fri, 15 Jul 2016 13:50:16   Retrying in 9 seconds...

Fri, 15 Jul 2016 13:50:19 Last status: Retrying in 6 seconds...

The client makes a SOAP request to the core server webservice and gets HTTP error 503 - Service Unavailable

 

Note: The default timeout for Vulscan to connect to the core is 10 minutes.   Connection will fail after this time.


Basic Troubleshooting

    • Does the HKLM\SOFTWARE\Intel\LANDESK\LDWM registry key have the correct core name listed?
    • Can you ping the core server?  Try IP address, netbios name, and FQDN
    • Does the client have connectivity otherwise?
    • Can you browse to http://coreservername/WSVulnerabilityCore/vulcore.asmx from the client browser?
      • Is the LDAppVulnerability application pool running on the core and is is the identity assigned to it correct?
      • Is IIS running on the core?

 

Useful Articles

IIS Troubleshooting and Ivanti Endpoint Manager: 101

How to troubleshoot IIS using Log Parser Studio from Microsoft

 


Core server unable to talk to the database

 

This error shows that something is wrong with the core to database communication or web service to database communication.  This can be a simple connectivity issue, database too busy, IIS/ASP.NET, etc.

 

Error: "Server busy"

 

Bad Vulscan.log entry:

Fri, 15 Jul 2016 13:31:07 In SendRequest: Action = SOAPAction: "http://tempuri.org/ResolveDeviceID"

 

 

Fri, 15 Jul 2016 13:31:07 SendRequest: SOAPAction: "http://tempuri.org/ResolveDeviceID"

 

 

Fri, 15 Jul 2016 13:31:22 Action SOAPAction: "http://tempuri.org/ResolveDeviceID" failed, socket error: 0, SOAPCLIENT_ERROR: 7.  Status code: 500, fault string: Server was unable to process request. ---> A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.) ---> The remote computer refused the network connection

Fri, 15 Jul 2016 13:31:22   Retrying in 5 seconds...

Fri, 15 Jul 2016 13:31:25 Last status: Retrying in 2 seconds...

Fri, 15 Jul 2016 13:31:26 Last status: Retrying in 1 seconds..

The client does a SOAP request to the core web service to resolve it's device ID and gets HTTP Error 500 - Internal Server Error

 

    • Can you browse from the client to http://<coreservername>/WSVulnerabilityCore/Vulcore.asmx?
    • Is the core server overloaded or is the database overloaded causing a lack of a timely response?
    • Do other functions that depend on database connectivity work?  (Inventory Scan, doing a search for computers, running an Ivanti query, etc)
    • Is the APP pool assigned to the right version of .NET (4.0)
    • Is ASP.NET 4.0 bound to IIS?
    • Are the database credentials on core correct?  Check in the Configure Services drop-down in the Ivanti Endpoint Manager console.
    • Is the database server up and running?  (Ping the database server, etc)

 


Useful Articles

Error: "Server Busy" When Running a Vulnerability Scan

Step 2  - Core downloads and applies various agent settings

At this step the core server downloads and applies various agent settings.  If a setting does not apply to the computer the file will be downloaded anyway.  (For example, if you have Endpoint Security in your

 

Good Vulscan.log entry

Fri, 15 Jul 2016 14:38:57 Checking whether to unzip 'C:\ProgramData\vulScan\ClientConnectivityBehavior_Apply.zip'.  Force: false

Fri, 15 Jul 2016 14:38:57 GetFileHash: could not find "C:\ProgramData\vulScan\ClientConnectivityBehavior_Apply.zip"

Fri, 15 Jul 2016 14:38:57 Calling 'PreApplyBehavior' in 'C:\Program Files (x86)\LANDesk\LDClient\ClientConnectivityBehavior_Apply.dll'

Fri, 15 Jul 2016 14:38:57 Client connectivity settings pre-apply dll

Fri, 15 Jul 2016 14:38:57 Allowing to download from the source

Fri, 15 Jul 2016 14:38:57 Downloading trusted certificates

Fri, 15 Jul 2016 14:38:57 In SendRequest: Action = SOAPAction: "http://tempuri.org/GetHashForFile"

 

Fri, 15 Jul 2016 14:38:57 SendRequest: SOAPAction: "http://tempuri.org/GetHashForFile"

 

Fri, 15 Jul 2016 14:38:57 Success

Fri, 15 Jul 2016 14:38:57 Self update: files are up to date.

Fri, 15 Jul 2016 14:38:57 Last status: Done

Fri, 15 Jul 2016 14:38:57 Calling 'ApplyBehavior' in 'C:\Program Files (x86)\LANDesk\LDClient\ClientConnectivityBehavior_Apply.dll'

Fri, 15 Jul 2016 14:38:57 Successfully loaded ClientConnectivityBehavior_apply behaviors from 'C:\ProgramData\vulScan\ClientConnectivityBehavior_LDMS2016_v499.xml'.

The client checks it's file hash for the behavior file and compares it through a SOAP request to the core web service function "GetHashForFile".

It then applies the behavior to the client.

What could go wrong?

Client cannot access the AgentBehaviors folder on the core server

 

The client needs to be able to access the \LDLOGON\Agentbehaviors folder on the core server.  It then downloads the agent behavior .XML files and applies them if they pertain to the computer, otherwise the settings come down, but they are not applied.

 

Error: " 'Applying XXX settings failed"

Bad vulscan entry:

Fri, 15 Jul 2016 15:20:53 Info: Core did not find file AgentBehaviors/RebootBehavior_LDMS2016_v503.xml

Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

Fri, 15 Jul 2016 15:20:53 Info: Core did not find file RebootBehavior_Apply.zip

Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

Fri, 15 Jul 2016 15:20:53 Info: Core did not find file AgentBehaviors/RCBehavior_LDMS2016_v511.xml

Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

Fri, 15 Jul 2016 15:20:53 Info: Core did not find file RCBehavior_Apply.zip

Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

 

Useful Articles

Issue: Vulscan is not applying agent setting changes or is using an incorrect agent setting

Error "Unable to get the setting from core" when running security scan (Vulscan.exe)

Error: "Core could not find a file" when running vulscan on clients

Error: "Failed to apply compliance settings" during vulnerability scan return code 451


 

Step 3 - Vulscan loads and caches local MSI information

In order for the vulnerability scanner to scan MSI information, the vulnerability scanner reads and caches the MSI information from the local computer's registry. This calls the MsiEnumProducts and MsiEnumPatches functions.  This depends on the existence of MSI.DLL in the C:\Windows\System32 directory.

 

Vulscan.log entry:

Fri, 15 Jul 2016 15:20:56   Loading MSI patch information

Fri, 15 Jul 2016 15:20:56   product {7A4192A1-84C4-4E90-A31B-B4847CA8E23A}

Fri, 15 Jul 2016 15:20:56   product {7E8833A1-AF24-4CAE-82DF-CFE14C14B94D}

Fri, 15 Jul 2016 15:20:56   product {2EDC2FA3-1F34-34E5-9085-588C9EFD1CC6}

Fri, 15 Jul 2016 15:20:56   product {E7D4E834-93EB-351F-B8FB-82CDAE623003}

Fri, 15 Jul 2016 15:20:56   product {764384C5-BCA9-307C-9AAC-FD443662686A}

Fri, 15 Jul 2016 15:20:56   product {5FCE6D76-F5DC-37AB-B2B8-22AB8CEDB1D4}

Fri, 15 Jul 2016 15:20:56   product {3D6AD258-61EA-35F5-812C-B7A02152996E}

Fri, 15 Jul 2016 15:20:56   product {45734758-4041-4EA8-8E62-DE661FC3879C}

Fri, 15 Jul 2016 15:20:56   product {23170F69-40C1-2702-0920-000001000000}

Fri, 15 Jul 2016 15:20:56   product {60EC980A-BDA2-4CB6-A427-B07A5498B4CA}

Fri, 15 Jul 2016 15:20:56   product {1F1C2DFC-2D24-3E06-BCB8-725134ADF989}

Fri, 15 Jul 2016 15:20:56   product {4C5EF2FF-EEA0-4314-8693-2AF565F14525}

Fri, 15 Jul 2016 15:20:56 Loaded 12 products and/or patches

 

Step 4 - Client requests vulnerability data information from core

 

  1. Vulnerability Definitions are downloaded from the Ivanti Patch Content servers and stored in the Ivanti Endpoint Manager database connected to the core server.
  2. When a client calls in to scan for particular data, it requests Vulnerability data of a certain type (Windows Vulnerabilities, LANDESK Updates, Custom Definitions, etc) and for the particular OS the client is running.
    1. If the client is close to up to date the client gets the vulnerability data directly from the web service.  If it is not close to up to date it downloads the entire vulnerablity data set from the .XML file(s) mentioned below.
    2. The core server also writes this information to XML files in \Program Files\LANDesk\ManagementSuite\LDLogon\VulnerabilityData
    3. The file that gets written is "type_os-bitlevel_language.timestamp".   So a Windows 7 x64 client requesting Windows Vulnerability Data information would cause the core server to write a file called "0_win7-x64_enu.1315869631.xml" and also a compressed .XMLZ version of the same file.  Only the first requesting client causes the .XML file to be initially written.  Thereafter the other clients will simply receive this .XML file.

      Note: Deleting a definition will cause the entire .XML file to be re-written and all clients will redownload the entire .XML file.

    4. LDZIP.DLL in \Program Files (x86)\LANDesk\ManagementSuite\WSVulnerabilityCore\Bin is responsible for writing the compressed version.
    5. The client then downloads this .XMLZ file, decompresses it and begins parsing it.

     

    Good vulscan.log entry:

    Fri, 15 Jul 2016 15:20:56 -------------------ProcessRules of type 0----------------------

    Fri, 15 Jul 2016 15:20:56 GetData(): agentconfig =

    Fri, 15 Jul 2016 15:20:56 Getting definition data from core LDMS2016

    Fri, 15 Jul 2016 15:20:56 HTTP POST: http://LDMS2016:443/WSVulnerabilityCore/VulCore.asmx

    Fri, 15 Jul 2016 15:20:56 Setting a proxy...

    Fri, 15 Jul 2016 15:20:56 Setting socket timeout to 1000 * 60 * 4

    Fri, 15 Jul 2016 15:20:56 Success

    Fri, 15 Jul 2016 15:20:56 Last status: Done

    Fri, 15 Jul 2016 15:20:56 Parsing information

    Fri, 15 Jul 2016 15:20:56 Decompressing data

     

    What could go wrong?

     

    Error: "0x8db30194" (404) from vulscan

    Error: "Client user does not have administrator rights" when running Vulnerability Scan

    Error: "Failed. Cannot Interpret Data" when running a Security and Compliance scan

     

    Step 5 - Vulnerability scanning results are sent to the core server

    After scanning the results are sent to the core server through http://<corename>:443/WSVulnerabilityCore/vulcore.asmx.  At this point the Web services processes the results and creates a scan result file (in this case ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz) that goes into the \Program Files\LANDESK\ManagementSuite\VulscanResults folder on the core.  This gets processed into the database and will show up in the Security and Compliance information for the client in the inventory.

     

    Good vulscan.log entry

    Mon, 18 Jul 2016 08:37:40 Sending scan results to core LDMS2016

    Mon, 18 Jul 2016 08:37:40 PutResultsAsFile uncompressed length: 4936

    Mon, 18 Jul 2016 08:37:40 compressed length: 914

    Mon, 18 Jul 2016 08:37:40 HTTP POST: http://LDMS2016:443/incomingdata/postcgi.exe?prefix=vulscanresults\&name=ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz

    Mon, 18 Jul 2016 08:37:40 Setting a proxy...

    Mon, 18 Jul 2016 08:37:40 Setting socket timeout to 1000 * 60 * 4

    Mon, 18 Jul 2016 08:37:40 Success

    Mon, 18 Jul 2016 08:37:40 In SendRequest: Action = SOAPAction: "http://tempuri.org/PutResultsByFile"

     

    What can go wrong?

    Failures to send the results can come from some of the following issues:

     

    • Incorrect permissions to the \Program Files\LANDESK\ManagementSuite\IncomingData folder.
    • Incorrect permissions to the \Program Files\LANDESK\ManagementSuite\VulscanResults folder.
    • Missing, corrupted or incorrect version of postcgi.exe in the IncomingData folder.
    • Inability to contact the web service to place results.

     

    Failure in vulscan.log

    Mon, 18 Jul 2016 08:49:37 Sending scan results to core LDMS2016

    Mon, 18 Jul 2016 08:49:37 PutResultsAsFile uncompressed length: 4936

    Mon, 18 Jul 2016 08:49:37 compressed length: 913

    Mon, 18 Jul 2016 08:49:37 HTTP POST: http://LDMS2016:443/incomingdata/postcgi.exe?prefix=vulscanresults\&name=ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz

    Mon, 18 Jul 2016 08:49:37 Setting a proxy...

    Mon, 18 Jul 2016 08:49:37 Setting socket timeout to 1000 * 60 * 4

    Mon, 18 Jul 2016 08:49:37 Failed http://LDMS2016:443/incomingdata/postcgi.exe?prefix=vulscanresults\&name=ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz on server (0), server status: 404.

    Mon, 18 Jul 2016 08:49:37 HTTP Error 404.  Giving up.

    Mon, 18 Jul 2016 08:49:37 Last status: Failed: No response from core

    Mon, 18 Jul 2016 08:49:37 Failed to put vulnerability results to core as file: 8DB301B1

    Mon, 18 Jul 2016 08:49:37 Skipping repair step because scan errors occurred.

    Mon, 18 Jul 2016 08:49:37 ReleaseMutex 'Global\vulscan_scan' succeeded. Code: 0

    Mon, 18 Jul 2016 08:49:37 Failed

    In this case the postcgi.exe was missing in the incomingdata folder.  The service responded with an HTTP 404 error "File or directory not found".

     

    Additional articles:

    Issue: Vulnerability Scans are not updating on the core

    Error: "HTTP Error 403" Vulscan Return Code 433

     

    Step 6 - Vulnerability scanner checks for autofix patches

    The vulnerability scanner then checks with the core server to see if there are any patches that should be auto fixed at this time.  This is done through the http://localhost/wsvulnerabilitycore/vulcore.asmx web service using the GetAllPatches function.  If patches are found that need to be auto fixed one of the following methods is called:

     

    • Getallpatches2 -  GetAutofix Patches for all definitions specified
    • GetAutofixPatchesForGroup - If scanning against a group, get all Autofix definitions for that group.
    • GetPatchesForGroup - Get all patches for a group (remember, you can push a repair job against a group and it will be able to scan and repair in one scan)
    • GetPatchesForVulnerability - Get all auto fix patches for patches manually selected and scanned.

     

    The core then builds a list of the repair logic that vulscan will follow and it gets sent to the client through the web service, the client then writes an .XML file to follow as it repairs patches.   This information is all of the repair logic from the definition.


    Viewing all articles
    Browse latest Browse all 1121

    Trending Articles



    <script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>