?

Log in

No account? Create an account
 
 
10 August 2016 @ 11:11 pm
Remote PowerShell  
We tried to run PowerShell remotely (in order to automate build deployment).
We managed to make it work on developers' machines, but on production server it just refuses to work:
----------------------------------------------------------------
C:\Windows\system32>powershell
Windows PowerShell

PS C:\Windows\system32> Enable-PSRemoting -SkipNetworkProfileCheck -Force
WinRM is already set up to receive requests on this computer.
Set-WSManQuickConfig : <f:WSManFault
xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2"
Machine="localhost"><f:Message><f:ProviderFault provider="Config provider"
path="%systemroot%\system32\WsmSvc.dll"><f:WSManFault
xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2"
Machine="sv7731"><f:Message>Unable to check the status of the firewall.
</f:Message></f:WSManFault></f:ProviderFault></f:Message></f:WSManFault>
At line:65 char:17
+                 Set-WSManQuickConfig -force -SkipNetworkProfileCheck
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : InvalidOperation: (:) [Set-WSManQuickConfig], InvalidOperationException
   + FullyQualifiedErrorId : WsManError,Microsoft.WSMan.Management.SetWSManQuickConfigCommand


PS C:\Windows\system32> Enter-PSSession -ComputerName localhost
Enter-PSSession : Connecting to remote server localhost failed with the
following error message : Access is denied. For more information, see the
about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   + CategoryInfo          : InvalidArgument: (localhost:String) [Enter-PSSes
  sion], PSRemotingTransportException
   + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

----------------------------------------------------------------

Note "Unable to check the status of the firewall." part of the message.
Why would Enable-PSRemoting command try to check the status of the firewall?

Another Windows WTF reported by yatur


Update
Finally we were able to fix this remote powershell issue.
The problem was in Group Policy for IpV4Filter on our production machine.
IpV4Filter was limited to a single IP address (main address of that production machine).
I have no idea why it was setup that way.

This is how I fixed WinRM localhost access problem:
Run gpedit.msc
Local Computer Policy
Computer Configuration
Administrative Templates
Windows Components
Windows Remote Management (WinRM)
WinRM Service
Allow remote server management through WinRM

In "IPv4 filter:" change "208.43.198.72" to "*":
IPv4 filter: *

FixIpV4FilterInGroupPolicy.jpg

In the end, PowerShell and Microsoft server tools leave a negative impression due to bugs and pathetic diagnostic.

Consider another PowerShell surprise:
"ls" and "dir" commands produce empty output in case when folder is empty. No headers, no message that says there are no files. Just nothing. WTF?
 
 
 
СБsab123 on August 11th, 2016 06:32 pm (UTC)
Is the machine domain-joined? If not, try to enable the settings for the HTTP protocol on the client side:

Set-Item -Force WSMan:\localhost\Client\AllowUnencrypted true
Set-Item -Force WSMan:\localhost\Client\TrustedHosts *

In the firewall on the server, check the ports 5985 (WinRm HTTP) and maybe also 5986 (WinRM HTTPS).

Edited at 2016-08-11 06:32 pm (UTC)
Dennis Gorelikdennisgorelik on August 11th, 2016 07:31 pm (UTC)
1) Production server is NOT part of a domain.

2) Creating "Windows Remote Management (HTTP-In)" firewall rule helped to fix "Enable-PSRemoting" command:
---
PS C:\Windows\system32> Enable-PSRemoting -Force
WinRM is already set up to receive requests on this computer.
WinRM is already set up for remote management on this computer.
---
It is still a mystery why Enable-PSRemoting command cares about "Windows Remote Management (HTTP-In)" firewall rule.
And if Enable-PSRemoting checks for that firewall rule, why Enable-PSRemoting cannot create that rule in case if that rule is missing.
Or, at least, why Enable-PSRemoting cannot explicitly complain about missing "Windows Remote Management (HTTP-In)" firewall rule.
Enable-PSRemoting is able to enable disabled "Windows Remote Management (HTTP-In)" firewall rule (but not create it).

3) I successfully ran your commands:
------------------------------------
PS C:\Windows\system32> Set-Item -Force WSMan:\localhost\Client\AllowUnencrypted true
PS C:\Windows\system32> Set-Item -Force WSMan:\localhost\Client\TrustedHosts *
------------------------------------

That did not help running "Enter-PSSession" command:
============================
PS C:\Windows\system32> Enter-PSSession -ComputerName localhost
Enter-PSSession : Connecting to remote server localhost failed with the
following error message : The client cannot connect to the destination
specified in the request. Verify that the service on the destination is
running and is accepting requests. Consult the logs and documentation for the
WS-Management service running on the destination, most commonly IIS or WinRM.
If the destination is the WinRM service, run the following command on the
destination to analyze and configure the WinRM service: "winrm quickconfig".
For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName localhost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (localhost:String) [Enter-PSSes
   sion], PSRemotingTransportException
    + FullyQualifiedErrorId : CreateRemoteRunspaceFailed
============================


Edited at 2016-08-11 07:51 pm (UTC)
СБsab123 on August 11th, 2016 08:08 pm (UTC)
2) No idea. The other command that can be used is "winrm quickconfig", and I think it creates the firewall rule among other things. BTW, Server 2016 should be pre-configured out of the box.

3) If it's not domain-joined, you have to use the option -Credential. Just give it the user name (like "~\administrator") and it will prompt you in the GUI for the password. Or generate the credential object from the name and password:

function mkcr
{
<#
.SYNOPSIS
Make a credentials object out of the user name and password.
#>
    param(
        ## User name.
        [string] $user,
        ## Password in plain text.
        [string] $passwd
    )
    $secpwd = ConvertTo-SecureString -Force -AsPlainText $passwd
    New-Object System.Management.Automation.PSCredential @($user, $secpwd)
}


then use -Credential (mkcr ".\administrator" "password")
For some weird reason the GUI form doesn't allow the syntax with "." for the host part in the username but does allow "~". If you don't use the host part on a non-domain-joined server, your connection will be very slow.

Setting up an HTTPS connection is a whole separate world of pain :-)
Dennis Gorelikdennisgorelik on August 11th, 2016 08:37 pm (UTC)
Adding -Credential parameter prompted me for username/password in the GUI (like you said).
I entered password, but it did not change anything - it still returned the same generic error:
"Enter-PSSession : Connecting to remote server localhost failed with the following error message : The client cannot connect to the destination specified in the request. ....."

I actually do NOT need -Credential parameter in order to Enter-PSSession on another Windows computer (which is also "not domain-joined").

Should I try reinstalling PowerShell?
СБsab123 on August 11th, 2016 09:22 pm (UTC)
I've tried, and yes, the localhost connection works without a password even without a domain. No idea what is the problem. Have you tried reading about_Remote_Troubleshooting? :-)
Dennis Gorelikdennisgorelik on August 11th, 2016 10:09 pm (UTC)
It looks like about_Remote_Troubleshooting covers other remote access issues.

They do not cover such obvious problem as an inability to run "Enter-PSSession -ComputerName localhost".

My guess is that some components are broken, but WinRS is unwilling to diagnose that problem.
So the best course of action is to either reinstall PowerShell or just move to another server with clean Windows install.

Windows remoting sucks.
СБsab123 on August 11th, 2016 10:28 pm (UTC)
It's the common Windows problem: the diagnostics tends to be limited to returning an error code that says "you've done something wrong", and even a lot of the error codes are unpublished externally.

Just in case, did you reboot the machine? If not, you can try restarting the WinRM service, i.e. "restart-service WinRM". Or rebooting.

Have you tried running "winrm quickconfig"? If not, try it. You can also test the connection by "winrm identify" (run it with /? to see the help).

Edited at 2016-08-11 10:33 pm (UTC)
Dennis Gorelikdennisgorelik on August 11th, 2016 10:37 pm (UTC)
1) I did not reboot the server yet.
But restarted WinRM:
restart-service WinRM
Restarting WinRM did not help.

Do you think full reboot still may help?

Usually we reboot our server once per month (when we install windows updates).


2) Do you know why Windows tools are so poor at diagnosing problems? Is it intentional?
СБsab123 on August 11th, 2016 10:59 pm (UTC)
1) Don't know, maybe. I think a reboot was required on the older versions. You can try connecting telnet to the port 5985, to see if the winrm server is running at all and the firewall is open.

BTW, try this:

PS C:\> dir WSMan:\localhost\Listener\
Type            Keys                                Name
----            ----                                ----
Container       {Transport=HTTP, Address=*}         Listener_1084132640


Does your result look anything like this?

Look in WSMan:\localhost\Service\ , does it look reasonable? In particular, what does WSMan:\localhost\Service\Auth\ contain for the authentication "negotiate"? What about WSMan:\localhost\Client?

Here is an example of what works for me:

PS C:\> dir WSMan:\localhost\Client
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   NetworkDelayms                                 5000
System.String   URLPrefix                                      wsman
System.String   AllowUnencrypted                               true
Container       Auth
Container       DefaultPorts
System.String   TrustedHosts                                   *


PS C:\> dir WSMan:\localhost\Client\Auth\
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          true
System.String   Digest                                         true
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    true
System.String   CredSSP                                        true

PS C:\> dir WSMan:\localhost\Service\Auth\
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          false
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    false
System.String   CredSSP                                        true
System.String   CbtHardeningLevel                              Relaxed


PS C:\> dir WSMan:\localhost\Service\
Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   RootSDDL                                       O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)...
System.String   MaxConcurrentOperations                        4294967295
System.String   MaxConcurrentOperationsPerUser                 1500
System.String   EnumerationTimeoutms                           240000
System.String   MaxConnections                                 300
System.String   MaxPacketRetrievalTimeSeconds                  120
System.String   AllowUnencrypted                               false
Container       Auth
Container       DefaultPorts
System.String   IPv4Filter                                     *
System.String   IPv6Filter                                     *
System.String   EnableCompatibilityHttpList...                 false
System.String   EnableCompatibilityHttpsLis...                 false
System.String   CertificateThumbprint
System.String   AllowRemoteAccess                              true


(this machine also has CredSSP enabled, you don't really need that for the basic connections).

2) Looks like just nobody cares, it's not considered an important property and requires extra effort. The developers tend to create the one-off solutions for the problems that reach them, and these get occasionally published, so you can find a whole lot of the workaround recipes on the internet. Everything is built as workarounds on top of workarounds. So try going through the support channel, maybe you'll get a solution.
Dennis Gorelikdennisgorelik on August 11th, 2016 11:33 pm (UTC)
Configuration on our production server looks very similar to your configuration.
I noticed couple of differences:
1) CredSSP is false for me and true for you. But you expected this.
2) IPv4Filter for you is "*", but for my server is "GPO 208.43.198.72"
Could that be the culprit?
On my local machine (where remote PowerShell works), I, like you, have
"IPv4Filter *"

How do I change IPv4Filter?

PS C:\> dir WSMan:\localhost\Listener\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Listener

Type            Keys                                Name
----            ----                                ----
Container       {Transport=HTTPS, Address=*}        Listener_1305953032
Container       {Transport=HTTP, Address=*}         Listener_1084132640


PS C:\> dir WSMan:\localhost\Client


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Client

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   NetworkDelayms                                 5000
System.String   URLPrefix                                      wsman
System.String   AllowUnencrypted                               true
Container       Auth
Container       DefaultPorts
System.String   TrustedHosts                                   *


PS C:\> dir WSMan:\localhost\Client\Auth\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Client\Auth

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          true
System.String   Digest                                         true
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    true
System.String   CredSSP                                        false


PS C:\> dir WSMan:\localhost\Service\Auth\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Service\Auth

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   Basic                                          false
System.String   Kerberos                                       true
System.String   Negotiate                                      true
System.String   Certificate                                    false
System.String   CredSSP                                        false
System.String   CbtHardeningLevel                              Relaxed

PS C:\> dir WSMan:\localhost\Service\


   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Service

Type            Name                           SourceOfValue   Value
----            ----                           -------------   -----
System.String   RootSDDL                                       O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
System.String   MaxConcurrentOperations                        4294967295
System.String   MaxConcurrentOperationsPerUser                 1500
System.String   EnumerationTimeoutms                           240000
System.String   MaxConnections                                 300
System.String   MaxPacketRetrievalTimeSeconds                  120
System.String   AllowUnencrypted                               false
Container       Auth
Container       DefaultPorts
System.String   IPv4Filter                     GPO             208.43.198.72
System.String   IPv6Filter                     GPO
System.String   EnableCompatibilityHttpList...                 false
System.String   EnableCompatibilityHttpsLis...                 false
System.String   CertificateThumbprint
System.String   AllowRemoteAccess              GPO             true
СБsab123 on August 11th, 2016 11:42 pm (UTC)
Try Set-Item, like in the example above. I've actually never seen the filter set like this. "GPO" probably means the Group Policies, so you might have some weird settings there.
Dennis Gorelikdennisgorelik on August 11th, 2016 11:51 pm (UTC)
PS C:\> Set-Item -Force WSMan:\localhost\Service\IPv4Filter *
Set-Item : The config setting IPv4Filter cannot be changed because is controlled by policies. The policy would need to be set to "Not Configured" in order to change the config setting.
At line:1 char:1
+ Set-Item -Force WSMan:\localhost\Service\IPv4Filter *
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Set-Item], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.PowerShell.Commands.SetItemCommand


So yes - some group policy is responsible for that.
I just need to find out how to find and change that Group Policy...
СБsab123 on August 12th, 2016 12:49 am (UTC)
Looks like the group policies are stored in the registry under HKEY_LOCAL_MACHINE, so you can just search there for this particular address (and maybe change it right there too). Since the machine is not domain-joined, it shouldn't be downloading these policies from the domain.
Dennis Gorelikdennisgorelik on August 12th, 2016 04:49 am (UTC)
Finally fixed - IpV4Filter Global Policy configuration
Thank you - changing IpV4Filter to "*" helped.
I updated this post.
---
Run gpedit.msc
Local Computer Policy
Computer Configuration
Administrative Templates
Windows Components
Windows Remote Management (WinRM)
WinRM Service
Allow remote server management through WinRM

In "IPv4 filter:" change "208.43.198.72" to "*":
IPv4 filter: *
---
СБsab123 on August 12th, 2016 06:09 pm (UTC)
Re: Finally fixed - IpV4Filter Global Policy configuration
By the way, completely accidentally, today I've stumbled on a post on how to enable the logging on WSman:

https://blogs.msdn.microsoft.com/wmi/2010/03/16/collecting-winrm-traces/
Dennis Gorelikdennisgorelik on August 12th, 2016 10:43 pm (UTC)
Re: Finally fixed - IpV4Filter Global Policy configuration
So:
Computer Management
System Tools
Even Viewer
Applications and Services Logs
Microsoft
Windows Remote Management
Menu - View - Show Analytics and Debug Logs
Right-click on Analytic - Enable log.
Right-click on Debug - Enable log.

Then I sent remote powershell command from my dev machine to production:
Invoke-Command -ComputerName 208.43.198.72 -Script { cd \; ls }

"Operational" section got multiple events ("None", "WSMan API call", "Response handling", "Request handling").
"Analytic" section got lots of events ("Response handling, "None", "WSMan API call").
"Debug" section got no events at all.

When I re-executed powershell commands, "Operational" section got more entries, but "Analytic" section got no new entries.
"Debug" section stays empty.

I did not find any single error message in these logs.

These logs could have helped with diagnostic, but they are clearly not reliable.

Were you able to find something useful in these Windows Remote Management logs?
СБsab123 on August 12th, 2016 11:38 pm (UTC)
Re: Finally fixed - IpV4Filter Global Policy configuration
Never looked at them. :-)

"Operational" should be like the normal syslog level. "Analytical" and "Debug" get enabled by increasing the level of collected events, I didn't look too deeply into that yet. I guess the script had enabled only up to the analytical level.

If I remember right, the level numbering goes approximately like this:

0-5 - operational ("Error" to "Verbose")
up to 16 - analytical
up to 255 - debug
Dennis Gorelikdennisgorelik on August 12th, 2016 11:44 pm (UTC)
Re: Finally fixed - IpV4Filter Global Policy configuration
Are you using remote PowerShell frequently or it's only an occasional exercise for you?
СБsab123 on August 13th, 2016 01:33 am (UTC)
Re: Finally fixed - IpV4Filter Global Policy configuration
All the time.
Dennis Gorelikdennisgorelik on August 11th, 2016 10:43 pm (UTC)
winrm identify
PS C:\Windows\system32> winrm identify
IdentifyResponse
    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
    ProductVendor = Microsoft Corporation
    ProductVersion = OS: 6.3.9600 SP: 0.0 Stack: 3.0
    SecurityProfiles
        SecurityProfileName = http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos, http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/spnego-kerberos


Does it tell you anything?
СБsab123 on August 11th, 2016 11:23 pm (UTC)
Re: winrm identify
Looks like the server is working and can be connected to. Try also "winrm identify -r:localhost"
Dennis Gorelikdennisgorelik on August 11th, 2016 11:35 pm (UTC)
Re: winrm identify
PS C:\> winrm identify -r:localhost
WSManFault
    Message = The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation
 for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure th
e WinRM service: "winrm quickconfig".

Error number:  -2144108526 0x80338012
The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Ma
nagement service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM servic
e: "winrm quickconfig".

So, pretty much the same error message, but, for some reason, repeated twice.
Dennis Gorelikdennisgorelik on August 11th, 2016 10:44 pm (UTC)
PS C:\Windows\system32> winrm quickconfig
WinRM service is already running on this machine.
WinRM is already set up for remote management on this computer.