Vista - Vista computer periodically refuses connections from non Vista machines

I am the administrator for a small network of ten computers. Two are running Server 2003, five are running Vista, and the remaining three are XP professional. For some unknown reason, one of the Vista computers will periodically refuse a connection from the XP and Server 2003 machines. It appears that if we restart the Vista machine, it will allow the connection again, but then, at some point in the future, it will start denying the connection again. The message I receive is as follows:


\\Officepc is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out if you have access permissions.

The specified server cannot perform the requested operation.


As I said before, if I restart this machine, I will have no problem accessing it again. In the meantime, all of my Vista machines can access this computer without any problems. I have logged in as myself (Domain Admin) from both types of machines to make sure it was not a user issue. Any thoughts?

Interesting issue.

Do you know how to use a packet sniffer to collect a network trace? It's not hard - install the sniffer app, start a capture, repro the problem, stop and save the capture.

Wireshark (www.wireshark.org) is a great sniffer and free too.

If you can create a small trace of a failure-to-access-from-Win2k3, zip it up, and upload here, it might be possible to give you more info about what's going wrong.

Also, you might want to check the global "paged pool" and "non-paged pool" numbers on the problem Vista box. It wouldn't surprise me if a leaky driver was causing pool memory to be depleted, and hence leading to a gradual breakdown in services, though that's just a wild stab in the dark.

Thanks for the advice. I will start with the packet sniffer and see if I can locate anything suspicious.

By the way, I had our secretary (the person using the Vista machine in question) restart her computer, and, of course, the problem went away. However, I suspect it may be back within a few days. I'll post again when I have been able to reproduce the problem.

The problem resurfaced, so I have used Wireshark to run some tests on the connection, both from a successful computer (running Vista), and an unsuccesful attempt (running Server 2003).

I have attached both trace files to this post as txt files. The correct file extention is pcap.

The response to the 2003 Server is "Error: Out of memory", while the Vista computer connects without any problems. Does a connection between two Vista machines use a different port or protocol than between XP/Server 2003 and Vista? It appears the 2003 server tries to connect on the NBSS protocol (port 139), while Vista uses SMB2 (port 445). Is this correct?

If the problem is that port 139 on the problem computer is somehow getting locked on another connection, how do I go about fixing this.

Thanks again for any help anyone can provide!

Unless computer networks are your bread'n'butter, you've done extraordinarily well here, both in the collection of data and the way you analysed it. Have some rep

A bit of background...

Vista-to-Vista or Vista-to-Win2K8, or any combination thereof, will indeed use a new protocol called SMB2, as opposed to the old version which (now) gets labeled SMB1. For obvious backwards compatibility reasons, Vista and Win2K8 also operate quite happily via SMB1, although it's far slower over high-latency links. All prior versions including Win2K, XP, and Win2K3 understand only SMB1.

However, port 445 is used by both SMB2 and SMB1. It's called "direct-hosted SMB" to distinguish it from SMB messages wrapped up in a small NetBIOS session header, which the Server service listens to via TCP port 139. If you do a NETSTAT -NA on one of the Win2K3 servers, you should see it LISTENING on both 139 and 445.

By default, a Win2K3 or XP box acting as the client (initiating an SMB session) will launch a two-pronged connection attempt to both 445 and 139, since it doesn't know whether the target is NT4 (which only supported TCP139) or Win2K+ (445 also available). If it receives positive connection responses via both, it'll just dump the 139 session and continue on 445. What's slightly odd in your case is the lack of a concurrent 445 attempt from Win2K3 to Vista, but that might just be the way the bindings are configured (on Win2K3) and it's not the root cause of the failure to connect.

As you pointed out, the "out of memory" error in response to the "Negotiate" command via 139 is the lethal factor that kills the Win2K3->Vista connection attempt, and obviously it hasn't (yet) started to affect the slightly different handler on port 445.

I think your Vista box has a memory leak. The fact that it always takes about a week to manifest itself suggests a nice medium-rate leak, probably in paged or non-paged pool down in kernel-mode. Once a certain threshold is reached, stuff starts failing and it's likely that if you left the Vista box in that state for another week - without a reboot - other services would start showing symptoms of breakdown as well.

Wireshark has served its purpose and it can't reveal the reason for the "out of memory" condition, but perfmon can. If you periodically keep an eye on the "Memory\Pool paged bytes" and "Memory\Pool non-paged bytes" during the week, I've got a hunch you'll see at least one of those creeping upwards. If the Vista box is 32-bit, it's really only got several hundred MB of each of those pools, irrespective of the amount of RAM. Once the NPP utilisation reaches say 150MB or higher, the box will start doing straaange stuff.

I don't want to drone on pointlessly, just in case my hunch is wrong, so I'd be interested to hear what happens to those pool counters before suggesting how to find the "leaker" (it's a driver, but which one?).

Thank you so much for all of the information! I am going to be watching the information you suggested via perfmon for the next few days, and I will be sure to post again when I see some changes. Thanks!

Well, I may have a verdict concerning the pool paged/nonpaged bytes. The problem of connecting to the this specific Vista machine resurfaced today. I have been following both the paged and non paged bytes on that computer just about every day over the last week or so. The non-paged bytes has never gone above 42MB, and the paged pool has stayed fairly consistently between 110-133MB. In fact, the problem machine has somewhat similar hardware to my laptop (Intel Core 2 Duo (2.2Ghz) with 2 GB RAM), and the numbers between the two machines have stayed very consistent with one another. I understand that the specs may not have an effect on the pool paged/nonpaged bytes based on the previous post, but I suspect they were similar between these two machines because of the similar hardware. Would that be correct?

In addition, I also kept an eye on a third computer to use it for comparison. This computer has lower hardware specs, but the non-paged numbers were about the same as the other two machines. However, the paged pool was quite a bit lower, which I suspect is due to the fact it only has 1 GB RAM.

Is there something else that could be eating the memory for this?

I think I speak for both of us when I say - bummer

A pool leak would have been the best outcome from the perspective of a relatively quick fix, but there's no use wishing. For what it's worth, the answer to your question is that given similar hardware (and therefore similar drivers) it would be expected for two machines to share roughly comparable pool utilisation figures.

OK, so the error message that percolates up to the network layer is "out of memory", and the symptom is periodic in the sense that it occurs a certain time after a reboot, but yet there's no obvious evidence of a pool memory leak. Hmmmm.... this won't be simple...

Can you run that Vista box without anti-virus for a while? Outright uninstall it. There's a chance that the AV filter driver is the culprit.

If you haven't already done so, it would be worthwhile to update all relevant drivers: NIC driver, non-default firewalls (if any), backup agents... anything on that Vista machine which may include a kernel-mode driver which would participate in file access.

The next time it happens check:

- "Memory\Committed Bytes". Is it anywhere near the combined total of the RAM size + pagefile size(s)? For example, with 2GB of RAM and say 3GB of pagefile, the "commit limit" is 5GB. Is that committed bytes counter creeping anywhere even near 5GB?

- "Process\svchost<instance>\private bytes", where the svchost instance is the one whose "ID process" counter (PID) matches the LanmanServer container when you type TASKLIST /SVC. Is the "private bytes" counter going beyond 1GB or perhaps even beyond 1.5GB?

For example, on this machine the relevant PID would (currently) be 420:

W:\>TASKLIST /SVC
Image Name PID Services
========================= ======== ============================================
...

svchost.exe 372 AudioSrv, Dhcp, Eventlog, lmhosts,
p2pimsvc, PNRPsvc, wscsvc

svchost.exe 420 AeLookupSvc, Appinfo, AppMgmt, BITS,
Browser, gpsvc, IKEEXT, iphlpsvc,
LanmanServer, MMCSS, ProfSvc, RasMan,
Schedule, seclogon, SENS, ShellHWDetection,
Themes, Winmgmt, wuauserv

So in perfmon, I'd be checking the "process\private bytes" of the svchost instance whose current "ID process" is 420 - it contains LanmanServer.

- What happens if you just restart the Server service on the Vista machine instead of rebooting (NET STOP SERVER, NET START SERVER). Does that clear up the issue without a reboot?

Anything can be fixed. It just depends on how much time you're willing to invest.

First of all, thank you so much for all of the help! In addition to hopefully discovering the solution, I have learned quite a bit from all of this information.

I wrote out a rather long response a few minutes ago under the impression that we were still experiencing the problem this morning. Then I realized that an automatic backup process from one of the W2K3 servers to the problem computer ran successfully this morning at 4:00 am, which simply means that the computer must have been restarted last night. (I hate to admit that there is the slight possibility that I was the one who restarted it, but I just do not remember for sure; it's been a long week ) I guess I will have to wait until the problem resurfaces to run some of the tests you suggested.

I do have one question. I will admit that I got a little lost on the "private bytes" section. However, that was only because after determining the PID for the svchost process (1172), I could not locate the corresponding service in perfmon. Under "process\private bytes" on both my laptop and the trouble machine, the svchost processes are simply numbered from 1 on up (although there is one svchost without a number). So under perfmon, I only had choices of "svchost", "svchost#1", "svchost#2", and so on. How do I determine which svchost instance in perfmon corresponds to the correct PID under the tasklist?

Again, thanks for the help!

Yeah, I know what you mean about long weeks

What I wrote for the "private bytes" bit is clear as mud. Sorry. You're still on the right track though.

In perfmon, each one of those svchost#1, svchost#2, svchost#3... instances will have its own "ID Process" counter. Once you use TASKLIST /SVC to work out the PID of the svchost instance in which you're interested - let's say 1172 - you can tie that up to the corresponding instance in perfmon by looking at every svchost "ID Process" until you find the one that's 1172. Only one of them will have that ID, and that's the instance whose "private bytes" you want to monitor because it contains the LanmanServer control "applet" (for lack of a better name) for the SMB Server service.

What we're trying to work out is where the "out of memory" message might be coming from. There are more sophisticated ways to do that, but they're impractical over a web forum.