FileSystemWatcher may cause problems in containers - inotify limits and ASP.NET Core

[shanselman]

Looking around the web I'm seeing years of issues with FileSystemWatcher saying "The configured user limit (n) on the number of inotify instances has been reached."

UPDATE: Looks like https://github.com/dotnet/corefx/blob/a10890f4ffe0fadf090c922578ba0e606ebdd16c/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs#L371 will assume when inotify_add_watch fails with an ENOSPEC it must but an issue with inotify instances being out of range. In fact, ENOSPEC can also mean "the kernel failed to allocate a needed resource."

From the Man Page - The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource

Phrased differently. There's two Error Cases and we throw a message that implies there's just One.

This is becoming more prevalent in container situations in constrained sandboxes. I'm trying to deploy https://github.com/shanselman/superzeit (just clone and "now --public" or run locally with docker) to Zeit.co and I'm hitting this regularly. I don't think I'm hitting a limit. I think Zeit (and others) are blocking the syscall.

I think there are two issues here:

1 We should return a different error message if inotify_add_watch fails, and then circuit break so that FileSystemWatcher doesn't prevent the app from starting. If we CAN startup without a watch successfully, we should.

2 It seems DOTNET_USE_POLLING_FILE_WATCHER=1 is used in dotnet-watch and the aspnet file providers but the base System.IO FileSystemWatcher class doesn't support DOTNET_USE_POLLING_FILE_WATCHER? We should probably be consistent.

- If I change reloadOnChange: false in Program.cs to bypass the first watch that is set on AppSettings.json, I end up hitting it later when Razor/MVC sets up its FileWatchers. 

We need at a minimum, to have DOTNET_USE_POLLING_FILE_WATCHER respected everywhere. Another idea would be for a way to have FileSystemWatcher "fail gracefully." We need to test on systems with

Unhandled Exception: System.IO.IOException: The configured user limit (8192) on the number of inotify instances has been reached.
> [0]    at System.IO.FileSystemWatcher.StartRaisingEvents()
> [0]    at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
> [0]    at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
> [0]    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
> [0]    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
> [0]    at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
> [0]    at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__0_0()
> [0]    at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
> [0]    at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
> [0]    at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
> [0]    at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
> [0]    at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
> [0]    at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
> [0]    at superzeit.Program.Main(String[] args) in /app/superzeit/Program.cs:line 17

Related issues?

• https://github.com/dotnet/corefx/issues/5660
• https://github.com/dotnet/corefx/search?q=inotify&type=Issues
• This issue may have been related AspNetCore IO Exception Invalid Arguement aspnet/Mvc#8299 because the FileWatcher failed to watch.

@stephentoub @natemcmaster @muratg @pranavkm

dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   2.1.401
 Commit:    91b1c13032

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.401\

Host (useful for support):
  Version: 2.1.3-servicing-26724-03
  Commit:  124038c13e

.NET Core SDKs installed:
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

[spboyer]

TryEnableFileSystemWatcher() - gets called regardless if DOTNET_USE_POLLING_FILE_WATCHER is set or not.

https://github.com/idefa/AspnetCoreHome/blob/master/FileSystem-rel-2.0.0/src/Microsoft.Extensions.FileProviders.Physical/PhysicalFilesWatcher.cs#L93

[sebastienros]

I think this PR should fix your problem: aspnet/FileSystem#333
And it's solving this: aspnet/Mvc#8173

/cc @pranavkm

[shanselman]

@sebastienros Thanks. I think that just makes watching these files "more correct" and "more efficient."

I'm digging into this. Our code looks right.

[shanselman]

MOVED TO https://github.com/dotnet/corefx/issues/32024

[rtrimurthulu]

use command
export DOTNET_USE_POLLING_FILE_WATCHER=true
it will work