Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with watcher over a text file #343

Open
MiniaczQ opened this issue Aug 1, 2021 · 7 comments
Open

Problem with watcher over a text file #343

MiniaczQ opened this issue Aug 1, 2021 · 7 comments
Labels

Comments

@MiniaczQ
Copy link

MiniaczQ commented Aug 1, 2021

Hello, this is a question/discussion, but just to give a full context of my problem I've partially filled out the bug form.

System details

  • OS: Windows 10 v10.0.19043
  • Rust version: 1.56.0-nightly
  • Notify version: 5.0.0-pre.11

What you did (as detailed as you can)

I'm working on an utility app that needs to read minecraft's client log.
I cannot read from stdout for irrelevant reasons, so I need to look into a file called 'latest.log', which is a text file.

  1. I tried opening the file in read mode, to read all the logs as they come.
  2. Later I created a watcher that tracks any changes in that file.

What you expected

  1. Since the file is actively being modified, I expected to never find the EOF in it and just get stuck on reading, until the client closes the file.

  2. I expected every addition to the file to report an event.

What happened

  1. The file had EOF, so it stopped reading after it caught up the first time.

  2. No events were reported.
    Further testing revealed that polling would work.
    Also, opening (and closing, no r/w) the file in an asynchronous loop, while the watcher is running also resulted in modifications being detected.

The file seems to get modified only when I try to access it (by opening it or trying to read metadata).

My thoughts and questions

The results appears like a problem with FSEvents (polling works, but not events), but since I'm running Windows, that'd be the FileSystemAudit.
Does this problem even exists on Windows?
If so, is there a way to get the permission for the file and use the event system?
If not, is there any other way I can evade polling? (maybe a way to open the file as a device? So that I don't get EOF, until the client actually closes it)

There are moment where I need extremely quick responses and I'd rather not poll multiple times per second.
If I cannot get this to work with events I'll probably create a variable period poll-watcher.

If there is anything else I can provide that would help with this lmk.

@0xpr03
Copy link
Member

0xpr03 commented Aug 3, 2021

Have you tried watching the whole directory with the logs ? File "writes" come in different ways, so it might be the file is actually re-created.

I cannot read from stdout for irrelevant reasons, so I need to look into a file called 'latest.log', which is a text file.
[..] need extremely quick responses and I'd rather not poll multiple times per second

I can definitely not recommend watching file changes for high-frequency monitoring. Example from a project for doing stdout/err monitoring asynchronously. Your windows defender will scan ever file access and the IO overhead is too high. If you're lucky your OS might at least keep the content in memory and thus prevent re-reading it every time. Though that is completely dependent on the size of the file, which as you can see yourself, will go to staggering numbers depending on your server, mods etc. As you also figured out: There is no notion of getting a "stuck reading" instead of EOF. A file has the content its provided at that time or it doesn't, nothing in between - at least not that I know of such behaviour or that it would make any sense in regards to synchronisation and deadlocking (you're essentially expecting the OS to deadlock your program indefinitely).

@0xpr03 0xpr03 added Z-needs info Needs more information os-windows labels Aug 3, 2021
@MiniaczQ
Copy link
Author

MiniaczQ commented Aug 3, 2021

Have you tried watching the whole directory with the logs ?

I just tried watching the whole directory, but that lead to the same result,
where the 'latest.log' gets modified when the file gets closed (when I close my game client)
and recreated (when I reopen it).

I can definitely not recommend watching file changes for high-frequency monitoring. Example from a project for doing stdout/err monitoring asynchronously.

As much as I'd like to just read the stdout/err, I don't start the minecraft client process.
It is done through launchers.
So even if I were to start the launcher, it would separate me from the minecraft client's stdout/err anyways.
Launching client myself is not an option, because I'm trying to be compatible with already used launchers.

Since you've also said that a system dependent deadlock sounds silly and probably doesn't exist,
then my only option remain the original variable duration poll-watcher
or (if even possible) connecting to stdout of an existing process.
I'll be investigating the second option and currently I found sysinfo crate for finding the process.
Although irrelevant to this Git, if you have any advice on what I can use for this please let me know.
And thank you for the help so far. 😃

Quick update:
I've made few google searches.
Getting stdout of a Windows process is extremely janky and error prone, it seems I'll have to go with the first solution.

@0xpr03
Copy link
Member

0xpr03 commented Aug 4, 2021

I can currently reproduce this problem using C# and FileSystemWatcher, running notify & the c# version side by side.

Another fun thing is that the debug.log file itself has 0KB size, until I click on it inside explorer.exe, then its refreshed and suddenly both applications also receive a change event.

@0xpr03
Copy link
Member

0xpr03 commented Aug 4, 2021

Repro https://www.youtube.com/watch?v=3x_48HhXoA4 Simply Pressing F2 to start a rename triggers an file size refresh in explorer and events for both programs.

@0xpr03
Copy link
Member

0xpr03 commented Aug 4, 2021

What I do not know is how notepad++ is still able to realize file changes.

@jmquigs
Copy link
Contributor

jmquigs commented Aug 8, 2021

I'd be curious if watchers from other language runtimes (like golang) are able to see changes to the file in the use case you describe. Usually a file being written on windows can't be read by other processes at all until the file is closed. Since you can read it (with explorer and notepad++) I presume the file was opened with the FILE_SHARE_READ share access flag.

@0xpr03
Copy link
Member

0xpr03 commented Aug 8, 2021

The C# version from microsoft themself isn't able to do so, as shown above. I guess notepad++ tries to poll or open the file, which activates probably a flush in the OS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants