Lately, I have been requested to build a proof of concept of a process that would continuously read the end of a text file as soon as another process was updating it. Imagine a log file in which an application is frequently adding new rows at the end and you want to process only the new rows.
Available source code
Both VB and C# versions are provided this month.
The solution was created using Visual Studio 2019 but should also work in previous versions.
Limitations
My requirement about reading only the end of the file come from the fact that the file can be huge and always re-opening it is time consuming. In my case, it is a kind of log file and rows are only appended to it.
I have to warn that I have found this mechanism to be helpful but I wouldn’t rely on it for life-dependant processes as once you started to play with it a lot, chances are that you will surely experiment glitches (i.e. some newly added rows might be ignored). But even with that limitation, it was still valuable for my case.
Building the UI
The UI is very simple. As shown in figure 1, you have a button and a listbox.
Figure 1: The demo application in action
One thing the figure 1 does not show is that a FileSystemWatcher control has also been added to the form. This control is used to detect changes to the file.
The code
As you can surely guess by the controls on the form, there will be very little interactions. A click on the button will start the whole process, and the FileSystemWatcher will trigger events when changes are detected.
Let’s start with the button.
Private ReadOnly _strPath As String = "c:\_temp" Private ReadOnly _strFileName As String = "test.txt" Private _textReader As StreamReader Private _fileLength As Integer Private Sub btnStartMonitoring_Click(ByVal sender As Object, ByVal e As EventArgs) Handles btnStartMonitoring.Click listBox1.Items.Add($"Start monitoring {_strFileName} at {DateTime.Now.ToLongTimeString()}") 'load the current file FullLoad() 'initialize the file system watcher and start monitoring fileSystemWatcher1.Path = _strPath fileSystemWatcher1.Filter = _strFileName fileSystemWatcher1.NotifyFilter = NotifyFilters.LastWrite fileSystemWatcher1.EnableRaisingEvents = True End Sub Private Sub FullLoad() Try Dim fs = New FileStream(Path.Combine(_strPath, _strFileName), FileMode.Open, FileAccess.Read, FileShare.ReadWrite) _textReader = New StreamReader(fs) Do Dim line As String = _textReader.ReadLine() If line Is Nothing Then Exit Do listBox1.Items.Add(line) Loop listBox1.SelectedIndex = listBox1.Items.Count - 1 _fileLength = CInt(_textReader.BaseStream.Length) Catch ex As FileNotFoundException MessageBox.Show("You need to create your test file first") End Try End Sub
private readonly string _strPath = @"c:\_temp"; private readonly string _strFileName = "test.txt"; private StreamReader _textReader; private int _fileLength; private void btnStartMonitoring_Click(object sender, EventArgs e) { if (!string.IsNullOrWhiteSpace(fileSystemWatcher1.Path)) { fileSystemWatcher1.Changed -= fileSystemWatcher_Changed; } //load the current file listBox1.Items.Add($"Start monitoring {_strFileName} at {DateTime.Now.ToLongTimeString()}"); FullLoad(); //initialize the file system watcher and start monitoring fileSystemWatcher1.Path = _strPath; fileSystemWatcher1.Filter = _strFileName; fileSystemWatcher1.NotifyFilter = NotifyFilters.LastWrite; fileSystemWatcher1.EnableRaisingEvents = true; fileSystemWatcher1.Changed += fileSystemWatcher_Changed; } private void FullLoad() { try { string line; var fs = new FileStream(Path.Combine(_strPath, _strFileName), FileMode.Open, FileAccess.Read, FileShare.ReadWrite); _textReader = new StreamReader(fs); while ((line = _textReader.ReadLine()) != null) { listBox1.Items.Add(line); } listBox1.SelectedIndex = listBox1.Items.Count - 1; _fileLength = (int)_textReader.BaseStream.Length; } catch (FileNotFoundException ) { MessageBox.Show("You need to create your test file first"); } }
The first real thing happening here is the call to the FullLoad method. This method simply opens a StreamReader containing the file specified from the members defined at the top of the form (make sure you adjust the path and the filename to match a test file you will use), reads all the lines from that file to display them into the ListBox control so we can see what’s happening. Finally, the current stream length (number of characters) is kept into the _fileLength variable for future use.
After this FullLoad method has been executed, a few properties of the FileSystemWatcher control are set to ensure we receive events when this file is modified. Do you know I wrote an article on that control almost 15 years ago? You can still read it from https://www.emoreau.com/Entries/Articles/2005/04/The-FileSystemWatcher-component.aspx.
The other meaningful section of code is the one found in the Changed event handler of the FileSystemWatcher control.
Private Sub fileSystemWatcher_Changed(ByVal sender As Object, ByVal e As FileSystemEventArgs) Handles fileSystemWatcher1.Changed 'sometimes the event is triggered with an invalid stream If _textReader.BaseStream.Length = 0 Then Return listBox1.Items.Add(New String("-"c, 40)) listBox1.Items.Add($"{e.ChangeType} - Previous length={_fileLength} / Current length={_textReader.BaseStream.Length}") If _textReader.BaseStream.Length > _fileLength Then listBox1.Items.Add(New String("-"c, 40)) listBox1.Items.Add($"Adding new items at {DateTime.Now.ToLongTimeString()}") Dim strEndOfFile = _textReader.ReadToEnd() For Each strNewItem As String In strEndOfFile.Split({vbCrLf}, StringSplitOptions.None) listBox1.Items.Add(strNewItem) Next listBox1.SelectedIndex = listBox1.Items.Count - 1 _fileLength = CInt(_textReader.BaseStream.Length) ElseIf _textReader.BaseStream.Length < _fileLength Then 'file is shorter, just reload to start fresh listBox1.Items.Add($"Stream is shorter than the previous one. Fully reloading the file at {DateTime.Now.ToLongTimeString()}") FullLoad() End If End Sub
private void fileSystemWatcher_Changed(object sender, FileSystemEventArgs e) { //sometimes the event is triggered with an invalid stream if (_textReader.BaseStream.Length == 0) return; listBox1.Items.Add(new string('-', 40)); listBox1.Items.Add($"{e.ChangeType} - Previous length={_fileLength} / Current length={_textReader.BaseStream.Length}"); if (_textReader.BaseStream.Length > _fileLength) { listBox1.Items.Add(new string('-', 40)); listBox1.Items.Add($"Adding new items at {DateTime.Now.ToLongTimeString()}"); var strEndOfFile = _textReader.ReadToEnd(); foreach (string strNewItem in strEndOfFile.Split(new[] { "\r\n" }, StringSplitOptions.None)) { listBox1.Items.Add(strNewItem); } listBox1.SelectedIndex = listBox1.Items.Count - 1; _fileLength = (int)_textReader.BaseStream.Length; } else if (_textReader.BaseStream.Length < _fileLength) { //file is shorter, just reload to start fresh listBox1.Items.Add($"Stream is shorter than the previous one. Fully reloading the file at {DateTime.Now.ToLongTimeString()}"); FullLoad(); } }
If the current length of the stream is longer then what was previously read (remember the _fileLength variable?), then the end of the stream is read and added to the listbox control so we can see what was added.
If the length is shorter, that would mean that the file has been somewhat truncated. We have no other choice than to start over the process by calling the FullLoad method again.
Before running the project
Be sure that you modify the values of the _strPath and _strFileName to match an existing text file you have on your computer.
While the demo application is running, after you clicked the button to load the initial values into the listbox, open your file using an application like Notepad or Excel, add rows to your file and save it. Your test application should reflect the newly added rows instantaneously.
Why a FileStream?
You might be wondering why I have used a FileStream object to open the file and then pass it to the StreamReader object? I could have used the path and filename members directly to the StreamReader constructor and it should have worked as well no?
In theory yes. But have you ever seen an exception saying: "The process cannot access the file 'file.txt' because it is being used by another process" as shown in figure 2?
Figure 2: Exception trying to open a file already opened in Excel
While building my proof of concept application, I found that some applications (namely Excel but others do too) don’t like to share files. They hold it and prevent doing anything with it. In my case here, since my application reading the file only need to read the file, there was no need of holding it that tight! The solution is to use the FileStream object specifying FileShare.ReadWrite (the fourth argument). These arguments are not available to the StreamReader object.
Conclusion
It might not be a 100% reliable solution but there are some scenarios in which this simple mechanism can be very useful.