You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With Java 8+ it's possible to read file using streams instead of using loop.
This is so cool, as file isn't read entirely to memory.
Now I'm facing this issue with go.
I need to read hundreds of text files with millions of records and process them.
Thus I need reader and writer that will be compatible with this library.
I tried to do so but it's hard considering implementing generic interface IIterator
The strength of java in this case is the fact that streams became standard.
Do you think it's possible to include such io.reader or bufio.scanner to this library?
The text was updated successfully, but these errors were encountered:
Actually It's not secret so I can share the actual problem.
I maintain DNS with domain blocklist. Blocklist is built using hostlists.
Hostlists are just text files with domains written line-by-iline but there are plenty of formats. They may be obtained locally from disk or from network by https
Consider just a few formats of hostlists written that way:
Each one I need to process that way they are cleaned from unwanted expressions (0.0.0.0, 127.0.0.1 etc.).
All set of rules is more complicated, so that's why I use lazy streams for efficient mapping and filtering.
In this case I need reader of type string.
The problem is how to implement such generic reader?
Shall we write two readers for instance: LazyReader of T interface that implements IIterator and LazyLinewiseReader (with fixed T = string) for this specific case?
In the future someone may need to implement reading integers byte by byte so he/she would Implement such reader as LazyIntegerByteReader.
Hi!
With Java 8+ it's possible to read file using streams instead of using loop.
This is so cool, as file isn't read entirely to memory.
Now I'm facing this issue with go.
I need to read hundreds of text files with millions of records and process them.
Thus I need reader and writer that will be compatible with this library.
I tried to do so but it's hard considering implementing generic interface
IIterator
The strength of java in this case is the fact that streams became standard.
Do you think it's possible to include such
io.reader
orbufio.scanner
to this library?The text was updated successfully, but these errors were encountered: