parallel-reader
A Java Reader can read lines from a file or a reader parallelly
Purpose
Reading a big file line by line can be quite time consumption. It is often due to the IO bottleneck rather than the CPU power. Dividing a big file into smaller parts and reading all of it parallelly can be a solution.
Maven Repo
<dependency>
<groupId>com.github.wnameless.io</groupId>
<artifactId>parallel-reader</artifactId>
<version>1.0.0</version>
</dependency>Quick Start
LineReaders with Reader:
// Reader must be provided by a Supplier
Supplier<Reader> readerSupplier = () -> new StringReader("1\n2\n3\n4\n5\n");
// the max line number of each part
int maxLines = 2;
List<CompletableFuture<Integer>> futures =
LineReaders.readParallelly(readerSupplier, maxLines,
(part /* the index of each part starting from 0 */, lineReader) -> {
while (lineReader.hasNext()) {
String line = lineReader.readLineQuietly();
System.out.println(line);
}
return part;
});LineReaders with File:
File file =new File("path_to_your_file");
// the max line number of each part
int maxLines = 2;
List<CompletableFuture<Integer>> futures =
LineReaders.readParallelly(file , maxLines,
(part /* the index of each part starting from 0 */, lineReader) -> {
while (lineReader.hasNext()) {
String line = lineReader.readLineQuietly();
System.out.println(line);
}
return part;
});LineReaders with Executor:
// Optional
Executor executor = Executors.newFixedThreadPool(4);
File file =new File("path_to_your_file");
// the max line number of each part
int maxLines = 2;
List<CompletableFuture<Integer>> futures =
LineReaders.readParallelly(file , maxLines,
(part /* the index of each part starting from 0 */, lineReader) -> {
while (lineReader.hasNext()) {
String line = lineReader.readLineQuietly();
System.out.println(line);
}
return part;
}, executor);
