A simple to use efficient java csv marshalling library driven by annotations. Converts a csv source into a stream of java beans for processing within an application, equivalent to a handwritten marshaller in performance. The annotation processor generates a CSV marshaller at build time for any class annotated with @CsvMarshaller.

• Simple to use annotations, automatically map CSV to POJO fields
• Fully RC4180 compliant csv parser for reading and writing csv
• CSV data pipeline transformations
• Integrates with lombok to reduce boilerplate code
• Override header to property name mapping
• Index or named column support for input files
• JavaBean and fluent accessors/mutators style supported
• Optional field support with default value injection
• Full escaping support for quoted fields
• Missing column support allows partial evaluation
• Comments ignored, configurable
• Empty lines ignored, configurable
• Pluggable custom serializers framework
• Pluggable validation framework
• Pluggable lookup support to convert field values
• Error handling and reporting
• Designed to integrate with java.util.stream.Stream
• Compiles AOT, zero startup cost
• Zero GC field serializers for primitive types and CharSequence
• Array and List support for primitives and Strings
• Serializers for LocalTime, LocalDate, LocalDateTime
• Re-use marshalled instance or new instance per record
• No external runtime dependencies
• No runtime byte code generation

• csv-compiler-processor an annotation processor that executes at build time to generate a RowMarshaller. Not required at runtime, with maven use provided scope or explicitly load as an annotation processor.
• csv-compiler a runtime library providing utility functions, zerogc utilities and interface definitions

<dependency>
    <dependency>
        <groupId>com.fluxtion.csv-compiler</groupId>
        <artifactId>csv-compiler</artifactId>
        <version>0.1.18</version>
    </dependency>
    <dependency>
        <groupId>com.fluxtion.csv-compiler</groupId>
        <artifactId>csv-compiler-processor</artifactId>
        <version>0.1.18</version>
        <scope>provided</scope>
    </dependency>
</dependency>

Required steps to set up a successful processing pipeline:

1. Add CVS compiler dependencies to you project.
2. Create a java bean with getters and setter for persistent properties
3. Add a @CSVMarshaller annotation to the java bean source file
4. Load marshaller using RowMarshaller.load([Bean.class])
5. Stream from a reader or a String into the loaded RowMarshaller, add stream logic to process marshalled instances .stream([Reader|String])

Behaviour can optionally be injected into the processing chain:

1. Register a validation logger to record marshalling errors. RowMarshaller.setValidationLogger
2. Register a row validator with RowMarshaller.setRowValidator
3. Register a fatal exception handler for parse errors with RowMarshaller.setFatalExceptionHandler
4. Register a header transformer with RowMarshaller.setHeaderTransformer
5. Register a lookup function with RowMarshaller.addLookup

This example loads a CSV file transforms it and writes the results to another CSV file. The pipeline demonstrates:

• Extracting a subset of columns from the source data
• Replaces missing values with default values
• Columns are one of, input only, input/output or derived output only
• Header names have spaces
• Rows are filtered from output if they do not meet criteria
• Derived value transforms a textual field to numerical representation
• Derived value transforms a numerical value by an operation
• The output is written to a new file excluding the input only columns

We use the ames housing from kaggle as an input source. The goal is to clean and transform this data set, so it can be used in a machine learning model. The source has 82 columns we want to reduce this to a CSV file with a subset of columns that only contains valid records.

A sample from the input data set:

The post processed output data set should have:

• a subset of the input columns
• replace missing values such as Lot frontage with a default value.
• calculate the Lot frontage squared
• remove Lot frontage field from the output
• convert MS Zoning from a text value to a numerical value stored in ms_zone_category field
• remove MS Zoning field from the output
• use fluent accessors to mirror property names, no get/set
• filter rows from the output with missing Lot frontage
• filter rows from the output that are not in a defined MS Zoning category set

A sample output that should be generated

Solving this problem requires two steps; firstly annotate a POJO's fields with @CsvMarshaller, secondly use the utility method RowMarshaller.transform().

The transform function performs the following operations, loads the input file from the supplied path, transforms/filters records with a java.util.stream.Stream provided by the user and then finally writes records to an output file from the supplied path.

The HouseSaleRecord.java class represents the mapping of input, output and derived fields using annotations. The @CsvMarshaller annotation is used in conjunction with lombok to generate a POJO that has fluent accessors, removing the need for boilerplate code. The RowMarshaller for HouseSaleRecord is generated at compile time by the csv-compiler annotation processor.

@Data
@CsvMarshaller(fluent = true)
@Accessors(fluent = true)
public class HouseSaleRecord {
    //input only
    @ColumnMapping(outputField = false)
    private int Order;
    @ColumnMapping(outputField = false, columnName = "Lot Frontage", defaultValue = "-1")
    private int Lot_Frontage;
    @ColumnMapping(outputField = false, columnName = "MS Zoning")
    private String MS_Zoning;

    //input and output
    private int PID;
    @ColumnMapping(columnName = "MS SubClass")
    private int MS_SubClass;

    //derived no inputs
    @ColumnMapping(optionalField = true)
    private int Lot_Frontage_Squared;
    @ColumnMapping(optionalField = true)
    private int ms_zone_category;
}

• Input only fields are annotated with @ColumnMapping(outputField = false)
• Input column names are mapped with @ColumnMapping(columnName = "MS SubClass")
• Output only derived fields are annotated with @ColumnMapping(optionalField = true)
• Annotation free read/write properties are both input and output fields
• Default values are marked with @ColumnMapping(defaultValue = "-1")
• Annotations can be combined - @ColumnMapping(outputField = false, columnName = "Lot Frontage", defaultValue = "-1")

The AmesHousingCsvPipeline.java creates a csv data pipeline using the utility function from the RowMarshaller.transform() class

public class AmesHousingCsvPipeline {

    @SneakyThrows
    public static void main(String[] args) {
        RowMarshaller.load(HouseSaleRecord.class).transform(
                Path.of("data/AmesHousing.csv"),
                Path.of("data/PostProcess.csv"),
                s -> s.filter(record -> record.Lot_Frontage() > 0)
                        .map(AmesHousingCsvPipeline::squareLotFrontage)
                        .map(AmesHousingCsvPipeline::ms_zone_to_category)
                        .filter(record -> record.ms_zone_category() > 0));
    }

    public static HouseSaleRecord squareLotFrontage(HouseSaleRecord houseSaleRecord) {
        int lotFrontage = houseSaleRecord.Lot_Frontage();
        houseSaleRecord.Lot_Frontage_Squared(lotFrontage * lotFrontage);
        return houseSaleRecord;
    }

    public static HouseSaleRecord ms_zone_to_category(HouseSaleRecord houseSaleRecord) {
        switch (houseSaleRecord.MS_Zoning()) {
            case "A" -> houseSaleRecord.ms_zone_category(1);
            case "FV" -> houseSaleRecord.ms_zone_category(2);
            case "RL" -> houseSaleRecord.ms_zone_category(3);
            case "RM" -> houseSaleRecord.ms_zone_category(4);
            default -> houseSaleRecord.ms_zone_category(-1);
        }
        return houseSaleRecord;
    }
}

This example converts String -> csv -> bean -> java.util.stream.Stream for processing, any validation errors are printed to console. The example is here StreamBeans.java

Mark a java bean with annotation @CSVMarshaller use lombok @Data to remove the boilerplate getter/setter methods. The RowMarshaller is generated at compile time by the csv-compiler annotation processor.

@Data
@CsvMarshaller
public class Person {
    private String name;
    private int age;
}

The RowMarshaller.load(Person.class) loads the RowMarshaller for the Person javaBean.

To set the validation logger use .setValidationLogger(ValidationLogger.CONSOLE), in this case directs validation messages to the console.

A string is supplied as input to the marshaller to create a stream of Person instances with .stream("name,age\n" + ..... The stream method returns a java.util.stream.Stream for downstream processing.

Processing logic is added to the returned stream as per standard java .peek(System.out::println) ....

public class StreamBeans {
    public static void main(String[] args) {
        RowMarshaller.load(Person.class)
                .setValidationLogger(ValidationLogger.CONSOLE)
                .stream("name,age\n" +
                        "Linda Smith,43\n" +
                        "Soren Miller,33\n" +
                        "fred,not a number\n")
                .peek(System.out::println)
                .mapToInt(Person::getAge)
                .max()
                .ifPresent(i -> System.out.println("\nRESULT - Max age:" + i));
    }
}

Application execution output:

Main.Person(name=Linda Smith, age=43)
Main.Person(name=Soren Miller, age=33)
Person problem pushing 'not a number' from row:'4' fieldIndex:'1' targetMethod:'Person#setAge' error:'java.lang.NumberFormatException: For input string: "not a number"'
Max age:43

This example converts a single message into a single bean instance

Mark a java bean with annotation @CSVMarshaller use lombok @Data to remove the boilerplate getter/setter methods. The RowMarshaller is generated at compile time by the csv-compiler annotation processor.

@Data
@CsvMarshaller
public class Person {
    private String name;
    private int age;
}

The RowMarshaller.parser(Person.class) loads a SingleRowMarshaller for the Person javaBean.

A call to SingleRowMarshaller.parse will generate a single instance of Person if the message can be parsed. Validation loggers can inject into the flow if called before the parser method.

public class SingleMessageParse {

    public static void main(String[] args) {
        SingleRowMarshaller<Person> parser = RowMarshaller.parser(Person.class);
        //headers - no output
        parser.parse("name,age\n");

        System.out.println("parsed:" + parser.parse("Jane,56\n"));
        System.out.println("parsed:" + parser.parse("Isiah,12\n"));
        System.out.println("parsed:" + parser.parse("Sky,42\n"));
    }
}

Application execution output:

parsed:SingleMessageParse.Person(name=Jane, age=56)
parsed:SingleMessageParse.Person(name=Isiah, age=12)
parsed:SingleMessageParse.Person(name=Sky, age=42)

The CSV compiler annotation processor generates a marshaller during compilation. When deployed as a stateless function in the cloud the only cpu cycles billed are used to parse the data. For smaller documents CSV compiler can finish well before interpreting parsers have completed. Combined with Graal native results in low startup times as marshalling code is statically compiled AOT.

10 column version, 9 columns are ignored.

The parser itself employs several optimisations at runtime:

• Can operate as a zerogc source, re-using the target bean as a flyweight
• Converts primitives in a zerogc manner
• Highly optimised number parsers
• Native support of CharSequence, reusing buffers
• Zero cost abstraction only generating features if specified in annotations
• Skips columns if they are not required in the target bean
• Reduced internal conditional execution to aid branch prediction

If reducing costs and energy consumption are important to you please consider using or help improve CVS compiler.