NativeInterop — Generic pointers and native 64-bit arrays for .NET

Package	NuGet Release
NativeInterop
NativeInterop.SIMD

This library provides generic pointer operations for all of .NET, building on the capabilities provided by FSharp.NativeInterop, which this package extends with features like 64-bit capabilities, exposed in an OOP-friendly manner as NativePtr<T> and extension methods to System.IntPtr.

NativeArray<T>, an array-like collection of items of unmanaged (blittable) type T allocated on the unmanaged heap, supports 64-bit addressing and item access with and without bounds checks, scalar as well as vectorized operations, utilizing the CPU's AGUs for address offset calculation where possible.

Auxiliary modules like the Stream and Buffer modules provide functionality for easy and efficient serialization/deserialization of unmanaged structs to/from managed arrays or direct manipulation of stack-allocated values (Structure.Read/Write/Get/Set methods).

Motivation

Update [2017-08-20]: Using System.Runtime.CompilerServices.Unsafe native pointers can now be dereferenced in a generic way in C# without using NativeInterop.

In C# (and probably VB as well) it is not possible to write functions that operate on generic pointer types. For example, writing a library for handling generic native arrays or for reading binary structured data would at some point require writing a function like

#!c#
    static unsafe T Read<T>(T* ptr) {
        return *ptr;
    }

which, however, isn't possible to express in C#, because only unmanaged types can be dereferenced that way; C#, however, lacks a way of expressing this type constraint (unmanaged types are either primitive types or structs composed of only unmanaged types; note that struct is not a sufficient constraint, as a struct might be composed of managed types). F#, on the other hand, is able to express that (see MSDN: Constraints (F#)). In F#, we thus simply write

#!f#
    open NativeInterop    
    let pVal = NativePtr.read ptr

to achieve our goal. Here ptr would be of type nativeptr<'T> where 'T would be constrained to unmanaged types.

Unfortunately, it is not possible to directly call NativePtr.read et al. from C#. NativeInterop therefore exposes this functionality via shallow wrappers around the NativePtr APIs and provides additional functionality that builts on top of this raw pointer handling functionality (all those types/modules can be found in namespace NativeInteropEx):

The add-on package NativeInterop.SIMD contains the following additonal types and modules to enable vectorized processing of NativeArray<T>s:

Note

1. Code using this library should be considered unsafe by default. It is the user's responsibility to ensure to only use it in conjunction with unmanaged types. Failing to do so may lead to undefined behavior. If you do not understand what this means, do not use this library!

2. The current release does not support endianess conversion, i.e. creating a binary file on a big endian machine using NativeInteropEx.Stream and then reading that file on a little endian machine will only return garbage.

Installation via NuGet

The easiest way to set up NativeInterop is to type

PM> Install-Package NativeInterop 

in the NuGet Package Manager Console. The current version is built both as a portable class library for .NET 4.5 and Windows 8 store apps as well as a .NET Standard 1.1 compatible package.

To install the SIMD extensions, use

PM> Install-Package NativeInterop.SIMD

NOTE: NativeInterop.SIMD depends on System.Numerics.Vectors, which right now doesn't work out of the box with F# projects. To make it work, make sure that in your fsproj file, the reference to System.Numerics.Vectors looks something like that:

<Reference Include="System.Numerics.Vectors, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
  <HintPath>..\packages\System.Numerics.Vectors.4.1.1\lib\net46\System.Numerics.Vectors.dll</HintPath>
  <Private>True</Private>
</Reference>

The above works for v4.1.1 of System.Numerics.Vectors. For other versions, you may want to create a C# project, add the System.Numerics.Vectors dependency and copy the generated Reference from there to your fsproj file.

Building from Source

Download the current version of the NativeInterop source using Mercurial

hg clone https://bitbucket.org/frank_niemeyer/nativeinterop

... and build the Visual Studio solution.

Redistribution

If you want to redistribute software that uses NativeInterop, make sure to include the class library NativeInteropEx.dll as well as FSharp.Core.dll.

The SIMD extensions further require NativeInteropEx.SIMD.dll, NativeInteropEx.VectorView.dll and System.Numerics.Vectors.dll.

Examples

Reading from/writing to a pointer T*

#!c#
    using NativeInteropEx;
    // ...
    
    // initialize pointer from "somewhere"
    IntPtr p = ...;
    // read from p interpreting p.ToPointer() as float*
    var n1 = NativePtr.Read<float>(p);
    // read from p interpreting p.ToPointer() as int*
    var n2 = p.Read<int>(); // using extension methods
    // *(p + 10), interpreted as float
    var n3 = p.Get<float>(10);
     
    // use the NativePtr<T> wrapper to treat p as a typed pointer
    var pTyped = new NativePtr<float>(p);
    // write to memory location
    pTyped.Value = 42.0f;
    // read offset value
    var n4 = pTyped[10]; // n4 == n3

Note that p.Read<int>() has the same effect as Marshal.PtrToStructure<T>(p); yet the former is much more efficient (and much less safe...) as it boils down to

nop
ldarg.0
ldobj !!T
ret

(one or two simple mov instructions in x86, if inlined) while the latter is equivalent to

ldarg.0
ldtoken !!T
call class System.Type System.Type::GetTypeFromHandle(valuetype System.RuntimeTypeHandle)
call object System.Runtime.InteropServices.Marshal::PtrToStructure(native int, class System.Type)
unbox.any !!T
ret

Reinterpreting an array of type T[] as type U[]

The same techniques as in the example above can be used to interpret an array of one type as an array of another type, without copying it (e.g. using Buffer.Convert). You only need to pin the array first to get its address:

#!c#
    using NativeInteropEx;
    
    var longArray = new long[] { 1, 2, 3 };
    var arrayHandle = GCHandle.Alloc(longArray, GCHandleType.Pinned);
    var pArray = handle.AddrOfPinnedObject();
    
    // interpret longArray as byte[] (or rather nativeptr<byte>) using the 
    // IntPtr extension methods from NativeInterop.NativePtr
    var byte0 = pArray.Get<byte>(0);
    pArray.Set<byte>(1, byte0);
    
    arrayHandle.Free();    

Of course you could also use the fixed keyword instead of a GCHandle for more efficient pinning.

Another option is to use ReinterpretCast<T, U> (NativeInterop v2.4+) which takes care of the pinning/unpinning:

#!c#
    using NativeInteropEx;
    
    var longArray = new long[] { 1, 2, 3 };
    // interpret longArray as nativeptr<byte>
    using (var byteView = new ReinterpretCast<long[], byte>(longArray)) {
        var byte0 = byteView[0];
        byteView[1] = byte0;
    }

Using the Buffer and Structure modules:

#!c#
    using NativeInteropEx;
    // ...
    
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    struct Foo
    {
        int i;
        float f;
    }
     
    // copy the bytes of foo to a new byte[] fooBuff
    var foo1 = new Foo();
    var fooBuff = Buffer.FromStructure<Foo, byte>(foo1);
    // interpret the bytes in fooBuff as a Foo object foo2
    var foo2 = Structure.FromBuffer<byte, Foo>(fooBuff);
    
    // directly access foo1 in a byte-wise fashion
    // e.g. read the first four bytes (= field i)
    var b1 = Structure.Get<Foo, byte>(ref foo1, 0);
    var b2 = Structure.Get<Foo, byte>(ref foo1, 1);
    var b3 = Structure.Get<Foo, byte>(ref foo1, 2);
    var b4 = Structure.Get<Foo, byte>(ref foo1, 3);
    // change the second byte of foo1 to 123 (basically *((byte*)&foo1 + 1) = 123)
    Structure.Set(ref foo1, 1, (byte)123);

    // "memcpy": Copy the bytes of an int[] to a byte[]
    var ints = new[] { 1, 2, 3, 4 };
    var intBytes = Buffer.Convert<int, byte>(ints);

Note that Buffer.Convert converts from T[] to U[] by copying its contents to a new array. If you only want to reinterpret some of its contents, use the technique demonstrated in the previous sample instead.

Using NativeArray<T>:

#!c#
    using NativeInteropEx;
    // ...
    
    // create a native array of 1000 floats
    using (var narr = NativeArray.Initialize(1000, i => i * 2.0f)) {
        // output some diagnostics
        Console.WriteLine("Allocated {0} bytes", narr.AllocatedBytes);
        Console.WriteLine("Base address: {0:X}", narr.BaseAddress);
        
        // write to native array
        narr[0] = 1.0f;
        narr[1] = 2.0f;
        narr[2] = 3.0f;
        // ...
        // read from native array
        Console.WriteLine(narr[0] + narr[1] + narr[2]);

        // compute the dot product (squared L2 distance)
        var dp = narr.Dot(narr);
    }           

Using Stream:

#!c#
    using NativeInterop;
    // ...
    
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    struct Foo
    {
        int i;
        float f;
    }
    
    Foo[] foos;
    using (var fooStream = ...) {
        // read 1000 struct objects from the underlying byte stream
        foos = fooStream.ReadUnmanagedStructRange<Foo>(1000);
    }

License

The NativeInterop library extension is distributed under the terms of the Apache 2 license.

Copyright (c) 2014, 2015, 2016 Frank Niemeyer All rights reserved.