FIP: A better Storable type class

[NicolasDP]

Overview

Foundation aims to bring better typed object representation in order to help users to quickly understand a given API, but also to gives better type checking and therefor semantic validation at compile time.

This is a draft proposal on what could look like a new interface for Foreign Storable.

Foreign interface: The state of the art

Storable object (object which can be marshalled to/from foreign interfaces, but not only) are originally defined as follow:

• it has a size in byte;
• it has an alignment;
• it can be read from a given ptr address;
• it can be written to a given ptr address;
• it can be read from a given ptr address at a given offset;
• it can be written to a given ptr address at a given offset.

class Storable a where
    sizeOf :: a -> Int
    alignement :: a -> Int
    peek :: Ptr a -> IO a
    poke :: Ptr a -> a -> IO ()
    pokeElemOff :: Ptr a -> Int -> a -> IO ()
    peekElemOff :: Ptr a -> Int -> IO a

This type class seems to provide what one would need to safely marshall objects to/from a given address.

Limitations of the current state

1. partially typed interface (sizeOf, alignment and p***ElemOff), this is not the culture of the Foundation to leave such non labeled size and offsets;
2. peek and poke element at a given offset from a given address is not obviously checking the alignment of the newly computed address (the default implementation of peekElemOffis peekByteOff ptr (off * sizeOf undef));
3. an object can be read from a foreign address but not necessary serialised into a consecutive part of the memory (i.e. p***ElemOff may not apply for certain types).

New storable type classes

Storable type class

Without having any information about the size of a given type, we want to be able to peek or poke an object from a given pointer. In this case, the sizeOf information is not relevant. A given object can have a variable size and yet be peeked or poked at a given address.

class Storable a where
    peek :: Ptr a -> IO a
    poke :: Ptr a -> a -> IO ()

Some primitives already interface this Storable: Word8, Int... PrimType type class

An example of object that can be shared with foreign libraries is a CString. It is litterally an array of Word8 terminated by a zero.

There are also C Structure of variable size which can be defined:

struct array {
    size_t length;
    uint8_t array[];
};

array* new(size_t length) {
  array* arr = malloc(sizeof(array + length));
  arr->length = length;
  return arr;
}

In this case, the structure will have a dynamically known size that peek or poke can easily implements:

newtype Array = Array [Word8]

-- NB: this is only pseudo code, you would obviously need to cast the pointers...
instance Storable Array where
    peek ptr = do
        csize <- peek ptr
        forM [1..csize] $ \off -> peek (ptr `plusPtr` 4 `plusPtr` off)
    poke ptr (Array l) = do
        poke ptr (length l)
        forM_ (zip [1..] l) $ \(off, a) -> poke (ptr `plusPtr` 4 `plusPtr` 1) a

while this structure seems pretty easy to implement, one must keep in mind the burden of alignment and the architecture (CSize may be 8 bytes long). Also a packed structure may have a different size on different architecture.

StorableFixed type class

A SizedStorable is a Storable element which can be peeked/poked from/to an offset and an address in memory and to do so, the only information we need is its size and its alignment.

class Storable a => StorableFixed a where
    size :: proxy a -> Size Word8
    alignment :: proxy a -> Size Word8

-- convenient function to perform aligned pointer arithmetic
plusStorable :: StorableFixed a => Ptr a -> Size a -> Ptr a
plusStorable ptr (Size num) = ptr `Foreign.Ptr.plusPtr` (num * (size ptr `align` alignment ptr))
  where
    align (Size sz) (Size a) = sz + (sz `mod` a)

And now it is very simple to define generic function using the 2 given interfaces.

-- only use a combination of `peek` and aligned `ptr`
peekOff :: StorableFixed a => Ptr a -> Offset a -> IO a
peekOff ptr off = peek (ptr `plusStorable` (offsetAsSize off))

-- only use a combination of `poke`and aligned `ptr`
pokeOff :: StorableFixed a => Ptr a -> Offset a -> a -> IO ()
pokeOff ptr off = poke (ptr `plusStorable` (offsetAsSize off))

Compatibility with prim types

Every PrimtType are SizedStorable as we already know their size.

import Foundation.Primitive
import Foreign.Ptr

instance PrimType ty => Storable ty where
    peek (Ptr addr) = primAddrRead addr (Offset 0)
    poke (Ptr addr) = primAddrWrite addr (Offset 0)

instance PrimType ty => StorableFixed ty where
    size = primSizeInBytes
    alignment = primtSizeInBytes

[ndmitchell]

Great idea to do some work on this area. A few notes:

• Foreignable is not a great word, as a native speaker. It just sounds a bit wrong.
• Perhaps you want Storable instead of Foreignable, and StorableFixed instead of Storable?
• Could you have class StorableFixed a => PrimType a?
• If size and alignment took proxy a instead of Proxy a then Ptr would already be a valid proxy. The downside is I think it has issues with newtype deriving (not sure if they can be worked around or not).

I'm looking forward to this work.

[NicolasDP]

I agree Foreignable is not a good choice.

[NicolasDP]

What about SizedStorable? I updated the Proposal and added more comments and precisions.

[NicolasDP]

I think you are right that PrimType could have SizedStorable as constraint.

We could also remove primSizeInBytes which is only the SizedStroable's function size.

[vincenthz]

I think StorableFixed convey better the idea of fixed size than SizedStorable.

For PrimType, this is not related to Storable. I don't think it's a good idea to link the concept