Add support of array of variants

[charphi]

I need to call a COM method that returns an array of variant.
The var type is 8204 and correspond to VT_ARRAY | VT_VARIANT.

I guess that the code should be added in the class com.sun.jna.platform.win32.Variant.

Here is an example of use in vbs (see GetRows):

On Error Resume Next

Set conn = CreateObject("ADODB.Connection")
conn.Open "Provider=Search.CollatorDSO;Extended Properties='Application=Windows';"

If Err.Number <> 0 Then
  Wscript.Echo "Connection failed: [" &  Err.Number & "] " & Err.Description
  Wscript.quit(1)
End If

Set rs = CreateObject("ADODB.Recordset")
rs.Open "SELECT top 5 System.ItemUrl FROM SYSTEMINDEX WHERE System.FileName like '%readme%'", conn

If Err.Number <> 0 Then
  Wscript.Echo "Query failed: [" &  Err.Number & "] " & Err.Description
  conn.Close
  Wscript.quit(2)
End If

If Not (rs.EOF) Then
  ' >>> GetRows returns a 2-dimensions array
  rows = rs.GetRows
  For i = 0 to uBound(rows)
    Wscript.Echo rows(i,0)
  Next
End If

rs.Close
conn.Close

[dblock]

Please feel free to contribute.

[matthiasblaesing]

@charphi you can try this fork:

https://github.com/matthiasblaesing/jna/tree/safearray

It would be good if you could give it a spin. I used your sample code as basis to create a unittest excercising the SAFEARRAY function.

I will rebase that branch once the fix_com branch or a equivalent was merged.

[charphi]

I've just tested this fork and it works properly. Thanks :)
The only downside is that the code to get the content of an array is quite convoluted.

[twall]

On Jan 25, 2016, at 5:13 AM, Philippe Charles [email protected] wrote:

I've just tested this fork and it works properly. Thanks :)
The only downside is that the code to get the content of an array is quite convoluted.

—
Reply to this email directly or view it on GitHub.

Feel free to submit a PR with some utility methods on the SafeArray class or other appropriate place which makes the extraction more natural.

[charphi]

Here is the code i'm using to create a java array from a safearray:

    public static Object toJavaArray(OaIdl.SAFEARRAY sa) {
        int size = 1;
        int[] dimArray = new int[sa.rgsabound.length];
        for (int i = 0; i < dimArray.length; i++) {
            dimArray[i] = sa.rgsabound[i].cElements.intValue();
            size *= dimArray[i];
        }

        OleAuto.INSTANCE.SafeArrayLock(sa);
        try {
            Variant.VARIANT[] variantArray = (Variant.VARIANT[]) (new Variant.VARIANT(sa.pvData.getPointer()).toArray(size));
            Object result = Array.newInstance(Object.class, dimArray);
            fillJavaArray(result, dimArray, 0, variantArray, 0);
            return result;
        } finally {
            OleAuto.INSTANCE.SafeArrayUnlock(sa);
        }
    }

    private static void fillJavaArray(Object array, int[] dimArray, int dimIndex, Variant.VARIANT[] variantArray, int variantIndex) {
        if (dimArray.length == dimIndex + 1) {
            for (int i = 0; i < dimArray[dimIndex]; i++) {
                Array.set(array, i, toJavaObject(variantArray[variantIndex + i], null));
            }
        } else {
            for (int i = 0; i < dimArray[dimIndex]; i++) {
                fillJavaArray(Array.get(array, i), dimArray, dimIndex + 1, variantArray, variantIndex + dimArray[dimIndex + 1] * i);
            }
        }
    }

[twall]

Wouldn’t that be nicer with some Generics? Casting to an array from an Object seems so late 90s.

On Jan 25, 2016, at 9:21 AM, Philippe Charles [email protected] wrote:

Here is the code i'm using to create a java array from a safearray:

public static Object toJavaArray(OaIdl.SAFEARRAY

sa) {

int size = 1
;

int[] dimArray = new int[sa.rgsabound.
length];

for (int i = 0; i < dimArray.length; i++
) {
dimArray[i]
= sa.rgsabound[i].cElements.
intValue();
size
*=
dimArray[i];
}

OleAuto.INSTANCE.
SafeArrayLock(sa);

try
{

Variant.VARIANT[] variantArray = (Variant.VARIANT[]) (new Variant.VARIANT(sa.pvData.getPointer()).
toArray(size));

Object result = Array.newInstance(Object.
class, dimArray);
fillJavaArray(result, dimArray,
0, variantArray, 0
);

return
result;
}
finally
{

OleAuto.INSTANCE.
SafeArrayUnlock(sa);
}
}

private static void fillJavaArray(Object array, int[] dimArray, int dimIndex, Variant.VARIANT[] variantArray, int
variantIndex) {

if (dimArray.length == dimIndex + 1
) {

for (int i = 0; i < dimArray[dimIndex]; i++
) {

Array.set(array, i, toJavaObject(variantArray[variantIndex + i], null
));
}
}
else
{

for (int i = 0; i < dimArray[dimIndex]; i++
) {
fillJavaArray(
Array.get(array, i), dimArray, dimIndex + 1, variantArray, variantIndex + dimArray[dimIndex + 1] *
i);
}
}
}

—
Reply to this email directly or view it on GitHub.

[charphi]

Maybe but I don't know how to write it since a SAFEARRAY is a multidimensional array and you get the number of dimensions at runtime.

[twall]

Could you show me an example of usage? Presumably you have to do something to the Object in order to figure out its dimensions and then convert to something else. Ideally you'd encapsulate that in a few utility functions, like Object[] getSingleDimensionArray() or Object[][] getTwoDimensionArray().

[matthiasblaesing]

I revisited the branch and updated it with more functionality. I also added an object-oriented wrapper/helper (I admit it is a thin blanket).

This request is primary directed to @charphi, please take a look at the branch and give it a spin. I think the test demonstrates, that the common cases are covered, but then more eyes are better.

[charphi]

Your wrapper seems ok but I was thinking about something more "higher level".

Here is an example of usage:

import com.sun.jna.platform.win32.COM.util.Factory;
import com.sun.jna.platform.win32.COM.util.IComEnum;
import com.sun.jna.platform.win32.COM.util.annotation.ComInterface;
import com.sun.jna.platform.win32.COM.util.annotation.ComMethod;
import com.sun.jna.platform.win32.COM.util.annotation.ComObject;
import com.sun.jna.platform.win32.COM.util.annotation.ComProperty;

/**
 *
 * @author Philippe Charles
 */
public final class Adodb {

    public static void main(String[] args) {
        Factory factory = new Factory();
        try (Connection conn = factory.createObject(Connection.class)) {
            conn.open("Provider=Search.CollatorDSO;Extended Properties='Application=Windows';");
            try (Recordset rs = conn.execute("SELECT TOP 5 System.ItemPathDisplay, System.ItemName, System.ItemUrl FROM SYSTEMINDEX ORDER BY System.ItemUrl")) {
                while (!rs.isEOF()) {
                    System.out.println(rs.getFields().getItem(0).getValue());
                    rs.moveNext();
                }
//            Object[][] obj = rs.getRows();
//            System.out.println(java.util.Arrays.deepToString(obj));
            }
        } finally {
            factory.disposeAll();
        }
    }

    private Adodb() {
        // static class
    }

    @ComObject(progId = "ADODB.Connection")
    public interface Connection extends AutoCloseable {

        @ComMethod
        void open(String connectionString);

        @Override
        @ComMethod
        void close();

        @ComMethod
        Recordset execute(String commandText);
    }

    @ComInterface
    public interface Recordset extends AutoCloseable {

        @Override
        @ComMethod
        void close();

        @ComProperty(name = "EOF")
        boolean isEOF();

        @ComMethod
        void moveNext();

        @ComProperty
        Fields getFields();

        @ComMethod
        Object[][] getRows();
//        Variant.VARIANT getRows();
    }

    @ComInterface
    public interface Fields {

        @ComProperty
        Number getCount();

        @ComProperty
        Field getItem(int index);

        @ComProperty
        Field getItem(String name);
    }

    @ComInterface
    public interface Field {

        @ComProperty
        String getName();

        @ComProperty
        DataTypeEnum getType();

        @ComProperty
        Object getValue();

        @ComProperty
        Object getPrecision();
    }

    public enum DataTypeEnum implements IComEnum {

        adEmpty(0),
        adSmallInt(2),
        adInteger(3),
        adSingle(4),
        adDouble(5),
        adCurrency(6),
        adDate(7),
        adBSTR(8),
        adIDispatch(9),
        adError(10),
        adBoolean(11),
        adVariant(12),
        adIUnknown(13),
        adDecimal(14),
        adTinyInt(16),
        adUnsignedTinyInt(17),
        adUnsignedSmallInt(18),
        adUnsignedInt(19),
        adBigInt(20),
        adUnsignedBigInt(21),
        adFileTime(64),
        adGUID(72),
        adBinary(128),
        adChar(129),
        adWChar(130),
        adNumeric(131),
        adUserDefined(132),
        adDBDate(133),
        adDBTime(134),
        adDBTimeStamp(135),
        adChapter(136),
        adPropVariant(138),
        adVarNumeric(139),
        adVarChar(200),
        adLongVarChar(201),
        adVarWChar(202),
        adLongVarWChar(203),
        adVarBinary(204),
        adLongVarBinary(205),
        adArray(0x2000);

        private final long value;

        private DataTypeEnum(long value) {
            this.value = value;
        }

        @Override
        public long getValue() {
            return value;
        }
    }
}

In order to return Object[][] from Recordset#getRows(), I have to add #522 (comment) to com.sun.jna.platform.win32.COM.util.Convert :

        } else if (vobj instanceof WTypes.BSTR) {
            return ((WTypes.BSTR) vobj).getValue();
        } else if (vobj instanceof OaIdl.SAFEARRAY) {
            return toJavaArray((OaIdl.SAFEARRAY) vobj);
        }
        return vobj;

[matthiasblaesing]

Ok - I just pushed a second iteration (rebased on current master):

• Improve Test as Demo (Bind return of GetRows directly as SAFEARRAY instead of VARIANT)
• Improve variable naming in Test
• Add minimal benchmark for access of SAFEARRAY
• Bind more methods
• move method binding into SAFEARRAY itself
• adjust test to correct CoInitialize/CoUninitialize handling
• added a helper to optimize SAFEARRAY -> Object[] conversion (see: com.sun.jna.platform.win32.OaIdlUtil.toPrimitiveArray(SAFEARRAY, boolean))
• modify indices to int values
• handle non-VARIANT arrays

The most relevant part is most probably the helper, for which an example can be found in com.sun.jna.platform.win32.SAFEARRAYTest, line 665.

[boyarintsev]

Hi guys,

@matthiasblaesing , @charphi , you've implemented an excellent tool, which is very easy to use. But I cannot find a way, how to make it work faster with big arrays (>100000 elements). It takes minutes to fetch elements. As I understand these methods fetch SafeArray elements one at a time, and of course every call adds an overhead.

What I do is very simple call to the Mapping class:

OaIdl.SAFEARRAY safeArray = (OaIdl.SAFEARRAY) table.GetSafeArray(); Variant.VARIANT resultV = (Variant.VARIANT) sa.getElement(pointer); - pointer is int[]

Looks like it goes to memory every time. Is there a way to import this array once into Java and then work with it in the same fashion. I'm pretty sure this would be a major performance improvement.

[matthiasblaesing]

In general, please use the jna-users forum/mailinglist/group:

https://groups.google.com/forum/#!forum/jna-users

As a first hint: Have a look at OaIdl#toPrimitiveArray. That routine uses SafeArrayAccessData to access the raw memory of the SAFEARRAY. This way you can read large arrays of values at once and pay the costs for the Java<->Native transition only once.

[oyvfos]

Hi, here is a quick hack to improive speed on large arrays (in this case omitting row and colum names) and fetching doubles:

SAFEARRAY sa= new SAFEARRAY(pbr3.getValue());
                int size1=sa.getUBound(0);
        int size2=sa.getUBound(1);
        byte[] bt = sa.pvData.getPointer().getByteArray(0,24*(size1+1)*(size2+1));
              int index=0;
        double[] res= new double[size1*size2] ;
        for(int j = 1; j < size1; j+=1) { //columns
        	int from= j*24*(size2+1);
        	int to = (j+1)*24*(size2+1);
        	for(int k =from +24; k < to; k+=24) {
        		byte[] temp = Arrays.copyOfRange(bt,k+8, k+16);
        		//System.out.println(ByteBuffer.wrap(temp).order(ByteOrder.LITTLE_ENDIAN).getDouble());
        		res[index]=ByteBuffer.wrap(temp).order(ByteOrder.LITTLE_ENDIAN).getDouble();
        		index+=1;
        	}
        	
        }

[matthiasblaesing]

@oyvfos thanks for giving this advise, some comments:

• For accessing the backing memory the access should be braced by SAFEARRAY#accessData and SAFEARRAY#unaccessData
• size1 and size2 are only correct for 0-based arrays (if they are not 0 based, getLBound needs to be consulted - have a look at this changeset: 9cf1ef1 where such a bug was fixed in OaIdlUtil#toPrimitiveArray
• This works because VARIANT has a size of 24 bytes and the structure holds the double data (8 bytes) at an offset of 8 bytes
• This will only work if the SAFEARRAY holds exclusively VARIANTS holding double data and will fail in other cases
• The code works only for a two-dimensional and flattens that to a one dimensional one
• If the array is large, it might by better to do a native read via Pointer#getDouble