[Feature Request] Salesforce Tap excludes incompatible fields when BULK API is selected.

[ns-iknox]

👋
Certain data types (location and address specifically) are not accessible via the Salesforce bulk API. It would be great if any custom fields which are of those types were excluded instead of the manually created blacklist currently implemented.

This can be easily done by hitting the salesforce REST API with a query something like this:

      SELECT DeveloperName
            FROM EntityParticle
            WHERE DataType IN ('address', 'location')
            AND EntityDefinitionId = '{object_name}'

An example implementation of this logic can be found here:
https://github.com/ns-iknox/Salesforce_OAuth_Toy

Reference about bulk api limitations: https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/compound_fields_address.htm

[jengle-ns]

For additional context...what we are seeing with Stitch is that custom location and address fields are not automatically blacklisted. When we ask for all the fields in a stream and the salesforce object contains custom or location or address fields, replication breaks until we exclude those fields.

[KAllan357]

Hello @ns-iknox and @jengle-ns. If I'm reading this correctly, I think we already have a portion of this implemented:

https://github.com/singer-io/tap-salesforce/blob/master/tap_salesforce/__init__.py#L167-L169

The above code is in the Discovery loop as the tap iterates the Salesforce objects. It sounds like what could be missing is logic for the when the type is a location ie: f['type'] == "location"

Under the conditions in this code, when the type is location and we're using the Bulk API, we should render "unsupported" selection metadata for the column.

Are you seeing "address" field are actually listed as supported? Or are we only just missing the location case?

[ns-iknox]

Hi @KAllan357! Thanks for your quick response!
You're totally right about location, but there is an issue with addresses as well. I think the issue is that the API call used for field data type discovery doesn't return a type for custom address fields.

I'll run a quick test to double check that and provide results here.

[ns-iknox]

Well, I stand corrected @KAllan357 . It turns out it's just the location data types that have been giving us issues, sorry for the confusion.

Since that's an incredibly simple fix I went ahead and made you a PR.

[jengle-ns]

@KAllan357 @cmerrick just wanted to ping on this. there is a pretty small PR to fix this issue so wanted to see what a timeline would be to get it pushed in.

[ps2goat]

It looks like that PR was merged-- Can this issue be closed?