RFC: use GPUArrays, also for testing

[mcabbott]

This PR does two things.

First it replaces CUDA with GPUArrays, mostly to save startup time:

julia> @time using CUDA
  3.515312 seconds (9.57 M allocations: 654.933 MiB, 2.48% gc time, 16.35% compilation time)
  2.117964 seconds (4.08 M allocations: 328.683 MiB, 24.88% compilation time)  # after using GPUArrays

julia> @time using GPUArrays
  1.323729 seconds (5.49 M allocations: 326.357 MiB, 4.41% compilation time)  # on a fresh start
  0.641342 seconds (1.05 M allocations: 54.070 MiB, 5.79% gc time, 99.95% compilation time)  # after using CUDA

Second, it wants to run tests using a fake GPUArray when CUDA isn't around. I think this ought to catch many scalar indexing errors. Not entirely sure how official this JLArray code is, we could check. The actual tests are from Flux, minimally adapted.

Both of these might be worth doing more widely. But this little package (which I only saw today) might be an easy place to start. Thoughts?

[ToucheSir]

Is this correct? The number of dimensions for T would be <N.

[mcabbott]

After some confusion: The N here is what's called M or N+1 elsewhere in this file. The 4th type parameter, not the 3rd. So this line does change the result, although maybe it's not tested?

Is there a way around that?

One hack I can think of is relying on all ArrayStyles to be like CuArrayStyle{N} and have their N first. It seems pretty unlikely that any AbstractGPUArray is going to deviate from that pattern.

[ToucheSir]

That or some hack to swap out the ndims of the index array type to N. Then forwarding to BroadcastStyle would just do the right thing. I'm not sure which is more stable.

[mcabbott]

See what you think of ffb6f00

[ToucheSir]

Last I checked, JLArray was not terribly comprehensive and didn't fulfill the testing requirements for the PR I was considering it for. That said, it may be better than nothing.

[mcabbott]

Ok, I have not run into these limitations yet.

It does appear to catch some problems here, e.g. FluxML/Flux.jl#1905 .

It also causes some problems, as Zygote has special paths for CuArray not AbstractGPUArray. But perhaps that should be changed.

[ToucheSir]

LGTM, let's get GPU CI set up next

[mcabbott]

I don't have a merge button here, BTW.

[ToucheSir]

Looks like the default permissions are pretty strict, you should now :)