trained recurrent model can't be saved in BSON

[babaq]

hi,
I just found my model can't be saved in BSON.

using Flux,BSON
model = Chain(Dense(12,12),LSTM(12,12),Dense(12,5))
Flux.@epochs 4 Flux.train!((x,y)->Flux.crossentropy(softmax(model(x)), y),[(rand(12,30),rand(5,30))],SGD(params(model)))

BSON.@save "m.bson" model

UndefRefError: access to undefined reference

Stacktrace:
 [1] structdata(::TrackedArray{…,Array{Float64,2}}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\extensions.jl:80
 [2] lower at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\extensions.jl:84 [inlined]
 [3] _lower_recursive(::TrackedArray{…,Array{Float64,2}}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [4] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::TrackedArray{…,Array{Float64,2}}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [5] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:28
 [6] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [7] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [8] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:21
 [9] _lower_recursive(::Tuple{TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,2}}}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [10] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Tuple{TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,2}}}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [11] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:28
 [12] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [13] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [14] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:21
 [15] _lower_recursive(::Flux.Recur{Flux.LSTMCell{TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [16] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Flux.Recur{Flux.LSTMCell{TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [17] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:28
 [18] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [19] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [20] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:28
 [21] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [22] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [23] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:21
 [24] _lower_recursive(::Chain, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [25] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Chain) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [26] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\BSON.jl:21
 [27] _lower_recursive(::Dict{Symbol,Chain}, ::IdDict{Any,Any}, ::Array{Any,1}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:62
 [28] lower_recursive(::Dict{Symbol,Chain}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:73
 [29] bson(::IOStream, ::Dict{Symbol,Chain}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:81
 [30] #14 at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:83 [inlined]
 [31] #open#294(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Chain}}, ::String, ::Vararg{String,N} where N) at .\iostream.jl:369
 [32] open at .\iostream.jl:367 [inlined]
 [33] bson(::String, ::Dict{Symbol,Chain}) at C:\Users\Alex\.julia\packages\BSON\kxdIr\src\write.jl:83
 [34] top-level scope at In[22]:1

I am using Julia 1.0.3 on Windows 10 with Flux 0.6.10 and BSON 0.2.1.

Interestingly, if I do not train the model, or do not use recurrent LSTM layer, or do

Flux.truncate!(model)

it all worked.

[asbisen]

I am experiencing same issue ... using LSTM toy model.

[babaq]

      I am experiencing same issue ... using LSTM toy model.

I switched to JLD2, it worked well.

[appleparan]

I'm Julia 1.1, Flux 0.8.2, BSON 0.2.3 on Ubuntu 18.04, This works.

Anway, In my code, I don't use RNN. I use only Dense Layers but size is quite larger than this code

model = Chain(
        Dense(input_size, 100),
        Dropout(0.2),

        Dense(100, 100),
        Dropout(0.2),

        Dense(100, 100),
        Dropout(0.2),

        Dense(100, output_size),
        
        softmax,
    ) |> gpu

input_size is 792 and output_size is 24.

However, I got similar message with StackOverflow (I used cpu(model) to save model)

ERROR: LoadError: StackOverflowError:
Stacktrace:
 [1] ==(::Type, ::Type) at ./operators.jl:155
 [2] fieldcount(::Any) at ./reflection.jl:641
 [3] isprimitive(::Type) at /home/appleparan/.julia/packages/BSON/XPZLD/src/extensions.jl:80
 [4] structdata(::Function) at /home/appleparan/.julia/packages/BSON/XPZLD/src/extensions.jl:82
 [5] lower(::Function) at /home/appleparan/.julia/packages/BSON/XPZLD/src/extensions.jl:86
 [6] _lower at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:53 [inlined]
 [7] _lower_recursive(::Function, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:54
 [8] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Function) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:62
 [9] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Array{Any,1}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/BSON.jl:28
 [10] _lower_recursive(::Array{Any,1}, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:62
 [11] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Array{Any,1}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:62
 [12] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/BSON.jl:21
 [13] _lower_recursive(::Tracker.Call{getfield(Tracker, Symbol("##248#251")),Tuple{Tracker.Tracked{Float64},Tracker.Tracked{Float32}}}, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:62
 ... (the last 6 lines are repeated 6875 more times)
 [41264] (::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}})(::Tracker.TrackedReal{Float64}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:62
 [41265] applychildren!(::getfield(BSON, Symbol("##7#11")){IdDict{Any,Any},Array{Any,1}}, ::Dict{Symbol,Any}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/BSON.jl:21
 [41266] _lower_recursive(::Dict{Symbol,Any}, ::IdDict{Any,Any}, ::Array{Any,1}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:62
 [41267] lower_recursive(::Dict{Symbol,Any}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:73
 [41268] bson(::IOStream, ::Dict{Symbol,Any}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:81
 [41269] #14 at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:83 [inlined]
 [41270] #open#310(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(BSON, Symbol("##14#15")){Dict{Symbol,Any}}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
 [41271] open at ./iostream.jl:367 [inlined]
 [41272] bson(::String, ::Dict{Symbol,Any}) at /home/appleparan/.julia/packages/BSON/XPZLD/src/write.jl:83
 [41273] train!(::Flux.Chain{Tuple{Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},typeof(NNlib.softmax)}}, ::Array{Tuple{CuArrays.CuArray{Float32,1},CuArrays.CuArray{Float32,1}},1}, ::Array{Tuple{CuArrays.CuArray{Float32,1},CuArrays.CuArray{Float32,1}},1}, ::Function, ::getfield(MLToys, Symbol("#accuracy#10")){Flux.Chain{Tuple{Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},Flux.Dense{typeof(identity),TrackedArray{…,CuArrays.CuArray{Float32,2}},TrackedArray{…,CuArrays.CuArray{Float32,1}}},Flux.Dropout{Float64},typeof(NNlib.softmax)}}}, ::Flux.Optimise.ADAM, ::Int64, ::String) at /home/appleparan/src/MLToys.jl/src/jongro01_DNN/model.jl:86

EDIT: My problem was TrackedReal. I don't know why but I can't write TrackedReal type in .bson (or other format such as JLD, JLD2). I convert TrackedReal to Float64 by data() then have no problem

EDIT2: I measured accuracy of model as custom accuracy function inspired by evaluateAccuracy (https://github.com/FluxML/model-zoo/blob/0d3bae170638a3398a848712b7944623c15b516f/audio/speech-blstm/01-speech-blstm.jl#L105)

function RSME(dataset, model)
    # RSME
    acc = 0.0

    for (x, y) in dataset
        ŷ = model(x)
        @assert size(ŷ) == size(y)

        acc += sqrt(sum(abs2.(ŷ .- y)) / length(y))
    end

    acc / length(dataset)
end

this returned TrackedReal, but it is somewhat incomplete. This is the reason causing same error above. I didn't used RNN so not to choose to use Tracker.reset. If you need Real, just wrap it Flux.Tracker.data.

    Flux.Tracker.data(acc / length(dataset))

[appleparan]

This is not Flux.jl's problem (user problem), it should be closed.

[otobrzo]

I take an example directly from flux saving in Flux

using Flux

model = Chain(Dense(10,5,relu),Dense(5,2),softmax)
Chain(Dense(10, 5, NNlib.relu), Dense(5, 2), NNlib.softmax)

weights = params(model);


using BSON: @save

@save "mymodel.bson" model
@save "mymodel.bson" weights

And I got this error:

julia> @save "mymodel.bson" weights
ERROR: UndefRefError: access to undefined reference

This is connected to this error of BSON.jl:

error

[CarloLucibello]

this should be fixed by JuliaIO/BSON.jl#70