Why does @model build a declaration instead of a value?

[cscherrer]

Hello Turing,

I'm curious about the design decision to make @model return a declaration instead of a value. For example, turing.ml has this example:

@model gdemo(x, y) = begin
  # Assumptions
  σ ~ InverseGamma(2,3)
  μ ~ Normal(0,sqrt(σ))
  # Observations
  x ~ Normal(μ, sqrt(σ))
  y ~ Normal(μ, sqrt(σ))
end

In Soss.jl I would write this as

gdemo = @model (x, y) begin
  # Assumptions
  σ ~ InverseGamma(2,3)
  μ ~ Normal(0,sqrt(σ))
  # Observations
  x ~ Normal(μ, sqrt(σ))
  y ~ Normal(μ, sqrt(σ))
end

We can think of Soss's @model as being "anonymous", similar to an anonymous function. I see a few advantages to the latter approach:

• It's easy to add a name with myModel = @model...
• Models can be inlined. For example, in the Soss counterpart to sample(gdemo(1.5, 2), SMC(1000)), gdemo could be replaced with @model... code
• It's very natural in a context where models are "first class". One of the goals of Soss is to allow swapping out, @model... in place of (for example) Normal()

This makes me wonder, was having @model be a declaration a conscious design choice, or was the alternative not discussed? What benefits do you see from this approach? Is there any possibility of this changing at some point, or has it become too strong a dependency?

[mohamed82008]

Hi Chad,

Thanks for the issue. In Turing, defining @model ... gives us a model generator function which generates a concrete Turing.Model for each set of input data. By concrete, I mean that we can sample from it using the sample function, missing data have been handled properly, etc. Currently, in Turing only the syntax:

@model f(x, y) = begin
    ....
end

or the less common:

@model function f(x,y)
    ...
end

are supported. In fact any named function definition syntax is supported. It should also be trivial to support the syntax:

@model (x, y) -> begin
    ....
end

or even the Soss syntax:

@model (x, y) begin
   ....
end

in the future. It is just a matter of assigning some gensymed name to the anonymous function, and then calling the other function-based macro to do the heavy lifting. Note that currently nothing stops you from doing:

g = @model f(x, y) = begin
    ...
end

where g will be alias for f, so if the anonymous syntax is supported you can drop f from the above which will look similar to Soss. But note that whatever syntax is supported is orthogonal to the requirement that the output of @model will be a function that generates Turing.Models for each set of data inputs. This is necessary to handle missing data which are sampled from prior. Try defining the model with some data inputs left out, you will see these input variables are treated as parameters, using Turing.pvars(model) and Turing.dvars(model).

Regarding inlining, if by inlining, you mean inlining the Turing.Model generation code, i.e. gdemo(1.5, 2), this is called only once so it is not performance-critical. If by inlining you mean, inlining the call to the concrete Turing.Model which returns the logp and does a few other things, then this can be done easily by just using @inline in a few places:

1. All the inner_function_name definitions starting https://github.com/TuringLang/Turing.jl/blob/master/src/core/compiler.jl#L352, and
2. The callable model calling in https://github.com/TuringLang/Turing.jl/blob/master/src/Turing.jl#L57

Note that we can sprinkle @inline in a few other places to eliminate all function call overhead. This is an optimization that can be easily done and benchmarked, but it hasn't been a priority so far, as we are still trying to eliminate type instabilities which are more serious performance issues. #660 and #665 promise a significant speedup by removing almost all type instabilities in performance critical parts of Turing.

Regarding first class models and defining things like rand and logpdf for Turing.Models, this has been discussed in #634 . The mechanics of how it can be done still need to be figured out though. One way that has been discussed is to have a lazy equivalent of the sample function where parameters are sampled on demand using an iterator. The rand function can therefore just call iterate on the lazy sample iterator. Note that we need an inference algorithm to be attached to the model to be able to treat it as a distribution-like object. So using Turing.Model directly is not really possible since Turing.Model knows nothing about the inference algorithm it will be used with. This may also change in the future!

[mohamed82008]

I will close this issue for now. As for using models as drop-in for distributions, I think a separate API can be used to create a distribution from the model generator. I will think more about this after I move the Turing compiler to DynamicPPL and try to make Turing and Soss interoperable.