Use arrow keys to move the paddle
\ No newline at end of file diff --git a/_includes/experiments/game/instructions/chess.html b/_includes/experiments/game/instructions/chess.html deleted file mode 100644 index 4f40057b..00000000 --- a/_includes/experiments/game/instructions/chess.html +++ /dev/null @@ -1,11 +0,0 @@ -Human plays white, computer plays black. Drag and drop to move pieces. Standard rules apply
-Human plays black, computer plays white. Two consecutive passes result in a game over. The best 3 moves found are circled in green. Illegal moves are shown in red
- -Use arrow keys to move the paddle
\ No newline at end of file diff --git a/_layout/foot.html b/_layout/foot.html index 4d252996..e01cb41c 100644 --- a/_layout/foot.html +++ b/_layout/foot.html @@ -1,4 +1,9 @@ - + + + {{isnotpage index}} + + + {{end}}Thanks to Hannah Lepper and James Bradbury for feedback on this post.
+~~~ diff --git a/blogposts/2019-03-05-dp-vs-rl.md b/blogposts/2019-03-05-dp-vs-rl.md new file mode 100755 index 00000000..4491d292 --- /dev/null +++ b/blogposts/2019-03-05-dp-vs-rl.md @@ -0,0 +1,104 @@ ++++ +title = "Differentiable Control Problems" +published = "5 March 2019" +author = "Mike Innes, Neethu Maria Joy, Tejan Karmali" ++++ + +We've discussed the idea of [differentiable programming](https://fluxml.ai/2019/02/07/what-is-differentiable-programming.html), where we incorporate existing programs into deep learning models. This article shows what ∂P can bring to some simple but classic control problems, where we would normally use black-box Reinforcement Learning (RL). ∂P-based models not only learn far more effective control strategies, but also train orders of magnitude faster. The [code](https://github.com/FluxML/model-zoo/tree/master/contrib/games/differentiable-programming/trebuchet) is all available to run for yourself – they will mostly train in a few seconds on any laptop. + +## Follow the Gradient + +Differentiation is what makes deep learning tick; given a function $y = f(x)$ we use the gradient $\frac{dy}{dx}$ to figure out how a change in $x$ will affect $y$. Despite the mathematical clothing, gradients are actually a very general and intuitive concept. Forget the formulas you had to stare at in school; let's do something more fun, like throwing stuff. + + + +When we throw things with a trebuchet, our $x$ represents a setting (say, the size of the counterweight, or the angle of release), and $y$ is the distance the projectile travels before landing. If you're trying to aim, the gradient tells you something very useful – whether a change in aim will increase or decrease the distance. To maximise distance, just follow the gradient. + +Alright, but how can we get this magical number? The trick is a process called [algorithmic differentiation](https://github.com/FluxML/Zygote.jl), which is able to differentiate not only simple formulas like you learned in school, but also _programs_ of any complexity – like our trebuchet simulator. The upshot is that we can take a [simple simulator](https://github.com/Roboneet/Trebuchet.jl), written in Julia and [DiffEq](https://github.com/JuliaDiffEq/DifferentialEquations.jl) with no deep learning in mind, and get gradients for it in a single function call. + +```julia +# what you did in school +gradient(x -> 3x^2 + 2x + 1, 5) # (32,) +# something a little more advanced +gradient((wind, angle, weight) -> Trebuchet.shoot(wind, angle, weight), + -2, 45, 200) # (4.02, -0.99, 0.051) +``` + +Now we have that, let's do something interesting with it. + + +## Throwing Stuff + +A simple way to use this is to aim the trebuchet at a target, using gradients to fine-tune the angle of release; this kind of thing is common under the name of _parameter estimation_, and we've [covered examples like it before](https://julialang.org/blog/2019/01/fluxdiffeq). We can make things more interesting by going meta: instead of aiming the trebuchet given a single target, we'll optimise a neural network that can aim it given _any_ target. Here's how it works: the neural net takes two inputs, the target distance in metres and the current wind speed. The network spits out trebuchet settings (the mass of the counterweight and the angle of release) that get fed into the simulator, which calculates the achieved distance. We then compare to our target, and _backpropagate through the entire chain_, end to end, to adjust the weights of the network. Our "dataset" is a randomly chosen set of targets and wind speeds. + + + +A nice property of this simple model is that training it is _fast_, because we've expressed exactly what we want from the model in a fully differentiable way. Initially it looks like this: + + + +After about five minutes of training (on a single core of my laptop's CPU), it looks like this: + + + +If you want to try pushing it, turn up the wind speed: + + + +It's only off by 16cm, or about 0.3%. + +What about just aiming the trebuchet directly? It is easy to do this by gradient descent given that we have gradients. However, this is a slow iterative process that takes around 100ms each time. In contrast, running the neural network takes 5μs (twenty thousand times faster) with only a small loss in accuracy. This "approximate function inversion via gradients" trick is a very general one that can not only be used with dynamical systems, but also lies behind the [fast style transfer](https://github.com/lengstrom/fast-style-transfer) algorithm. + +This is about the simplest possible control problem, which we use mainly for illustrative purposes. But we can apply the same techniques, in more advanced ways, to classic RL problems as well. + +## Cart, meet Pole + +A more recognisable control problem is [CartPole](https://gym.openai.com/envs/CartPole-v0/), the "hello world" for reinforcement learning. The task is to learn to balance an upright pole by nudging its base left or right. Our setup is broadly similar to the trebuchet case: a [Julia implementation](https://github.com/tejank10/Gym.jl) means we can directly treat the reward produced by the environment as a loss. ∂P allows us to switch seamlessly from model-free to model-based RL. + + + +The astute reader may notice a snag. The action space for cartpole – nudge left or right – is discrete, and therefore not differentiable. We solve this by introducing a _differentiable discretisation_, defined [like so](https://github.com/FluxML/model-zoo/blob/cdda5cad3e87b216fa67069a5ca84a3016f2a604/games/differentiable-programming/cartpole/DiffRL.jl#L32): + +$$ +\begin{aligned} + f(x) &= + \begin{cases} + 1 & x \ge 0 \\\ + -1 & x < 0 + \end{cases} \\\ + \frac{df}{dx} &= 1 +\end{aligned} +$$ + +In other words, we force the gradient to behave as if $f$ were the identity function. Given how much the mathematical idea of differentiability already gets abused in ML, it's perhaps not surprising that we can just cheat here; for training all we need is a signal to inform our pseudo-random walk around parameter space, and the rest is details. + +The results speak for themselves. Where RL methods need to train for hundreds of episodes before solving the problem, the ∂P model only needs around 5 episodes to win conclusively. + + + + +## The Pendulum & Backprop through Time + +An important aim for RL is to handle _delayed reward_, when an action doesn't help us until several steps in the future. ∂P allows this too, and in a very familiar way: when the environment is differentiable, we can actually train the agent using backpropagation through time, just like a recurrent net! In this case the environmental state becomes the "hidden state" that changes between time steps. + + + +To demonstrate this technique we looked at the [pendulum](https://github.com/openai/gym/wiki/Pendulum-v0) environment, where the task is to swing a pendulum until it stands upright, keeping it balanced with minimal effort. This is hard for RL models; after around 20 episodes of training the problem is solved, but often the route to a solution is visibly sub-optimal. In contrast, BPTT can beat the [RL leaderboard](https://github.com/openai/gym/wiki/Leaderboard#pendulum-v0) in _a single episode of training_. It's instructive to actually watch this episode unfold; at the beginning of the recording the strategy is random, and the model improves over time. The pace of learning is almost alarming. + + + +Despite only experiencing a single episode, the model generalises well to handle any initial angle, and has something pretty close to the optimal strategy. When restarted the model looks more like this. + + + +This is just the beginning; we'll get the real wins applying DP to environments that are too hard for RL to work with at all, where rich simulations and models already exist (as in much of engineering and the sciences), and where interpretability is an important factor (as in medicine). + +## The Map Is Not The Territory + +The limitation of these toy models is that they equate the simulated training environment with the test environment; of course, the real world is not differentiable. In a more realistic model the simulation gives us a coarse outline of behaviour that is refined with data. That data informs (say) the simulated effect of wind, in turn improving the quality of gradients the simulator passes to the controller. Models can even form part of a controller's forward pass, enabling it to refine its predictions without having to learn system dynamics from scratch. Exploring these new architectures will make for exciting future work. + +## Coda + +The core idea is that _differentiable programming_, where we simply write an arbitrary numerical program and optimise it via gradients, is a powerful way to come up with better deep learning-like models and architectures – especially when we have a large library of differentiable code to hand. The toy models described are really only previews, but we hope they give an intuition for how these ideas can be applied in more realistic ways. + +Just as functional programming involves reasoning about and expressing algorithms using functional patterns, differentiable programming involves expressing algorithms using differentiable patterns. Many such design patterns have already been developed by the deep learning community, such as for handling control problems or sequence and tree structured data. As the field matures, many more will be invented, and the resulting programs are likely to make even the most advanced current deep learning architectures look crude by comparison. diff --git a/blogposts/2019-05-30-JSoC-Cohort.md b/blogposts/2019-05-30-JSoC-Cohort.md new file mode 100755 index 00000000..93c1e2d9 --- /dev/null +++ b/blogposts/2019-05-30-JSoC-Cohort.md @@ -0,0 +1,56 @@ ++++ +title = "JSoC Cohort" +published = "30 June 2019" +author = "Tejan Karmali, Shreyas Kowshik, Kartikey Gupta, Manjunath Bhat, Avik Pal, Raghvendra Gupta" ++++ + +We are excited to introduce our JSoC students accepted this year! Flux is hosting 6 JSoC students this time. They are a mix of students funded by [Google Summer of Code](https://summerofcode.withgoogle.com/) and Julia Season of Contributions, which is run by the community, thanks to the generous support from [NumFOCUS](https://numfocus.org/). Large parts of Flux ecosystem were created or improved by the students accepted through JSoC, many of whom have gone on to become long time contributors to FluxML. This year, we are proud to have students who will be working on a variety of sub-domains of Machine Learning, ranging from Differentiable Programming and Reinforcement Learning to GAN and Ray Tracing; to demonstrate a fresh approach to these paradigms using Julia. + + +## Shreyas Kowshik: _Addition Of Baseline Models To Flux.jl_ + +The current state of the art algorithms in terms of reinforcement learning and generative models are not yet available in [Flux](https://github.com/FluxML). This project aims to add the following models with explicit documentation in the form of [blog](https://shreyas-kowshik.github.io/) and code : + +* CycleGAN +* pix2pix +* Proximal Policy Optimization +* Trust Region Policy Optimization +* Neural Image Captioning +* Deep Recurrent Q Networks +* Super Resolution GAN + +## Kartikey Gupta: _Reinforcement Learning Environments for Julia_ + +This project aims at providing a collection of Reinforcement Learning environments, to facilitate research and experimentation in RL, in spirit of the OpenAI Gym. This includes the classic control environments, algorithmic environments, toy text examples, 2D game environments, Atari 2600 based game environments and a major part of the MuJoCo based robotic environments, along with their documentation and working examples. A differentiable NES (Nintendo Entertainment System) emulator will also added and available to this collection. Progress can be tracked [here](https://github.com/FluxML/Gym.jl). +[Link to Project Blog](https://nextjournal.com/kraftpunk97) + +## Manjunath Bhat: _Enriching Model Zoo with Deep Learning Models_ + +The aim of this project is to enrich Flux Model Zoo with unsupervised deep learning models, in particular variants of Generative Adversarial Networks. This project proposes to add the following models: +* Spatial Transformer Networks +* StarGAN for facial expression synthesis +* VAE-GAN +* Energy Based GAN +* Gated Recurrent Convolutional Neural Network + +[Link to Project Blog](https://medium.com/@manjunathbhat9920) + +## Raghvendra Gupta: _Sparsifying Neural Networks using Sensitivity Driven Regularization_ + +This project aims to quantify the output sensitivity to the parameters i.e. their relevance to network output and introduce a regularization term that gradually lowers the absolute value of sub-sensitive parameters. Thus a very large fraction of parameters approach zero and are eventually set to zero by simple thresholding. The models on which the sparsification experiment would be carried out are: +* VGG16 +* VGG19 +* MobileNet +* ResNet +* RL models (extended goal) + +The success of the experiment would be measured using Memory Footprint of the parameters, Compression Ratio and Runtime Performance of the models. +[Link to Project Blog](https://medium.com/@raghav090897) + +## Avik Pal: _Differentiable Ray Tracing_ + +Ray Tracing is a rendering technique which allows us to create photorealistic images given the scene information. However, at times given an image, we want to infer the state of the scene, i.e., the light sources, materials of the objects, the orientation of the camera, etc. Being able to incorporate the ability to differentiate through the ray tracer allows us to calculate gradients wrt arbitrary scene parameters and in turn, optimize them to obtain their exact values. Apart from this, the ray tracer can be used to train a self-driving car using BPTT (Back Propagation through Time). The package is available at [here](https://github.com/avik-pal/RayTracer.jl). + +## Tejan Karmali: _Differentiable Duckietown_ + +[Duckietown](https://www.duckietown.org/) is an environment for autonomous vehicles research. It is a miniature town through which a car (or bot) needs to be navigated through. The town has multiple maps, which are different tasks the bot has to perform. The maps vary from a simple straight road to full fledged town. Over the summer I’ll be building this environment in julia, integrating it with the differentiable ray tracer to render the scene, and training the the bot for different tasks mentioned above. The package and progress can be tracked [here](https://github.com/tejank10/Duckietown.jl) diff --git a/blogposts/2019-09-11-simulating-the-motion-of-charges.md b/blogposts/2019-09-11-simulating-the-motion-of-charges.md new file mode 100755 index 00000000..1e719fe0 --- /dev/null +++ b/blogposts/2019-09-11-simulating-the-motion-of-charges.md @@ -0,0 +1,171 @@ ++++ +title = "Simulating the Motion of Charged Bodies" +published = "11 November 2019" +author = "Sudhanshu Agrawal sudhanshuagr27@ucla.edu" ++++ + +> “We demand rigidly defined areas of doubt and uncertainty!” – Douglas Adams, The Hitchhiker's Guide To The Galaxy + +## What do charges like doing? + +The answer is quite simple: like charges like to move apart and unlike charges like to move closer. If I were to place a 100 charges in a circle of radius 5cm and tell them to do their thing they would contentedly move around so that they attain the most stable state – where the total potential energy of the system is minimum. The problem is that even though charges know what their purpose is in life, and we know more or less what they’d tend to do, it’s quite difficult to predict their exact movements. Most of us can predict the movement of 3 or 4 charged bodies, but when it comes to accounting for the interactions of a 100 charges, the teachings of H.C. Verma and Resnick Halliday aren’t enough to help us. + +## The Main Idea + +What is the objective of my system of 100 charges? To minimise the total value of this potential energy. (Coincidentally this is the objective of the entire universe as well) +What is the objective of a typical machine learning model? To minimise the total cost/loss. Hmm… + +I’m going to see if I can predict the behaviour of a system of a 100 charges by differentiating the total potential energy and performing gradient descent on the x and y coordinates of each charge. What should happen is that the x and y coordinates start changing themselves to minimise the total potential energy. Therefore, the charges start moving to minimise the total potential energy, mimicking what would happen in the real world. + +## The Obvious Approach vs My Approach + +### The Obvious Approach + +The solution to this problem that immediately comes to mind is to use the amazing computing power we possess to resolve the vector components of the 99 forces on each of the 100 charges we have, use that net force to predict the acceleration of the charges, use that acceleration to produce movement in the charges, and repeat the process within a very short time interval to produce as continuous and as accurate a movement as possible. + +### My Approach + +Why do charges move? Due to acceleration. +Where does acceleration come from? Force. +Why is force applied? To lower potential energy. +Therefore, it’s potential energy that’s really pulling the strings. +What I’ve done is cut out the middleman that is force. I’ve changed my objective from simulating the effect that force has on charged bodies to directly simulating the effect that potential energy has on it. +(This may seem like the same thing, and the fact that it does is why I can do it) +The main advantage to doing so is that potential energy is a scalar quantity whereas force is a vector. Calculating the net potential energy of a system of 100 charges is a lot easier than calculating the net force on each charge in a 100-charge system. + +## The Science Part of It + +Let’s look at our classic formula to calculate potential energy + +$$U = \frac{1}{4 \pi \epsilon_0}\frac{q_1q_2}{r}$$ + +It quantifies the potential energy $U$ of a system of two point-charges with magnitudes q1 and q2, separated by a distance r in free space (vacuum). + +If we want the total potential energy of a system of $n$ charges, we just apply this formula $^nC_2$ times. + +## Let's Begin + +I’m going to create a system of 100 charged bodies and initialise them with random charge values. I’m sampling from a Gaussian Distribution since I’d like there to be a few slightly larger charges in the mix to make things interesting. + +```julia +struct Vertex # represents the charged body + charge::Float64 + pos::Vector{Float64} # x,y coordinates +end + +# Create the req no. of charges with random positions and amplitudes +function create_env(no_ch) + [Vertex(randn(), rand(-5:0.001:5, 2)) for _ in 1:no_ch] +end +``` + +I’m also going to place these charges randomly in a circle of radius 5cm centred at the origin. + +Later on, I’ll make sure that these charges never leave this circle. Why is this important? If I happen to get a couple of large charges with the same sign and these charges were allowed to go anywhere they wanted to, they would simply move apart to infinity, dragging all the smaller charges with them, and we wouldn’t get to see much. + +I’m also going to create a simple function to calculate potential. + +```julia +function potential(a::Vertex, b::Vertex) # classic potential formula + return (8.9e9)*a.charge*b.charge/(sqrt(sum(abs2.(a.pos .- b.pos)))) +end + +function potential(a::Vector) # potential for a system of charges + pot = 0 + for i in 1:length(a) # nC2 + for j in i+1:length(a) + pot = pot+potential(a[i],a[j]) + end + end + return pot +end +``` + +Now I’m going to define my loss function. Why can’t I just use my potential function? The reason is that I want to constrain my charges to a circle. The way I do that is by applying a high penalty(loss) every time a charge tries to move out of the circle. + +```julia +function loss(env::Vector) + s = 0 + for i in 1: length(env) + # if charge moves outside circle of radius 5 centred at origin, + # give it a high penalty + # this is to constrain it otherwise large like charges would + # just go to infinity and we wouldn't see much + if distance(env[i], [0,0])-5>0 + s = s+100*(distance(env[i],[0,0])-5) + end + end + # scaling potential so the distance moved is acceptably small + l = s + 1.0e-9*potential(env) + return l +end +``` + +Since the potential function gives a very high value of the order of 109 we need to scale it so that the loss is a low enough value to produce an acceptable movement in my charges. + +Let’s create the environment of charges + +```julia +env = create_env(100) +``` + +I’m using 2 main machine learning libraries. Zygote, for its AD capabilities, and Flux, for its optimisers. + +```julia +using Zygote +using Flux +``` + +Let’s start moving these charges around + +```julia +opt = ADAM(0.01) +for i in 1:200 + gs = Zygote.gradient(loss, env) + for j in 1: length(gs[1]) # each j is a charge + # making sure we update only the positions not the charge value + Flux.Optimise.update!(opt, env[j].pos, gs[1][j].pos) + end +end +``` + +Using some basic plotting functions, I graphed my charges at every training iteration and put them all together in a gif. + + + +*Red charges are positive, Blue charges are negative and size of the dot is proportional to the absolute value of charge.* + +## The Big Question. It looks cool, but does it work? + +A high-level observation shows that the like charges tend to move apart and the unlike charges tend to move closer. We can take a step further and notice that as time progresses, the charges tend to move less and less since they are more or less in a stable position. A sure sign of convergence. + +Let’s prove this. + +Let’s take a trivial case and define a system of two charges: one positive and one negative. Let’s place them diametrically opposite with respect to the origin. + + + +What do we expect to happen? The charges should move together. + + + +And they do! + +_(What’s interesting is that they seem to have overshot and actually crossed each other at one point, only to get drawn to each other once more. An apt parallel is charges overshooting due to inertia of motion)_ + +Let’s go back to that system of 100 charges and plot the potential energy at every training iteration. + + +It’s safe to say that the system has converged at a value of approximately -1360 . + +You’re probably wondering why the graph is so bumpy and what these extreme minimas are. The minimas are simply the points in the training loop when the charges got a bit overconfident and tried to escape the constraining circle. They are then immediately brought back to valid positions by the loss function. +They may also occur when a positive and a negative charge briefly overlap with each other giving a potential of negative infinity. Of course, such a situation is impossible in the real world. + +## Conclusion + +It works! +I created this simulation because it’s an interesting and very visual way to get a real feel of how charges move around when affected by almost a 100 different forces. +It also shows how it’s possible to solve a problem more easily by reframing the question being asked. +~~Simulate the effect of force on charged bodies?~~ +Simulate the effect of potential energy on charged bodies? +Most of all it shows the power of gradient descent in solving a problem that would be unwieldy to solve using classic approaches. diff --git a/blogposts/2020-06-29-acclerating-flux-torch.md b/blogposts/2020-06-29-acclerating-flux-torch.md new file mode 100755 index 00000000..28c3d432 --- /dev/null +++ b/blogposts/2020-06-29-acclerating-flux-torch.md @@ -0,0 +1,140 @@ ++++ +title = "Accelerating Flux.jl with PyTorch kernels" +published = "29 July 2020" +author = "Dhairya Gandhi, Mike Innes" ++++ + +Julia and Flux provide a flexible [differentiable programming](./2019-03-05-dp-vs-rl.md) system. Flux does not trade flexibility and abstraction for performance, and in fact strives to achieve both in the same package. For example, Flux is able to target multiple hardware accelerators such as [GPUs](https://fluxml.ai/Flux.jl/stable/gpu/) and [TPUs](https://arxiv.org/pdf/1810.09868.pdf). As a result, it is one of the pillars of Julia's [deep learning ecosystem](https://juliahub.com/ui/Packages/CUDAnative/4Zu2W/3.1.0?t=2), with almost 40 packages leveraging it. + +Here, we introduce [Torch.jl](https://github.com/FluxML/Torch.jl), a package that wraps optimised kernels from [PyTorch](https://pytorch.org). Even though Julia's [GPU compiler](https://developer.nvidia.com/blog/gpu-computing-julia-programming-language/) is already pretty good for [general use](https://juliahub.com/ui/Packages/CUDAnative/4Zu2W/3.1.0?t=2) and under [heavy development](https://github.com/JuliaGPU/CUDA.jl), we provide Torch.jl to leverage well-debugged high performance kernels that have been built by the PyTorch community, much in the same way we use BLAS and LAPACK for Linear Algebra. + +For popular object detection models - ResNet50, ResNet101 and VGG19 - we compare inference times for Flux using Torch.jl with our native tooling, and find Flux+Torch to be 2-3x faster. On larger batch sizes, the difference is much higher, since Julia's GPU stack needs further development in the area of memory management and GC. + +~~~ +
+ 
+ 
+
+
+ 
+ 
+ 
+
+
+
+ 
+
*AdversarialPrediction.jl provides a way to easily optimize generic performance metrics in supervised learning settings using the Adversarial Prediction framework.
-Tools to put data into the right order for creating a model.
- -
+
+
+
+
$(titles[i])""") + !isnothing(dates[i]) && write(io, " ($(monthname(dates[i])) $(year(dates[i])))") + write(io, "
\n") + end + return String(take!(io)) +end diff --git a/zygote.png b/zygote.png deleted file mode 100644 index 5e8d4920..00000000 Binary files a/zygote.png and /dev/null differ