question about pixel-wise output

[amiltonwong]

Hi, @alexgkendall ,

The comparison between the output from SegNet webdemo and trained SegNet (follow the tutorial steps) are detailed here:
https://goo.gl/LtNQgA

According to different predicted output from SegNet webdemo and trained SegNet (follow the tutorial steps), what other training procedures for the model in SegNet webdemo are applied ?
Use other training dataset such as ImageNet??
Use pre-trained model such as VGG-16/19??
Other training techniques not mentioned in tutorial??

Best~
Milton

[alexgkendall]

Hi @amiltonwong ,

At first glance, your output does not have smooth class boundaries therefore it does not look like your model has fully converged. Perhaps try training for longer? Can you please show me a plot of your training loss/accuracy curve?

I've added some additional information to the tutorial, including a performance table and results images. The webdemo had a pre-trained encoder, and was trained on additional data which is not publicly available - but you should be able to achieve the results in the table!

"The following table shows the performance we achieved with SegNet on the CamVid dataset. If you have followed this tutorial correctly, you should be able to achieve the first two results. The final result was trained on 3.5K additional labelled images from publicly available datasets, see the paper for further details. The webdemo has been trained on further data which is not publicly available, and on an extra class (road marking)."

If you are interested in training on more of the 32 semantic classes in CamVid, you should look at the raw data here: http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/

Alex

[amiltonwong]

Hi, @alexgkendall ,
Thanks for your detailed reply.
For the evaluation , could you also provide the scripts to compute the "Global Accuracy", "Class Accuracy" and "Mean I/U" ?

Best~
Milton

[alexgkendall]

Hey sorry for the delay. I've added the test script which I use in Matlab - you will need to modify it to iterate through your results. Hope this helps!

https://github.com/alexgkendall/SegNet-Tutorial/blob/master/Scripts/compute_test_results.m

[amiltonwong]

@alexgkendall, thank you so much for your kind help :)

Here is my evaluation result. (Use original 367 images for training and choose snapshot in 32000 iterations):
Global acc = 0.75218 Class average acc = 0.49258 Mean Int over Union = 0.34644

I'll append the further result using pretrained imagenet encoder.

Now, one more evaluation such as the running time is missing.
For evaluating the running time in performing prediction for each test image, how could I evaluate it in segnet? Could you give me some hints?

Furthermore, according to your words in tutorial:
"Training on this small dataset shouldn't take too long. After about 50-100 epochs you should see it converge. You should be looking for greater than 90% training accuracy. Once you are happy that the model has converged then we can now test it."

When the training is performing in segnet, how could we confirm the converge appears? (By what metrics?)

Thanks in advance~
Milton

[alexgkendall]

To evaluate run time use the command
./SegNet/caffe-segnet/build/tools/caffe time -gpu 0 -model /SegNet/Models/segnet_train.prototxt

The easiest way to watch for convergence is by watching the accuracy print outs you get when your model trains. You can also plot accuracy or loss over time using plot_training_logs.py, which is built into caffe. See the caffe docs for more info.

Cheers,
Alex