wrong predictions

[AqsaHassan]

Hey...thanks a lot for resolving the last issue and sorry for delay
The model has successfully trained but unfortunately giving wrong predictions i.e predicting all test images as result : /resnet152/resnet152.pt [[ 5.5552482e+11 4.6270378e+10 -5.7006516e+11]]
ford
OUTPUT:
/home/aqsa/anaconda3/envs/VGG/lib/python3.7/site-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no pr**edicted samples.
'precision', 'predicted', average, warn_for)
(VGG) aqsa@aqsa-OptiPlex-7060:~/Car-Model-Recognition-master-try3$ python3 main.py -i hondatest/filehonda2.jpg
input: hondatest/filehonda2.jpg
results/resnet152/resnet152.pt
[[ 1.5563196e+12 5.7977353e+11 -2.1043264e+12]]
ford

input: hondatest/filehonda3.jpg
results/resnet152/resnet152.pt
[[ 1.5563196e+12 5.7977353e+11 -2.1043264e+12]]
ford

input: fileford5.jpg
results/resnet152/resnet152.pt
[[ 5.5552482e+11 4.6270378e+10 -5.7006516e+11]]
ford

input: fileford4.jpg
results/resnet152/resnet152.pt
[[ 5.5552482e+11 4.6270378e+10 -5.7006516e+11]]
ford

[Helias]

Hey, how many epochs did you train?

[AqsaHassan]

Configuration

DEBUG = False

training

EPOCHS = 1
MOMENTUM = 0.9
LEARNING_RATE = 0.01
BATCH_SIZE = 16
THREADS = 8

USE_CUDA=False

file paths

IMAGES_PATH = "dataset"
TRAINING_PATH = "train_file.csv"
VALIDATION_PATH = "test_file.csv"
TEST_PATH = "test_file.csv"

RESULTS_PATH="results"

[Helias]

try to put EPOCHS = 10 or at least EPOCHS = 3

[AqsaHassan]

okay

[Helias]

More epochs you trainer more the model elaborate your data, I used with a 8 GB NVIDIA GPU to make 25-80 epochs, 3 epochs probably is too low, if you can try with at least 25.

If you still get some problems send the resnet152_graph.png file here, it allows to understand better the state of the model.

If you reach your goal, leave a feedback here and close the issue ;-)

[AqsaHassan]

What should I do now? already I have reduced the batch size... Previously I was training on 3 thousand images per class and now I'm training on only 605 images per class.

[AqsaHassan]

there's nothing in this graph as well :/
Could you please mention the maximum accuracy you achieved after training ?

[AqsaHassan]

again detecting all images as honda_civic

Configuration

DEBUG = False

training

EPOCHS = 25
MOMENTUM = 0.9
LEARNING_RATE = 0.01
BATCH_SIZE = 2
THREADS = 8

USE_CUDA=True

file paths

IMAGES_PATH = "dataset"
TRAINING_PATH = "train_file.csv"
VALIDATION_PATH = "test_file.csv"
TEST_PATH = "test_file.csv"

RESULTS_PATH="results"

[Helias]

Try to increase to 1k or 2k the items inside honda_civic and the same for Ford
and train again

Or if you already have a lot of items, try to increase the epochs also because the graph seems to be still unstable.

Try 50 epochs.

[AqsaHassan]

okay

…

On Tue, Oct 22, 2019 at 4:27 PM Stefano Borzì ***@***.***> wrote: Try to increase to 1k or 2k the items inside honda_civic and the same for Ford and traing again — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3?email_source=notifications&email_token=AL3HJWUSDQBZ5NYJYPF4S7TQP3PTTA5CNFSM4JCC3RDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB5MPAQ#issuecomment-544917378>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL3HJWQSANLSBT3RUCWNL4LQP3PTTANCNFSM4JCC3RDA> .

[Helias]

A good graph "stable" could be like this one:

After 45-50 epochs (number bottom of the graph), the blue curve is stable and does not have peaks down.
Also the testing curve (the orange one) is pretty "stable", even with some peaks, for the testing is normal that the peaks are frequently.

[AqsaHassan]

this is what I got after 7 hours of training..

[AqsaHassan]

Is it possible to train it for at least 45 classes with 86% accuracy? right now I have just tested this project using 3 classes with 50 epochs with following results.

Accuracy: 0.6013851481338977

Confusion Matrix:
[[25209 15663 10078]
[16117 22565 12318]
[ 8328 11482 8190]]

F1 Score:
[0.50115304 0.44811836 0.27958898]

[Helias]

TL;DR; Yes, but you need data

Full answer:
In Machine Learning the problems are:

• DATA
• algorithms behind the neural network

But there are several neural network algorithms available online like resnet152 (that this project use), vggNET etc.

The DATA is the problem in your case, indeed, if you have a lot of data PER class you can do anything with this project, you have to train it a lot (I think that 50 epochs is enough, with more classes just look at the results_graph and try with 80 if you think aren't enough).

Good data should be 10k per class (for example), but I think that also 1-5k is enough to get 60% accuracy.
I made several tests, and the problem was the DATA "labelled", indeed, I only found public data labelled for "vehicle detection" not "vehicle model recognition".

Anyway, you can try to train the model with 80 classes, but it's a big deal 😅.

P.S. = if you solved, we can close this issue 😉

[Helias]

Anyway, thank you, I just added more info about training and epochs parameter in the README for future users/developers 😉

[vokhidovhusan]

I got good training accuracy but test accuracy is not good as I expected. What does cause bad testing accuracy?

[Helias]

so using the model that you trained on testing you do not get good results?
try to select only balanced data like 1k items per class during the training

[zula-baldez]

@vokhidovhusan have you deal with it? I have the same issue, test accuracy is not growing. I have 5 classes and equal number of images per class

[Helias]

how many images do you have per class?

[zula-baldez]

@Helias It's 2500 images per class