05-02

The model trained without JIT performed way better than

model trained without jit

So now we should figure out what's causing the training routine to get stuck on one picture.

I tried to do some adjustments (mainly creating a tensor and putting different indexes to it and passing it to training_step) but it didn't work. Tomorrow I will try some other ways to fix this.

07-02

I managed to fix the problem with JIT. I modified the training function so that it now takes two parameters which are the image and label tensors instead of taking the index and choosing the image and label inside the function.

The output from the net trained by the jitted training function is now the same as the one trained on a non-jitted function. The outputs aren't yet perfect but I can finally see a path to success. I will try to adjust the hyperparameters and see whether this helps.

I trained the net on all of the images from the benign category for 500 epochs using Tversky loss with alpha = 0.3 and beta = 0.7. The results are the best I've seen so far.

Below are predictions made on two sets of 4 images from the training set. There are some errors but this is nothing that cannot be fixed by adjusting parameters of Tversky.

Tomorrow I'm planning doing some test inference on the validation dataset. I don't expect great generalization performance here just yet but I'm still very interested in seeing the results.

first set of predictions

second set of predictions

08-02

I did some testing of the neural net trained on the full benign dataset and it fares better than expected on the validation set. It still definitely needs improvements in the realm of generalization though so I should look into some generalization methods that would work well for this type of workload.

inference output on val set 1

inference output on val set 2

I also did inference on four images taken from the malignant set (which I didn't use for training) and as expected - it fared very poorly. Although to the net's credit it looks like there weren't any false-positives present in the output.

inference output on the malignant set

I'm right now training the net on the entire dataset and so far it definitely fares better than before. Just look at the output for the same set of images from that net. All hyperparams are the same.

inference output on the malignant set (net trained on full dataset)

This is inference on a model that has trained for 200 epochs. I'm waiting for training to end to do a real apples to apples comparison. We can in the meantime write a function that does some uniform transforms on the images before every epoch in order to artificially increase the amount of examples. I think that this is the easiest way to gain some generalization. Then we could also try using Focal loss instead of Tversky because now that the net is actually being properly trained. I want to know whether Focal can actually get us any advantage vs Tversky on the hard examples.

tversky-a0_3-b0_7-malignant-full.png