{"id":1540,"date":"2021-10-15T16:49:57","date_gmt":"2021-10-15T14:49:57","guid":{"rendered":"https:\/\/maxfest.dk\/port\/?p=1540"},"modified":"2021-10-15T16:49:57","modified_gmt":"2021-10-15T14:49:57","slug":"noter-i-anvendt-machine-learning-optimization-and-regularization-15-10-2021","status":"publish","type":"post","link":"https:\/\/maxfest.dk\/port\/skiftligt-materiale\/noter\/noter-i-anvendt-machine-learning-optimization-and-regularization-15-10-2021\/","title":{"rendered":"Noter i Anvendt Machine learning – Optimization and regularization – 15\/10 2021"},"content":{"rendered":"\n

Memorising = overfitting.<\/p>\n\n\n\n

Large neural networks are tendent to overfit. And that is to be avoided.<\/p>\n\n\n\n

Trade off between model and data<\/h2>\n\n\n\n

Regularization: to change data in a way that aids in avoiding overfitting.<\/p>\n\n\n\n

flatten can be used on data that is flattend, but is not necesary.<\/p>\n\n\n\n

When overfitting, it is because it has the capcit to fit disturbance patterns in the data (which is random noice). It will make the model perform worse.<\/p>\n\n\n\n

The simplest way to avoid overfitting is to se a simple model. It can still happen, but there is less options to do so.<\/p>\n\n\n\n

Large models have the potential to perform better when not overfitted.<\/p>\n\n\n\n

Validation data as tool to detect overfitting<\/h2>\n\n\n\n

Vanishing gradient leads to weights of zero and small results.<\/p>\n\n\n\n

Exploding gradient leads to infinite weights and large results.<\/p>\n\n\n\n

Early stopping is stopping the model when it’s posible to tell that the model started stagnate.<\/p>\n\n\n\n

Weight regulariztion as tool to improve generalization<\/h2>\n\n\n\n

Weight is hard to regularize, as it can only be optimized through trial and error.<\/p>\n\n\n\n

Weight reularization can be done with weight. This is done by shinkage. Weights for poor classifiers are set to 0.<\/p>\n\n\n\n

Weight regulariation can be done with l1 and l2.<\/p>\n\n\n\n

Weight initialization can also be uses. Which is to adjust weights before training start.<\/p>\n\n\n\n

Dropout as tool to improve generalization<\/h2>\n\n\n\n

Dropout can be used to reguralize by removing some connections at a given layer. It introduces noice and limits the potential of hidden nodes to co-addapt.<\/p>\n\n\n\n

\"\"
How dropout works – Source: “Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929-1958.”<\/figcaption><\/figure>\n\n\n\n
\"\"
Dropout at training and test – Source: “Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929-1958.”<\/figcaption><\/figure>\n\n\n\n

Batch normalization as tool to improve training and generalization<\/h2>\n\n\n\n

Batch normalization is good to implement before dropout. This will be discoused later.<\/p>\n\n\n\n

Normalizing can help not only at the input layer, but at every layer. Batch normalizion is an option to do that.<\/p>\n\n\n\n

It does have some drawbacks. It reqires a slow learning rate, carefull initialization and it may saturate non-linearties by making gradients.<\/p>\n\n\n\n

\"\"
Batch normalization algorithm – Source: “Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.”<\/figcaption><\/figure>\n\n\n\n

\u03b3\u00a0and\u00a0B<\/em> (the betta) introduces bias, but since they are trainable, they can be optimized. Ecentialy noice is introduced only to be further reduced.<\/p>\n\n\n\n

Batch normalization is allmost allways a good idea.<\/p>\n\n\n\n

Small batch sizes don’t allways work so well – especially batch sizes of 1. To large batch sizes might introduce other problems too.<\/p>\n\n\n\n

It does require somewhat high computeational power.<\/p>\n\n\n\n

Batch normilization works well with other methods of normilization. Usually dropout.<\/p>\n\n\n\n

Data augmentation<\/h2>\n\n\n\n

With a set of data, eg. images, especially a small sample, the data can be transormed in some degree to be computated as new data.<\/p>\n\n\n\n

This could be done by rotating, asdjusting colors, fliping and introduce noice to the pictures.<\/p>\n\n\n\n

Augmentation will add a lot more data to the dataset.<\/p>\n","protected":false},"excerpt":{"rendered":"

Memorising = overfitting. Large neural networks are tendent to overfit. And that is to be avoided. Trade off between model and data Regularization: to change data in a way that aids in avoiding overfitting. flatten can be used on data that is flattend, but is not necesary. When overfitting, it is because it has the…<\/p>\n","protected":false},"author":1,"featured_media":715,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[34,30],"class_list":["post-1540","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-noter","tag-anvendt-maskinlaering-ds807","tag-sdu"],"_links":{"self":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1540","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/comments?post=1540"}],"version-history":[{"count":10,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1540\/revisions"}],"predecessor-version":[{"id":1553,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1540\/revisions\/1553"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/media\/715"}],"wp:attachment":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/media?parent=1540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/categories?post=1540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/tags?post=1540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}