{"id":1489,"date":"2021-10-11T17:58:26","date_gmt":"2021-10-11T15:58:26","guid":{"rendered":"https:\/\/maxfest.dk\/port\/?p=1489"},"modified":"2021-10-13T09:28:06","modified_gmt":"2021-10-13T07:28:06","slug":"deep-learning-noter-gradient-based-learning-11-10-2021","status":"publish","type":"post","link":"https:\/\/maxfest.dk\/port\/vearker\/deep-learning-noter-gradient-based-learning-11-10-2021\/","title":{"rendered":"Deep learning noter – Gradient based learning – 11 & 13\/10 2021"},"content":{"rendered":"\r\n

With a liminiear approximation can be made from the gradiant. This is done by going the oposite direction.<\/p>\r\n\r\n\r\n\r\n

\"\"\r\n
Usage of Derivates – Source: SDU<\/figcaption>\r\n<\/figure>\r\n\r\n\r\n\r\n

With multiple inputs it is used the same way, but returns an output with weights matching the amount of dimensions of the input.<\/p>\r\n\r\n\r\n\r\n

SGD<\/h2>\r\n\r\n\r\n\r\n

SGD is using mini batches. Instead of using all connections to get the most precesise results, batches are used to approximate the result somewhat wore propbably.<\/p>\r\n\r\n\r\n\r\n

From the mini batches an average gradient is calculated and used.<\/p>\r\n\r\n\r\n\r\n

The positive asspect of SGD is that it is faster.<\/p>\r\n\r\n\r\n\r\n

Deep learning specialities<\/h2>\r\n\r\n\r\n\r\n

Deep learning needs the same components as machine learning<\/p>\r\n\r\n\r\n\r\n

    \r\n
  1. Optimization procedure<\/li>\r\n
  2. Cost function<\/li>\r\n
  3. Model family<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n

    Differences causes non-convex loss.<\/p>\r\n\r\n\r\n\r\n

    Problems<\/h3>\r\n\r\n\r\n\r\n
      \r\n
    • Can end in local minimum<\/li>\r\n
    • Can end in local maximum<\/li>\r\n
    • Can end at saddle points – The worst point: learning is stopped, and the results are likely not good.<\/li>\r\n
    • Learning rate – Big learning rates might have large jumps in results and small learning rates might be slow.<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n

      If a particular bad promlem or cliff is encountered, new initialization might be the best solution.<\/p>\r\n\r\n\r\n\r\n

      Cliffs and exploding gradients<\/h3>\r\n\r\n\r\n\r\n

      Cliffs in Neural Network are sudden drops or rices in learning.<\/p>\r\n\r\n\r\n\r\n

      Theese can be midigated by implementing a gradient.<\/p>\r\n\r\n\r\n\r\n

      Inexact gradients are caused by biace or variance.<\/p>\r\n\r\n\r\n\r\n

      About gradients<\/h2>\r\n\r\n\r\n\r\n

      Gradients must be large and somewhat predictable to guide the learning algorithm.<\/p>\r\n\r\n\r\n\r\n

      Functions that saturate (flatten results) undermine gradients.<\/p>\r\n\r\n\r\n\r\n

      Negative log-likelihood helps avoid saturation problem for many models. It undoees gradient accidents as far as i understand.<\/p>\r\n\r\n\r\n\r\n

      Different optimizers<\/h2>\r\n\r\n\r\n\r\n

      <\/p>\r\n

        \r\n
      • SGD\r\n
          \r\n
        • Standard gradient decent<\/li>\r\n
        • utilizes decay, momentum, and Nesterov<\/li>\r\n<\/ul>\r\n
            \r\n
          • for frequent parameter<\/li>\r\n<\/ul>\r\n<\/li>\r\n
          • RMSProp\r\n
              \r\n
            • Root Mean Square Propagation<\/li>\r\n
            • utilizes exponentially decaying running average of the squared gradients (similar to momentum)<\/li>\r\n
            • decay is usually 0.9.<\/li>\r\n<\/ul>\r\n<\/li>\r\n
            • Adagrad\r\n
                \r\n
              • utilizes larger updates for infrequent parameters and smaller updates<\/li>\r\n
              • Good for word embeddings with that has infrequent words<\/li>\r\n
              • Often has diminising learning rate<\/li>\r\n
              • Tries to ease on the diminishing learning rate decay from Adagrad<\/li>\r\n<\/ul>\r\n<\/li>\r\n
              • Adadelta\r\n
                  \r\n
                • similar to RMSProp<\/li>\r\n
                • utilizes root squared parameter updates<\/li>\r\n
                • Does not need set learning rate<\/li>\r\n<\/ul>\r\n<\/li>\r\n
                • Adam\r\n
                    \r\n
                  • Adaptive Moment Estimation<\/li>\r\n
                  • prefers flat minima in the error surface<\/li>\r\n
                  • Adam is basically RMSProp with momentum<\/li>\r\n<\/ul>\r\n<\/li>\r\n
                  • Adamax\r\n
                      \r\n
                    • Same as Adam, but based on the max-norm ^inf.<\/li>\r\n<\/ul>\r\n<\/li>\r\n
                    • Nadam\r\n
                        \r\n
                      • Nesterov-accelerated Adaptive Moment Estimation<\/li>\r\n
                      • Nadam is basically Adam with Nesterov momentum<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n

                        \r\n\r\n<\/p>\r\n

                        Choosing a bad optimizer or bad parameters in general, can lead to inefeicent or slow learning.<\/p>\r\n

                        \r\n\r\n<\/p>\r\n

                        Batch gradient decent can use fixed learning rate.<\/p>\r\n

                        \r\n\r\n<\/p>\r\n

                        SGD jas a source error.<\/p>\r\n

                        \r\n\r\n<\/p>\r\n

                        \"\"\r\n
                        Source: SDU<\/figcaption>\r\n<\/figure>\r\n

                        \r\n\r\n<\/p>\r\n

                        Momentum<\/h3>\r\n

                        \r\n\r\n<\/p>\r\n

                        Momentum hels accelerea gradientdecent. Usefull when<\/p>\r\n

                        \r\n\r\n<\/p>\r\n

                          \r\n
                        • Surfaces that curve more steeply in one direction than in another direction
                          \u00a7 Facing high curvature
                          \u00a7 Small but consistent gradients
                          \u00a7 Noisy gradients<\/li>\r\n<\/ul>\r\n

                          \r\n\r\n<\/p>\r\n

                          \"\"\r\n
                          Dampening of the oscillation – Source: SDU<\/figcaption>\r\n<\/figure>\r\n

                          \r\n\r\n<\/p>\r\n

                          Note: The need for dampening of the oscilation can only be found trhough trial and error.<\/p>\r\n

                          \r\n\r\n<\/p>\r\n

                          Momentum introduces velocity, v. It is the direction and speed at which parameters move through parameter
                          space. Also known as mass in physics.<\/p>\r\n

                          \r\n\r\n<\/p>\r\n

                          \"\"\r\n
                          Source: SDU<\/figcaption>\r\n<\/figure>\r\n

                          \r\n\r\n<\/p>\r\n

                          Nesterov Accelerated Gradient<\/h3>\r\n

                          \r\n\r\n<\/p>\r\n

                          Calculation of gradient if implementet.<\/p>\r\n

                          \r\n\r\n<\/p>\r\n

                          \"\"\r\n
                          Nesterov Accelerated Gradient – Source: SDU<\/figcaption>\r\n<\/figure>\r\n

                          \r\n\r\n<\/p>\r\n

                          Adaptive Learning Rate (ada)<\/h3>\r\n

                          \r\n\r\n<\/p>\r\n

                          Learning rate for each adaptive parameter.<\/p>\r\n

                          <\/p>\r\n","protected":false},"excerpt":{"rendered":"

                          With a liminiear approximation can be made from the gradiant. This is done by going the oposite direction. With multiple inputs it is used the same way, but returns an output with weights matching the amount of dimensions of the input. SGD SGD is using mini batches. Instead of using all connections to get the…<\/p>\n","protected":false},"author":1,"featured_media":715,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,1],"tags":[33,30],"class_list":["post-1489","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-noter","category-vearker","tag-deep-learning","tag-sdu"],"_links":{"self":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1489","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/comments?post=1489"}],"version-history":[{"count":7,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1489\/revisions"}],"predecessor-version":[{"id":1521,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1489\/revisions\/1521"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/media\/715"}],"wp:attachment":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/media?parent=1489"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/categories?post=1489"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/tags?post=1489"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}