{"id":1511,"date":"2021-10-13T10:00:52","date_gmt":"2021-10-13T08:00:52","guid":{"rendered":"https:\/\/maxfest.dk\/port\/?p=1511"},"modified":"2021-10-13T10:25:36","modified_gmt":"2021-10-13T08:25:36","slug":"deep-learning-noter-backpropagation-13-10-2021","status":"publish","type":"post","link":"https:\/\/maxfest.dk\/port\/skiftligt-materiale\/noter\/deep-learning-noter-backpropagation-13-10-2021\/","title":{"rendered":"Deep learning noter – Backpropagation – 13\/10 2021"},"content":{"rendered":"\r\n

<\/h2>\r\n

Chain Rule of Calculus<\/h2>\r\n

\r\n\r\n<\/h2>\r\n

If is differentiable at and is differentiable at (), then the composite function = \u2218 defined by = is differentiable at and \u2032 is given by:<\/p>\r\n

\r\n\r\n<\/h2>\r\n
\r\n
\"\"
Source: SDU<\/figcaption><\/figure>\r\n\r\n<\/figure>\r\n

\r\n\r\n<\/h2>\r\n
\r\n
\"\"
if = () and = () – Source: SDU<\/figcaption><\/figure>\r\n<\/figure>\r\n

\r\n\r\n<\/h2>\r\n

Accumulation<\/h2>\r\n

\r\n\r\n<\/h2>\r\n
    \r\n
  • Forward Accumulation<\/li>\r\n
  • Backward Accumulation (backdrop)<\/li>\r\n<\/ul>\r\n

    \r\n\r\n<\/h2>\r\n

    Two different ways to do calculations.<\/p>\r\n

    \r\n\r\n<\/h2>\r\n

    Forwards has more steps. Backward only requires the lost functions of the direvatives to get the results.<\/p>\r\n

    We only care about the node, and not the entire network for accumulations during the calculation.<\/p>\r\n

    Accumulation is not learning.<\/p>\r\n

    xx udklip 3 – backward propogation visualization<\/p>\r\n

    Generalization to Vectors<\/h3>\r\n

    xx udklip<\/p>\r\n

    xx udklip 1<\/p>\r\n

    Vectors follows the same logic.<\/p>\r\n

    Backpropropagation methods<\/h3>\r\n
      \r\n
    1. Symbol to number differentation – takes computentioal graph and a set of numerical values (for the inputs to the graph) and returns a set of numerical values that descrie gradient at the input values. Relevant libs: Torch and Caffe.<\/li>\r\n
    2. Symbol to symbol differentation – takes computentioal graph and returns graph with\u00a0 additional nodes that provides a symbolic description of desired derivatives. Relevant libs: Theano and Tensorflow<\/li>\r\n<\/ol>\r\n

      Disabling eager excecution in tensorflow makes it use symbl to symbol.<\/p>\r\n

       <\/p>\r\n

       <\/p>\r\n

      <\/h2>\r\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":715,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[33,30],"class_list":["post-1511","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-noter","tag-deep-learning","tag-sdu"],"_links":{"self":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1511","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/comments?post=1511"}],"version-history":[{"count":9,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1511\/revisions"}],"predecessor-version":[{"id":1529,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/posts\/1511\/revisions\/1529"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/media\/715"}],"wp:attachment":[{"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/media?parent=1511"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/categories?post=1511"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/maxfest.dk\/port\/wp-json\/wp\/v2\/tags?post=1511"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}