Harrison Kinsley, Daniel Kukiea - Neural Networks from Scratch in Python (2020).pdf

1、Preface-Neural Networks from Scratch in Python 2 Neural Networks from Scratch in Python Harrison Kinsley&Daniel Kukiea Preface-Neural Networks from Scratch in Python 3 Acknowledgements Harrison Kinsley:My wife,Stephanie,for her unfailing support and faith in me throughout the years.Youve never doubt

2、ed me.Each and every viewer and person who supported this book and project.Without my audience,none of this would have been possible.The Python programming community in general for being awesome!Daniel Kukiea for your unwavering effort with this massive project that Neural Networks from Scratch beca

3、me.From learning C+to make mods in GTA V,to Python for various projects,to the calculus behind neural networks,there doesnt seem to be any problem you cannot solve and it is a pleasure to do this for a living with you.I look forward to seeing whats next!Preface-Neural Networks from Scratch in Python

4、 4 Daniel Kukiea:My son,Oskar,for his patience and understanding during the busy days.My wife,Katarzyna,for the boundless love,faith and support in all the things I do,have ever done,and plan to do,the sunlight during most stormy days and the morning coffee every single day.Harrison for challenging

5、me to learn Python then pushing me towards learning neural networks.For showing me that things do not have to be perfectly done,all the support,and making me a part of so many interesting projects including“lets make a tutorial on neural networks from scratch,”which turned into one the biggest chall

6、enges of my life this book.I wouldnt be at where I am now if all of that didnt happen.The Python community for making me a better programmer and for helping me to improve my language skills.Preface-Neural Networks from Scratch in Python 5 Copyright Copyright 2020 Harrison Kinsley Cover Design copyri

7、ght 2020 Harrison Kinsley No part of this book may be reproduced in any form or by any electronic or mechanical means,with the following exceptions:1.Brief quotations from the book.2.Python Code/software(strings interpreted as logic with Python),which is housed under the MIT license,described on the

8、 next page.Preface-Neural Networks from Scratch in Python 6 License for Code The Python code/software in this book is contained under the following MIT License:Copyright 2020 Sentdex,Kinsley Enterprises Inc.,https:/nnfs.io Permission is hereby granted,free of charge,to any person obtaining a copy of

9、 this software and associated documentation files(the“Software”),to deal in the Software without restriction,including without limitation the rights to use,copy,modify,merge,publish,distribute,sublicense,and/or sell copies of the Software,and to permit persons to whom the Software is furnished to do

10、 so,subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED“AS IS”,WITHOUT WARRANTY OF ANY KIND,EXPRESS OR IMPLIED,INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHAN

11、TABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,DAMAGES OR OTHER LIABILITY,WHETHER IN AN ACTION OF CONTRACT,TORT OR OTHERWISE,ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS I

12、N THE SOFTWARE.Preface-Neural Networks from Scratch in Python 7 Readme The objective of this book is to break down an extremely complex topic,neural networks,into small pieces,consumable by anyone wishing to embark on this journey.Beyond breaking down this topic,the hope is to dramatically demystify

13、 neural networks.As you will soon see,this subject,when explored from scratch,can be an educational and engaging experience.This book is for anyone willing to put in the time to sit down and work through it.In return,you will gain a far deeper understanding than most when it comes to neural networks

14、 and deep learning.This book will be easier to understand if you already have an understanding of Python or another programming language.Python is one of the most clear and understandable programming languages;we have no real interest in padding page counts and exhausting an entire first chapter wit

15、h a basics of Python tutorial.If you need one,we suggest you start here:https:/ cite this material:Harrison Kinsley&Daniel Kukiea Neural Networks from Scratch(NNFS)https:/nnfs.io Preface-Neural Networks from Scratch in Python8 Chapter 1Introducing Neural Networks We begin with a general idea of what

16、 neural networks are and why you might be interested in them.Neural networks,also called Artificial Neural Networks(though it seems,in recent years,weve dropped the“artificial”part),are a type of machine learning often conflated with deep learning.The defining characteristic of a deep neural network

17、 is having two or more hidden layers a concept that will be explained shortly,but these hidden layers are ones that the neural network controls.Its reasonably safe to say that most neural networks in use are a form of deep learning.Fig 1.01:Depicting the various fields of artificial intelligence and

18、 where they fit in overall.Preface-Neural Networks from Scratch in Python 9 A Brief History Since the advent of computers,scientists have been formulating ways to enable machines to take input and produce desired output for tasks like classification and regression.Additionally,in general,theres supe

19、rvised and unsupervised machine learning.Supervised machine learning is used when you have pre-established and labeled data that can be used for training.Lets say you have sensor data for a server with metrics such as upload/download rates,temperature,and humidity,all organized by time for every 10

20、minutes.Normally,this server operates as intended and has no outages,but sometimes parts fail and cause an outage.We might collect data and then divide it into two classes:one class for times/observations when the server is operating normally,and another class for times/observations when the server

21、is experiencing an outage.When the server is failing,we want to label that sensor data leading up to failure as data that preceded a failure.When the server is operating normally,we simply label that data as“normal.”What each sensor measures in this example is called a feature.A group of features ma

22、kes up a feature set(represented as vectors/arrays),and the values of a feature set can be referred to as a sample.Samples are fed into neural network models to train them to fit desired outputs from these inputs or to predict based on them during the inference phase.The“normal”and“failure”labels ar

23、e classifications or labels.You may also see these referred to as targets or ground-truths while we fit a machine learning algorithm.These targets are the classifications that are the goal or target,known to be true and correct,for the algorithm to learn.For this example,the aim is to eventually tra

24、in an algorithm to read sensor data and accurately predict when a failure is imminent.This is just one example of supervised learning in the form of classification.In addition to classification,theres also regression,which is used to predict numerical values,like stock prices.Theres also unsupervise

25、d machine learning,where the machine finds structure in data without knowing the labels/classes ahead of time.There are additional concepts(e.g.,reinforcement learning and semi-supervised machine learning)that fall under the umbrella of neural networks.For this book,we will focus on classification a

26、nd regression with neural networks,but what we cover here leads to other use-cases.Neural networks were conceived in the 1940s,but figuring out how to train them remained a mystery for 20 years.The concept of backpropagation(explained later)came in the 1960s,but neural networks still did not receive

27、 much attention until they started winning competitions in 2010.Since then,neural networks have been on a meteoric rise due to their sometimes seemingly Preface-Neural Networks from Scratch in Python 10 magical ability to solve problems previously deemed unsolvable,such as image captioning,language

28、translation,audio and video synthesis,and more.Currently,neural networks are the primary solution to most competitions and challenging technological problems like self-driving cars,calculating risk,detecting fraud,and early cancer detection,to name a few.What is a Neural Network?“Artificial”neural n

29、etworks are inspired by the organic brain,translated to the computer.Its not a perfect comparison,but there are neurons,activations,and lots of interconnectivity,even if the underlying processes are quite different.Fig 1.02:Comparing a biological neuron to an artificial neuron.A single neuron by its

30、elf is relatively useless,but,when combined with hundreds or thousands(or many more)of other neurons,the interconnectivity produces relationships and results that frequently outperform any other machine learning methods.Preface-Neural Networks from Scratch in Python 11 Fig 1.03:Example of a neural n

31、etwork with 3 hidden layers of 16 neurons each.Anim 1.03:https:/nnfs.io/ntr The above animation shows the examples of the model structures and the numbers of parameters the model has to learn to adjust in order to produce the desired outputs.The details of what is seen here are the subjects of futur

32、e chapters.It might seem rather complicated when you look at it this way.Neural networks are considered to be“black boxes”in that we often have no idea why they reach the conclusions they do.We do understand how they do this,though.Dense layers,the most common layers,consist of interconnected neuron

33、s.In a dense layer,each neuron of a given layer is connected to every neuron of the next layer,which means that its output value becomes an input for the next neurons.Each connection between neurons has a weight associated with it,which is a trainable factor of how much of this input to use,and this weight gets multiplied by the input value.Once all of the inputsweights flow into our neuron,they a