Batch와 Gradient Descent

Gradient Descent, Loss, update weights 정리

Three variants for gradient descent

• Batch Gradient Descent, Stochastic Grandient Descent, Batch Gradient Descent

1. Gradient Descent

• 전체 데이터(the whole data-set)를 input하여 Loss계산 -> weights 업데이트 1번(one update occurs)
• 가장 정확한 Gradients Descent
• batch size = training data set size

2. Stochastic Gradient Descent(SGD)

• 하나의 데이터를 input할 때마다 Loss 계산 -> weights 업데이트
• 데이터 갯수만큼 weights 업데이트
• 정확도는 떨어지만 train이 빠름
• batch size = 1

3. Batch Gradient Descent(Mini batch SGD)

• batch size만큼 input하고 Loss 계산 -> weights 업데이트
• eg. whole_data_set_size: 1000; batch_size: 100 -> iter= 1000/100(num of calculating loss); update weight=1000/100

• batch size: n

• reference

  • https://stats.stackexchange.com/questions/49528/batch-gradient-descent-versus-stochastic-gradient-descent
  • https://datascience.stackexchange.com/questions/27421/when-are-weights-updated-in-cnn/27425#27425