How to build a simple Neural Network

Hi there guys, You will be able to program and build a vanilla Feedforward Neural Network (FNN) starting today via PyTorch. Here is the python jupyter codebase for the FNN:

This guide serves as a basic hands-on work to lead you through building a neural network from scratch. Most of the mathematical concepts and scientific decisions are left out. You are free to research more on that part.

Getting Started:

1. Please make sure you have Python and PyTorch installed in your machine:

• Python 3.6 (installation)

• PyTorch (installation)

2. Check the correctness of Python installations by the commands at console:

python -V

The output should be Python 3.6.3 or later version

3. Open a repository(folder) and create your first Neural Network file:

mkdir fnn-tuto

cd fnn-tuto


Start Writing Codes:

All the following codes should be written in the file

Import PyTorch

import torch

import torch.nn as nn

import torchvision.datasets as dsets

import torchvision.transforms as transforms

from torch.autograd import Variable

It will load PyTorch into the codes. Great! A well beginning is half done.

Initialize Hyper-parameters

Hyper-parameters are the powerful arguments that are set up upfront and will not be updated along with the training of the neural network.

input_size = 784 # The image size = 28 x 28 = 784

hidden_size = 500 # The number of nodes at the hidden layer

num_classes = 10 # The number of output classes. In this case, from 0 to 9

num_epochs = 5 # The number of times entire dataset is trained

batch_size = 100 # The size of input data took for one iteration

learning_rate = 0.001 # The speed of convergence

Download MNIST Dataset

MNIST is a huge database with tons of handwritten digits (i.e. 0 to 9) aims for the usage of image processing.

train_dataset = dsets.MNIST(root=’./data’,




test_dataset = dsets.MNIST(root=’./data’,



Load the Dataset:

After downloading the MNIST dataset, we load them into our codes.

train_loader =,



test_loader =,



Note: We shuffle the loading process of train_dataset to make the learning process independent of data order, but the order of test_loader remains to examine it whether we can handle unspecified bias order of inputs.

Build the Feedforward Neural Network:

Now we have our datasets ready. We will start building the neural network. The conceptual illustration can be viewed as below:

Feedforward Neural Network Model Structure:

The FNN includes two fully-connected layers (i.e. fc1 & fc2) and a non-linear ReLU layer in between. Normally we call this structure 1-hidden layer FNN, without counting the output layer (fc2) in.

By running the forward pass, the input images (x) can go through the neural network and generate a output (out) demonstrating how are the likabilities it belongs to each of the 10 classes. For example, a cat image can have 0.8 likability to a dog class and a 0.3 likability to a airplane class.

class Net(nn.Module):

def __init__(self, input_size, hidden_size, num_classes):

super(Net, self).__init__()

# Inherited from the parent class nn.Module

self.fc1 = nn.Linear(input_size, hidden_size) # 1st Full-Connected Layer: 784 (input data) -> 500 (hidden node)

self.relu = nn.ReLU()

# Non-Linear ReLU Layer: max(0,x)

self.fc2 = nn.Linear(hidden_size, num_classes)

# 2nd Full-Connected Layer: 500 (hidden node) -> 10 (output class)

def forward(self, x):

# Forward pass: stacking each layer together

out = self.fc1(x)

out = self.relu(out)

out = self.fc2(out)

return out

Instantiate the FNN;

We now create a real FNN based on our structure.

net = Net(input_size, hidden_size, num_classes)

Enable GPU:

Note: You could enable this line to run the codes on GPU

# net.cuda() # You can comment out this line to disable GPU

Choose the Loss Function and Optimizer:

Loss function (criterion) decides how the output can be compared to a class, which determines how good or bad the neural network performs. And the optimizer chooses a way to update the weight in order to converge to find the best weights in this neural network.

criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

Training the FNN Model:

This process might takes around 3 to 5 minutes depending on your machine. The detailed explanations are listed as comments (#) in the following codes.

for epoch in range(num_epochs):

for i, (images, labels) in enumerate(train_loader):

# Load a batch of images with its (index, data, class)

images = Variable(images.view(-1, 28*28))

# Convert torch tensor to Variable: change image from a vector of size 784 to a matrix of 28 x 28

labels = Variable(labels)


# Intialize the hidden weight to all zeros

outputs = net(images)

# Forward pass: compute the output class given a image

loss = criterion(outputs, labels)

# Compute the loss: difference between the output class and the pre-given label


# Backward pass: compute the weight


# Optimizer: update the weights of hidden nodes

if (i+1) % 100 == 0: # Logging

print(‘Epoch [%d/%d], Step [%d/%d], Loss: %.4f’

%1(epoch+, num_epochs, i+1, len(train_dataset)//batch_size,[0]))

Testing the FNN Model:

Similar to training the neural network, we also need to load batches of test images and collect the outputs. The differences are that:

1 No loss & weights calculation

2 No wights update

3 Has correct prediction calculation

correct = 0

total = 0

for images, labels in test_loader:

images = Variable(images.view(-1, 28*28))

outputs = net(images)

_, predicted = torch.max(, 1) # Choose the best class from the output: The class with the best score

total += labels.size(0) # Increment the total count

correct += (predicted == labels).sum() # Increment the correct count

print(‘Accuracy of the network on the 10K test images: %d %%’ % (100 * correct / total))

Save the trained FNN Model for future use

We save the trained model as a pickle that can be loaded and used later., ‘fnn_model.pkl’)

Congrats. You have done building your first Feedforward Neural Network!

What’s Next

Save and close the file. Start running the file at the console:


You will see the training process going like the following:

Thanks for your time and hope you enjoy the tutorial. All the codes can be found here!



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s