Home webserver setup on a Raspberry pi 4 using balena and Nginx

Table of Contents

Apr 22, 2020 • Odysseas.eth • nginx, balena, iot

Introduction

In this post, we will be using a spare Raspberry pi 4 to host our very own website using the internet connection of our house.

We will start with some introductory terms to get a lay of the land and then we will continue with the tutorial itself.

If you are familiar with the relevant terms (IP, Domain Name, etc.), go ahead and jump to Let’s get to it.

Static Website

So, this idea came to me when I was considering alternatives for a new blog I wanted to start. Up to this point, I used Github pages which hosts for free any static website that belonged to an organization, a project or a person.

For those who are not familiar with web programming, as we read from Wikipedia:

A static web page (sometimes called a flat page or a stationary page) is a web page that is delivered to the user’s web browser exactly as stored, in contrast to dynamic web pages which are generated by a web application.

So, the website is not supported by a back-end web application, but it’s only a set of .html, .css, .js files that the server sends to the browser for the user to view the website. Wordpress sites, for example, are not static, since it’s supported by a PHP server, a SQL database and various other components.

Webserver

In our case, to keep things as simple and as lightweight as possible, we will be serving a static website using a static webserver, Nginx. It is one of the oldest and most performing webservers, allowing us to serve up to 1000 users without our Raspberry breaking a sweat.

Nginx is a very robust web server, allowing the user to perform a myriad of different uses, from serving a static website to performing inverse proxy. Its configuration is very straightforward (as we will see below) and as we will find out soon enough and in essence we will simply dictate the server to serve a set of static files (our website), each time there is a connection at a specific port. (What is a computer port?)

Blogging software - Jekyll

As we are lazy, we don’t want to write a blog website from scratch, as it would entail considerable overhead for each new post we want to make. What we want, is a framework that will have a certain theme and which will generate the static files of the blog for us, allowing us to focus solely on the content of the blog.

Luckily for us, there is a very easy-to-use framework, called Jekyll. It was created by Github’s co-founder Tom Preston-Werner. As we read from the project’s repository README:

Jekyll is a simple, blog-aware, static site generator perfect for personal, project, or organization sites. Think of it like a file-based CMS, without all the complexity. Jekyll takes your content, renders Markdown and Liquid templates, and spits out a complete, static website ready to be served by Apache, Nginx or another web server. Jekyll is the engine behind GitHub Pages, which you can use to host sites right from your GitHub repositories.

The power of Jekyll is that it is super easy to use, so easy, that you don’t even need programming knowledge (Verified from personal experience). In essence, you configure a Jekyll theme using a central configuration file and then you write the blog posts in markdown format (More on markdown here).

Domain

Now that we have the core pieces of the website, it is time to think about the domain name and the potential issue of dynamic IP.

If this doesn’t sound familiar, let’s spend a minute for a computer science 101 super-mini-course.

What’s an IP

Each computer that is connected to a network is identified by a unique address, or IP, very much like your home address. The Internet is a global network of computers, thus each server has an IP.

As your home router is connected to the internet, it has it’s own IP address, which you can find using a service like whatsmyipaddress. At the same time, the router creates a local network in which all your computers at home are connected, thus each computer has a local IP and all your computers, since they are connected to the Internet through the router, will have the same global IP, that of the router.

But what it has to do with domains?

Because it is hard for a person to remember an IP, there are services that have Huge registries, in which an IP is tied to a human-understandable word, or Domain. When you pay for a domain name, you pay to register your IP to these registries and tie that IP to the domain name that you have bought.

Now, each time someone enters that domain name, the computer automatically connects to several Domain Name Registries searching for an IP that is tied to that specific domain name. When it is found, the browser connects to the server using the IP and loads the content.

The problem arises when our IP is not static but it changes continuously, in other words, it’s dynamic.

Dynamic IPs aka “The Plot Thickens”

Many ISPs (Internet Service Providers) around the world offer a dynamic IP, meaning that the IP doesn’t stay the same but changes now and then, according to the policies of each ISP. This creates a challenge, as the domain name will have to point to a new IP each time our IP changes.

Luckily for us, most domain name providers offer a service called Dynamic DNS. This service allows the customer to use their API to update the IP to which the domain name must point to. We will be using a small program called ddclient which supports most of the known domain name providers.

Wait a minute, I don’t have a domain name

If you don’t have a domain, go ahead and grab one from one of the major Domain Name providers (just google buy domain name). Make sure that the Domain Name provider that you choose supports Dynamic DNS and ddclient. This guide was tested using Namecheap.

balena.io

At this point, it is apparent that we will be needing to install and configure a bunch of software not only to bootstrap the website but to also keep it up to date. To do that, we will be using balena.io to develop and deploy our software to the Raspberry pi with the ease and speed of using a cloud-service provider.

That’s right, using balena to provision and manage our embedded IoT device, we will be having the same tools and workflows that one would expect from AWS.

So what’s the deal, exactly?

The team behind balena.io were the first people to port docker to the Raspberry family, showcasing how the container visualization paradigm could serve the domain of the Internet of Things.

Balena now offers a full feature-set that enables us to manage-literally thousands- devices, such as a Raspberry pi, as easily as ever

We will develop our application as a multi-container application, meaning that the distinct services from which the project is constructed will run as distinct containers, completely isolated one from another.

You can think the docker engine balena-engine (our optimized for the IoT version of docker) as an oven where you can bake both a fish and a cake at the same time, while each will taste and smell just fine when you take them out.

In the same sense, each service can run independently, without having to worry with incompatible libraries or different versions. They will taste work just fine.

Finally, balena allows us to

  1. easily access the device’s logs
  2. ssh into the host OS or one of the containers
  3. push a new release by simply running a command.

This last part is pure black magic.

You simply define your application in a docker-compose.yaml, you define a couple of Dockerfiles and then you just push your project. balena takes care of building the project specifically for your device on its build-servers and then it simply sends the built project to the device. The smart supervisor is responsible for downloading and setting up your application according to the docker-compose file.

Developing and managing IoT devices have never been so easy and beautiful. Here is a sneak peek of the dashboard for our device:

Disclaimer: I worked at balena.io in the product team. Thus, you could say that I am a bit biased.

Complimentary Software:

Certbot

Certbot is a service offered by letsencrypt, a nonprofit Certificate Authority providing TLS certificates for anyone who may ask. This way, users will be able to connect securely on our website, using https.

We will be using the certbot-CLI program to request a certificate for our website. In essence, certbot will place a special file for our webserver to serve. When the authority tests the website, it will find the specific file and verify that the website (and thus the domain) is indeed ours. Giving us a certificate for 90 days.

Netdata

Netdata is a monitoring agent that runs on the device, aggregates, visualizes and presents various data about the operation of the machine. From fairly simple, such as RAM usage, to more complicated such as CPU interrupts. Moreover, it has collectors for specific apps that can auto-detect if it’s running and start gathering data.

We will be using Netdata because:

  1. It’s super light (about 5% CPU consumption) and thus ideal for the constraint nature of a Raspberry pi 4.
  2. It can auto-detect nginx and start gathering data using the stub_page of the webserver. You can read more about stub_page here.

Architecture

We will be using the multi-container functionality of the platform, thus the project will consist of several different containers, each one running a specific component. This architecture enables us to isolate one component from another, facilitating the management and configuration of the application.

Components

  1. webserver: Runs the nginx service and the certbot for SSL generation
  2. ddclient: Runs the ddiclient service
  3. netdata monitoring: Runs an instance of Netdata Monitoring software to overview the load of the server.

Let’s get to it

Provisioning the Device

The first order of business would be to provision our device, a Raspberry pi 4.

To do that, we need a new account at balena.io and we need to head over to the Get Started Guide of the balena platform and finish it. It will prompt you to install a demo-app in your device, but that’s ok. We need you to get familiar with the platform before we continue.

Disclaimer: After you finish with the guide, don’t turn off the device, we will need it for later. Just leave it be. Ok? Cool.

Go ahead, I’ll wait.

Waiting..


One Note: The first 10 devices on balena.io are for Free, so using your new account will be more than enough for this project.

Before going forward, we assume that:

  1. You have balena-CLI installed
  2. You have balena-etcher installed
  3. You have logged in balena dashboard

Installing the Software

Jekyll

Although this blog post focuses mainly on setting up a balena-powered raspberry pi webserver, we want to give some insight into how to create a website in the first place.

  1. Visit Jekyll’s website and follow the Get started guide.
  2. Get yourself familiar with the Jekyll templating engine
  3. Search for a Jekyll theme that is appealing to you:
    1. http://jekyllthemes.org/
    2. https://jekyllthemes.io/
  4. Download the theme locally and configure it according to the theme’s and Jekyll’s documentation.
  5. Build the website according to the theme’s documentation, the source files will be placed in a directory called _site.
  6. Upload the files inside _site to a Github Repository.


Disclaimer: If you haven’t used Github again:

  1. Follow this guide to create a new repository to Github.
  2. Follow this guide to upload all your website’s source files to the repository you just created.
  3. Congratulations! You have your very first project on Github!

Installing the server on the device

Well, you don’t have to do anything. The software is already shipped ready to be installed directly on the device. Jump to the next section in order to download it locally, change some configuration based on your setup and then ship it!

Configuring the Software

Environment Variables

  1. Go here and download the project’s repository locally. If you are not familiar with Github, please refer to their documentation.

  2. cd into the repository using a terminal program (cmd in windows).

  3. Open the file .balena/balena.yml using your favorite code editor. If you don’t have one, go ahead and install Visual Studio Code

  4. In this file, we need to change a couple of environment variables which we make available to the services via their respective Dockerfiles. Go ahead and change the variables:

    1. In order to find the REPO_ZIP_URL, go to your Github repository, click on clone or download and then right-click on Download ZIP and click on Copy Link Address.

    2. Fill in the domain of your website according to this certbot example. Please note that www.domain.com and domain.com are two different domains, it is best to include both.

REPO_ZIP_URL=https://github.com/OdysLam/odyslam.github.io/archive/master.zip
REPO_NAME=odyslam.github.io
CERTBOT_MAIL=odyslam@gmail.com
CERTBOT_DOMAIN_1=www.example.com
CERTBOT_DOMAIN_2=example.com

This environment variables are not expected to change while the server runs, thus we prefer to define them at build time.

On the other hand, there are 2 environment variables that can be set using balena dashboard and when the nginx container reloads, it will pick them up.

  1. SYNC_WEBSITE: If this environment variable is set to “1”, the container will always download the latest version of the website every time it restarts.

  2. CERTBOT_FORCED: If this environment variable is set to “1”, the container will always request a new certification every time it restarts. If the current certification is still valid, it will simply inform the user that the certification is up-to-date and will exit.

  • You can read more about environment variables in balena, in the documentation.
  • You can read more about build-time secrets in balena, in the documentation.

nginx configuration

  1. Run the commands bellow to generate a private key that will be used by nginx for SSL related functionality. As it might take some minutes, go ahead and read some information about it in this Stack Overflow Question.
cd nginx
openssl dhparam -out dhparam.pem 2048
cd ..
  1. Using a text editor, open nginx.conf which will find the nginx directory. Head over to the following excerpt and replace the www.example.com and example.com with your domain name.
server {
listen 443 ssl http2;
server_name www.example.com example.com
...

server {
listen 80;
listen [::]:80;
server_name www.example.com example.com;
...

If you want to read more about the nginx configuration file and what the various fields mean, you can read more about it here:

  1. Digital ocean article

  2. Nginx Beginner’s Guide

ddiclient configuration

  1. Create a configuration file for ddclient using a text editor:

    1. You can find examples in the ddclient’s documentation

    2. If you used Namecheap, you can find an example configuration in the namecheap documentation

  2. Place the configuration file you just created into the ddclient folder

Configuring the local network

In order for this project to succeed we need to do 2 things:

  1. Set the raspberry pi to have a static ip in the local network. This will ensure that the router will always know what’s the correct IP for the server.
  2. Set the router to allow connections from the Internet that are intended for the server. In other words, allow users to access our web-page!

Static IP

Balena allows us to set our device to static IP in a breeze.

  1. Find out the IP form that the router uses to assign IPs.

    1. Go to your balenaCloud dashboard, you can find the IP of your device at the summary page.

  2. Use a text editor to open the file static-ip that you will find in the directory tools of the repository you downloaded.

  3. Replace the field ROUTER_IP with the IP of the router.

  4. Replace the field DEVICE_IP with the IP of your balena device with a small change. Change the digits after the last . to 100.

  5. Close the text editor.

Here is an example, note that here the IP has the format 192.168.1.X

[connection]
id=my-ethernet
type=ethernet
interface-name=eth0
permissions=
secondaries=

[ethernet]
mac-address-blacklist=

[ipv4]
#This is the important line
address1=192.168.1.100/24,192.168.1.1
dns=8.8.8.8;8.8.4.4;
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

Port Forwarding

When someone attempts to connect to the IP that is assigned to your home connection, in essence he/she connects to the router, as it functions as the gateway between the outer network (the Internet) and the local network (LAN). Thus we want to tell the router that any time someone tries to continue to some specific port, in reality, he/she wants to connect to our server, thus the router must forward the connection to the Raspberry pi (the web server).

In other words, we need to forward ports 80 and 443 to the Raspberry pi.

  1. Visit portforward
  2. Find your router
  3. Follow instructions and forward the ports to the device’s IP

Deploy

Now that we have everything configured, let’s start deploying the various components.

Deploy the project to the device

  1. Open balenaCloud and create a new application for Raspberry pi 4. You can name it whatever you want. We will assume that you name it bananas.
  2. Download an development image for that application. Don’t bother filling in your wifi credentials, we will be using ethernet as wifi is not very reliable for a webserver.
  3. Burn the image using the best image format app in the world, balena-etcher.
  4. Pull out the sd card and re-insert it in the computer. Copy the file named static-ip we created earlier into the system-connections directory of the sd card.
  5. Insert the card into the Raspberry pi and connect it to power.
  6. Go to the balena-nginx directory, where your docker-compose.yaml is located, and push the project
balena push bananas

Generating the SSL certificate

To generate the SSL certificate, we don’t have to do anything.

The server will detect the absence of the certificates and will run the certbot service to register your website. Afterward, it will simply start the server and will serve your website.

The certificates will be saved into a persistent directory with a functionality called named volumes. This enables the device to persist your certificates (or any data in that directory)

Updating your certificates

You are very organized and want to plan ahead? No problem, certbot will automatically send you an e-mail when the certifications are about to expire.

When you receive that e-mail, you only need to:

  1. Log in to balena dashboard
  2. Go to device summary
  3. Open ssh session to the nginx container
  4. Type certbot renew
  5. Done

Push new content to the website

All we have to do in order to push new content to the website is to update the source files in the directory from which we have configured nginx to serve our website. In our case /usr/share/nginx/html/.ping

To do that, we have a script inside the nginx container, which downloads the latest version of our website from the GitHub repository we have defined.

To run the script:

  1. SSH into nginx container, preferably using balena dashboard
  2. ` /update-blog.sh`

But, what happens if we want to add new content to our website?

We have to upload the new source files into our GitHub Repository and then we have to run the update-blog.sh script in order to download the new version in the server.

So, in order to push new content to the website:

  1. Change the website’s source files
  2. Upload the new files to the Repository and add a commit message to describe the changes
  3. From balena dashboard, ssh into the nginx container and run: /update-blog.sh

Push new content to the website - For advanced users

The whole process has been automated, and you can simply run the script deploy-dev-loca.sh from the local directory of the project, like this:

./deploy-dev-local.sh "<commit message>"

The field <commit message> must be replaced with a commit message for the addition to the GitHub repository, just like you did when you uploaded the files using the website of GitHub.

This script does the following things:

  1. Builds the Jekyll website and outputs it to a pre-determined directory
  2. Commits the changed source files to the website’s local repository
  3. Pushes the new local repository to the remote website’s repository at Github
  4. SSH into the hostOS of the device, assuming it runs a development image, and then it SSH into the nginx container
  5. Runs the update-blog.sh script, which in turns:
    1. Downloads the website’s source files from the remote Github repository (the one we just uploaded to)
    2. Replaces the old website with the new source files
  6. Now, Nginx serves the new website

Before you can you can use the script, you have to open the file using your favorite text editor and replace the following fields:

# export J_OUTPUT= Absolute path to the directory of the website's source files
# export REPO = Absolute path to the directory of the website's source files
# export DEV_UUID= UUID of the device, can be found from the device's dashboard

Finally, a use-case

In case you want to see how this setup works, here is my setup divided into 3 repositories:

  1. balena-nginx: The source files for the setup of the webserver.
  1. clyell: Fork from a Jekyll theme which I have modified extensively. This repository has all the files that are used by Jekyll engine to create the source files of my blog.
  2. odyslam.github.io: Repository which holds the source files of my entire website. This is the repository that is downloaded to the webserver. It consists of the main website which is a plain html, CSS, JS website built by hand and the /blog directory which is built by Jekyll from the files of the clyell repository.

Your faithful correspondent is also on Twitter.