Building a Raspberry Pi NAS with a RAID-1 and Nextcloud as frontend
My idea was to have my own NAS (network attached storage) to safely store picture, music and other files. While at home I want to access these files using samba from every device (Windows: map network drive; Mac: Connect to server; Android: there a plenty of file managers with this possibility). Additionally, I wanted to have access to exactly the same data when I'm on the road, so I added nextcloud as a web front end for this. Samba and nextcloud work surprisingly well together. The data is backed up by using two HDDs together in a mirrored RAID. Commercial NAS systems offer a similar function, but of course cost some more money.
I had a Raspberry Pi (RasPi) for a while already that I used on and off for small coding projects and in general to get to know Linux. There are tutorials on the web that show you how to turn your RasPi into a NAS (and I will link to them at the bottom of this page), so I went ahead and tried it. The speed of the NAS is limited by the USB ports of the RasPi (USB 2.0), so don't expect wonders. I get about 10 MB/s read/write using samba.
Cloud storage solutions are very nice and come in handy if you need to access your data from anywhere in the world (where you have internet) and also to share files with others. Nextcloud works very much like dropbox and allows you to take this into your own hands and store your data at home on your own server. The good thing is that a RasPi can also be turned into a web server.
2. Material and Costs
- RasPi with microSD card and power supply, the starter kit is about 55€ (I use a Raspberry Pi 3 B+ with a 64 GB Samsung microSD, but other combinations will work fine I guess)
- Two hard drives for the RAID-1, 2x 90€ (see what you can get for cheap and how much storage you need. I got two Western Digital 2.5'' USB 3.0 3TB HDD from amazon while they where 35% off, in total 120€ )
- very likely: a powered USB 2.0 hub, 12€ (the RasPi cannot supply enough power for two 2.5'' USB HDDs, so you need some extra power here). Do not get a powered USB 3.0 hub, they don't work well with the RasPi. If you are going to use 3.5'' HDDs that come already with an external power supply, you can skip this of course.
- optional: a case for the RasPi and/or some rack to store everything nicely together (I got a 4 layer acrylic rack for the RasPi, it also fits the HDDs and the USB hub)
In total you probably end up with less than 250€, but this really depends on the HDDs you want to use. They are the expensive parts.
3. Setting up the RasPi
There are plenty of instructions around the web how to set up a RasPi for the first time and I think they do a much better job then I could do it here.
Just a few tricks I learned: You don't need any network cable, monitor, mouse or keyboard, it is possible to entirely set up a RasPi "headless". All you need is a possibility to put the Raspian (Lite) image onto a microSD card. Once this is done add two files to the main folder of the "root"-partition: One is an empty file without any extension called ssh. This enables the SSH functions of your RasPi and allows you to connect to it via Putty (if you use Windows). The other file is for the WLAN connection (if you want to use this, otherwise use a network cable and skip this step). It needs to be called wpa_supplicant.conf and should look like this:
The ISO code for your country can be found here and should consist of two letters, e.g. GB for the United Kingdom or DE for Germany.
This is all to get the RasPi running, now you need to find out the IP of your RasPi and connect to it. Don't forget to extend the file system and change the password of your pi-user account.
One thing we should do is give the RasPi a static IP address, otherwise it might change every time you restart it or your router. You need to edit a file called /etc/dhcpcd.conf and add some lines to give your WLAN-adapter a static IP.
$ sudo nano /etc/dhcpcd.conf
At the bottom of the file add the following lines, where xxx should be the current IP address of your RasPi (that you used to connect to it via Putty). 192.168.yyy.1 is the IP address of your router, change this if needed. 126.96.36.199 and 188.8.131.52 are the DNS of google, it's not necessary to add them, but it does not harm either. If you are not going to use WLAN, but a wired network connection you need to replace wlan0 by eth0.
static domain_name_servers=192.168.yyy.1 184.108.40.206 220.127.116.11
Don't forget to perform software updates:
$ sudo apt-get update
$ sudo apt-get upgrade
(optional, this updates the RasPi firmware)
$ sudo rpi-update
Reboot your RasPi and you are set for the next steps.
$ sudo reboot now
4. Building the RAID-1
It's time to connect your HDDs to the RasPi to build the RAID.
It should not matter if you connect one device to the RasPi directly and the other via the powered USB hub, or both via the USB hub. As far as I understand it all USB ports on the RasPi are linked internally and will share the bandwidth.
Check if the drives are recognised by typing
$ sudo blkid
You will see a couple of devices there, the HDDs are very likely called /dev/sda and /dev/sdb. There is no need to mount the HDDs now, because we need to re-format them first. I'm using the native linux filesystem ext4, since the driver is integrated at the kernel level and will give you the best performance. Other file systems like exFAT or NTFS can be used as well, but there are no native linux drivers available. This means all drivers that you can use are run on the user level and will stress the CPU of the RasPi a lot if you read/write data on your NAS.
The next step will erase all existing data on the HDDs!
First delete all partitions that are already on the HDDs
$ sudo fdisk /dev/sda
$ d (delete an existing partition)
$ n (create a new partition)
$ w (write partition table)
These steps just create a new partition (called /dev/sda1), but it does not contain any file system. This will be done below when we combine the two HDDs into the RAID-1.
Repeat these steps for /dev/sdb as well.
Now we want to create the RAID volume. For this we need a program called mdadm that provides to ability to use multiple HDDs in a software raid. It can easily be installed:
$ sudo apt-get install mdadm
Using mdadm is very simple, it does everything on its own. We have only two HDDs, but you could simply use more HDDs for different RAID systems, for example three HDDs with RAID-5. More information on RAID can be found here and the mdadm documentation is also available. For a RAID-1 with 2 HDDs use the following command, the new RAID volume will be called /dev/md0
$ sudo mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 /dev/sdb1
To make it work for me had to leave out the 1 behind sda-b, otherwise I would get an error. Try it first with the 1, because this how it should work, but if it doesn't try it without.
We can now write the file system. As written above we want to use ext4:
$ sudo mkfs.ext4 -v /dev/md0
This will take a while depending on the size of the HDDs. Once finished it is a good idea to save the RAID configuration. There seems to be some kind of bug in Raspian and also using sudo will not allow us to do this. The trick is to use
$ sudo -i
$ mdadm --detail --scan >> /etc/mdadm/mdadm.conf
To check if everything worked we use again:
$ sudo blkid
Look for /dev/md0, this is the RAID-1. Also take note of the UUID which is unique to your RAID. It's a good idea to automatically mount the new RAID volume after booting (in case you have to restart your RasPi at some point) and the UUID will help us with this. We should also create a folder where we will mount it every time. I'm using /mnt/NAS.
$ sudo mkdir /mnt/NAS
$ sudo nano /etc/fstab
(to the bottom of the file add)
UUID=YOUR_UUID /mnt/NAS ext4 defaults 0 2
Now we mount the RAID to /mnt/NAS and we are done.
$ sudo mount /dev/md0 /mnt/NAS
The RAID will synchronise itself, which will take some time. For my two 3TB HDDs it took two days, but you can use it already during this time.
5. Installing the apache web server (including PHP and MySQL/MariaDB)
To run nextcloud we need a webserver that supports PHP (we will use apache) and a database. For smaller installations nextcloud says that SQLite will be sufficient, however I suggest to still use a proper database. You never know with how much data you will en up in 2 years. You should also use at least the most recent major PHP version, which ist PHP 7.0.
Everything can be installed via apt-get:
$ sudo apt-get install apache2 mariadb-server
$ sudo apt-get install php7.0 php7.0-gd php7.0-json php7.0-mysql php7.0-curl php7.0-mbstring php7.0-intl php7.0-mcrypt php-imagick php7.0-xml php7.0-zip
$ sudo service apache2 restart
Open your browser on your PC and type the IP address of your RasPi (see point 3). You should be greeted by the apache webserver welcome page.
You should also test the PHP installation, for this create a new file in the directory of the webserver. Usually this should be /var/www/html/ on Raspian.
$ sudo nano /var/www/html/phptest.php
(this opens an empty document, put the following in there)
(save with ctrl+o, exit with ctrl+x)
Switch to your browser again and type the IP address of the RasPi followed by /phptest.php, which should open the newly created file in your browser. You should see all information related to your PHP installation there.
We want to create a new database for the nextcloud system in our MySQL installation and a new user to access the database, since it is not a very good idea to use the root user for this. Make a note of all the data, because we will need in the next step when we install nextcloud. The database will be called nextcloud and the user nxcloud.
$ sudo mysql -u root
mysql> CREATE DATABASE nextcloud;
mysql> CREATE USER 'nxcloud'@'localhost' IDENTIFIED BY 'A_NEW_PASSWORD';
mysql> GRANT ALL PRIVILEGES ON nextcloud.* TO 'nxcloud'@'localhost';
mysql> FLUSH PRIVILEGES;
6. Setting up nextcloud
We can proceed to the next step and download as well as install nextcloud. The easiest is to download the latest version of nextcloud directly to the webserver folder and extract it afterwards. The latest.tar.bz2 file contains a folder called nextcloud and will unpack everything in it. This means our nextcloud will reside in /var/www/html/nextcloud.
$ cd /var/www/html/
$ curl https://download.nextcloud.com/server/releases/latest.tar.bz2 | sudo tar -jxv
During the installation nextcloud might need to write or change some files within its installation folder. However, since we used sudo to unpack the archive the folder is owned by the root user. We need to change this.
$ sudo chown -R www-data:www-data /var/www/html/nextcloud/
Nextcloud comes with a web based installer that will do all the remaining steps for you. Open your browser and type the IP address of your RasPi followed by /nextcloud. You should now see the nextcloud install interface, as shown below. Here we need to change one important part: As data folder put the mounted RAID-1 volume /mnt/NAS. Fill in the rest with the MySQL details from before. The database user is nxcloud, the database name is nextcloud. The first user that will be created by nextcloud is the admin user (don't call it admin as username). I suggest to only use it for the setup, but not for uploading and managing data. We will create another nextcloud user later.
You should now be able to log into your nextcloud. There might be an error message that the datafolder (/mnt/NAS in our case) can be accessed by other users and nextcloud will not let us proceed. In this case, change the chmod and to make sure also the ownership:
$ sudo chmod -R 770 /mnt/NAS
$ sudo chown -R www-data:www-data /mnt/NAS
Back to your browser you should be able to proceed.
7. Speeding nextcloud up a bit
APCu and Redis
GPU RAM 16MB, SWAP
8. Installing and configuring samba
9. Changing the user permissions
This a very important step, since it ensures that samba and nextcloud can work well together.
The web server usually runs as it's own user and group, something like www-data:www-data. The same is true for the user we created for our samba access, it will be something like nas:nas.
Nextcloud also requires that the data-directory cannot be accessed by every user of the system. If you try to change the chmod of the nextcloud data folder to 777, nextcloud will give you an error and will not work. I think it requires at least 774 (no write permission for "other" users). This means the samba user "nas" cannot write anything into the mounted drive at the moment. You can try it with your PC, it should not work.
However, if we put the users "www-data" and "nas" into the same user group it will work. I decided to add both users to each others group, alternatively it should also be possible to create a new group and add both users to this (however it might require to remove them from their original group, not sure about this). The reason I did it this way is that maybe some other web application or apache mod only works if the user and group are www-data.
$ usermod -a -G nas www-data
$ usermod -a -G www-data nas
Now we should make sure that everything works. Copy a simple text file from your PC via samba to the NAS and open it in the nextcloud web interface. Maybe write something into it. You should also be able to create a new text file using the web interface and open it on you PC via samba. Again, maybe write something into it to see if everything works.
10. Forwarding your router ports
Every router has another interface and therefore works different in this regard. You might need the static IP address of your RasPi (see point 3) for this.
To access nextcloud from the web you need to forward ports 80 (http) and 443 (https). Normally, if someone puts you public IP address into their browser (or wherever) your router will block this request. If you forward http(s) traffic to your RasPi, every request that comes on port 80 and 443 will not be blocked by your router, but will be addressed to your RasPi. And since there is an apache server running you should see something. Requests on other ports, like 22 (FTP) will still be blocked by your router.
11. Securing your apache and nextcloud