09 April 2017

Full Bitcoin node on a Raspberry Pi 3 with or without a PiDrive (updated for UASF support)

For a long time the Raspberry Pi has seemed tempting for people who wanted to run a full node on the Bitcoin network because of its small size, low power use and cheap price-tag. With their relatively low performance specs, the first generations of the Pi were really not quite up to the task, though. Some people were able to compile and run Bitcoin Core (this very blog, for instance), but only barely, and others didn't succeed in making it work at all.

This has changed to the better with the newer generations of the Pi. The Raspberry Pi 2 and especially the Raspberry Pi 3 have more RAM and multi-core processors. While we're still far from modern desktop computers, of course, you're much better off with the current specs than before. That being said, syncing the blockchain can still be quite painful on a Pi 3.

With the blockchain growing larger and larger, you basically have two options. One option is to have enough storage space to contain the entire blockchain (and then some), be it by means of a really large Micro SD card, a Western Digital PiDrive like explained here, or a large USB pen-drive. The other option is pruning the blockchain, which became available in the Bitcoin Core software recently. Pruning is basically storing only the most recent part of the blockchain, and cutting away everything else.

Also, for Ubuntu, pre-compiled binaries of Bitcoin Core and its dependencies have been made available for the Raspberry Pi, which has made the process much easier since you no longer have to compile yourself if you don't wish to go through the trouble.

This guide will show how to set up a complete headless full node system with a Pi 3, either as a pruned node, or using a Western Digital PiDrive (or another USB harddisk) to store the blockchain. A Raspberry Pi 3 with a connected PiDrive will consume around 3-4 watt of power. A pruned node on just the Pi 3 is just above 2 watt. Compare that to the 100-200 watt that is consumed by a typical desktop computer running Bitcoin software with the screen turned off.

It will be a bit technical, but my goal was to make it quite comprehensive and explaining, so even if you have little to no experience with using Linux/Unix systems, you should be able to follow along, as long as you're not afraid of trying out new things.


Sections of this guide


The hardware requirements

Raspberry Pi 3: This guide will be using the Raspberry Pi 3 model B, but the process should be the same on the Pi 2, if you have that laying around. If you're going to buy a new Pi for this project, I recommend going for the Pi 3, since its price-tag is the same, and the specs are higher.

Micro SD card, min 4-8 GB (min 16 GB without PiDrive), class 10 or better: The Pi boots off an SD card. For that you'll need one that's fairly fast, so get a class 10 or better. 4 or 8 GB should be more than enough for setups with the PiDrive. This one is fine, but be aware that PiDrives often come with a 4 GB MicroSD card included, so you may not have to buy one separately. If you're planning on a pruned node without a harddisk, choose a fairly larger SD card, depending on how much of the blockchain you want to prune (also take into account the space taken up by the system plus a swapfile). 16 GB is probably the minimum without a PiDrive. Warning: Don't buy cheap SD cards off Ebay and elsewhere from China, as these are often small cards technically disguised as larger cards.

SD card reader: You'll need an SD card reader and a MicroSD card adapter or some other way to plug your Micro SD card into your ordinary computer during installation of the Pi's operating system.

WD PiDrive (optional if you're going to run a pruned node): Since SD cards of the required size to store the entire blockchain are quite expensive still, we will be using a harddisk to store the space hungry blockchain files. Western Digital's PiDrive is very fit for the purpose; it comes in sizes of 250 GB, 314 GB and even 1 TB. In the spirit of Pi, I used the 314 GB version (π ≈ 3.14, get it?), a storage capacity that at the time of writing can hold the blockchain three times, so there's plenty of space left for the rapid growth the blockchain. At the same time, it comes with a USB plug that fits right into one of the Pi's USB slots, instead of a SATA connector as usual, and a nice lead that allows you to power both the Pi and the PiDrive from the same power supply. Alternatively, you could use an external USB-harddisk of any make and model, or an internal harddisk with a powered USB adapter. Also, if you want to save on power, an SSD or a USB pendrive instead of a physical harddrive will be a good option. And of course, if you plan on running a pruned node or on getting a huge MicroSD card, you won't need an extra drive at all.

Micro USB power adapter, min 2.5 A: As the Pi models grow bigger in specs, so do their power requirements. Traditionally, Pi owners have used old phone chargers as a power supply. While you can still do that, you should be very cautious that the supply delivers enough power for the newer Pi models. A minimum of 2.5 A is the official recommendation. The easy route (and the safe bet) is to just get the official Pi 3 power supply which can power both the Pi 3 and the PiDrive at the same time.

Ethernet cable (optional): Even though the Pi 3 has built-in wifi support, I recommend using a cabled network for a Bitcoin node. For that you'll need a piece of Ethernet cable from your network router/switch to your Pi. The Raspberry Pi supports only 10/100 Mbit/s, so any category 5e or better is fine. If you must, you can of course use wifi instead, but in my experience, cabled network is just much more reliable.

Raspberry Pi case (optional): In order not to risk damaging the electronics parts of your Pi, I recommend to get a case for it. This might be anything from one of the many different cases that are available, or you might make a case yourself that fits both the Pi and your PiDrive.

Heatsink and/or fan for the Pi (optional): Either an inactive aluminium heatsink, a cooling fan (comes with some cases), or maybe both. The Raspberry Pi Foundation says cooling is not necessary on a Pi, and the Pi does throttle down performance when it gets too hot to prevent damage. However, especially when syncing with the blockchain, the Pi's CPU runs quite hot to a degree that I can't hold my finger on it for more than a couple of seconds. Also, I see the "thermometer" symbol on the screen (meaning performance is being throttled) quite often. After adding a heatsink and a fan I haven't noticed performance being throttled. It should be noted that after synchronisation is done, load on the CPU decreases drastically, and you will probably have no problems running without cooling day-to-day.

A spare HDMI screen, USB keyboard and mouse: These will only be used during setup, so don't go out buying these.

Several of these hardware items are available from online store Pi Supply. I am not affiliated with them in any way, other than I'm a happy customer there; they send worldwide, they have great customer service, and they take Bitcoin.


Initial setup

Download finished image file instead

Following this guide will take some time, especially for downloading the blockchain. For the impatient, I have made an almost ready-to-go image file with the (at the time of writing) updated blockchain files for a pruned node. Download this at the bottom of this guide.

⇓   ⇓

For this project, I have chosen to let the Pi run Ubuntu Linux instead of the otherwise very nice Raspbian, since that makes the installation of Bitcoin Core a bit easier.

Installing Ubuntu on the SD card is not too hard. It's a matter of downloading an image file, and writing that image file onto the MicroSD card that the Pi will boot off. Download the latest LTS (long term support) image file. The link here is actually Ubuntu Mate, which is normal Ubuntu with the Mate graphical desktop system, but since we won't be using the desktop system with the finished headless Bitcoin node, it doesn't really matter which flavour of Ubuntu it is. I also tried with the unofficial Ubuntu Server image without a desktop system, but I had boot problems after updating it, so I went for Ubuntu Mate instead.

After downloading the image file, it needs to be written to the MicroSD card. How this is done differs depending on whether you're on Windows, Linux, Mac, etc., but the Raspberry Pi Foundation has a very nice guide to doing that.

After writing the image file to the MicroSD card, insert it into the Pi, and also connect the PiDrive (unless you're going for the pruned node), Ethernet cable, keyboard, mouse, and screen.


Updating and configuring the system

Terms

The function of a Bitcoin node is to validate the blocks of the blockchain when they are found by the miners and distributed throughout the network. The more distributed nodes, the better, since a miner not playing by the book will have their invalid blocks rejected.

The term full node refers to the fact that it "fully validates transactions and blocks".[source]

A pruned node keeps only the most recent part of the blockchain. The node operator can set an amount of megabytes to use for storing the blockchain, and the node will delete old parts of the chain that exceeds this amount of storage. A pruned node will still have to download the entire blockchain at first run, though, but it will never use more harddisk space than was allocated for it.

A pruned node will in most cases also be a full node, since the "full" part does not refer to how much of the blockchain is stored on the node.

Plug in the power cable, and the Pi will boot up and you will be presented with a few dialogue boxes where you should set up your time zone, user account and password for the system and a few other things. The remainder of this guide will assume user name pinode, but you can of course use which ever user name you wish. You are encouraged to use a strong password, especially if you plan to be able to log in from the Internet. Once completed, you will be able to login to the Mate desktop.

Start up a Terminal (sort of a command line window; opened through the Applications menu, System Tools, MATE Terminal), and give the following command to let the Pi boot up into text mode in the future, since there is no need for a headless Bitcoin node to spend precious RAM on loading a full graphical desktop:

sudo systemctl set-default multi-user.target

Now, reboot the Raspberry Pi and make sure that you are logged in to the text based command line interface (all black screen with white text).

The next six commands will: 1: Tell the system from where it should fetch the Bitcoin software. 2: Update the system's information about which packages are available to download and install. 3: Download and install the text based version of Bitcoin Core, plus another tool that will be used next. 4+5: Remove the entire graphical desktop environment so that it doesn't take up space on the SD card. Run the last of these two remove commands repeatedly (maybe 4-5 times) until it doesn't remove any more packages. 6: Download and install all available updates to the entire system. Type them in one command at a time, as each command will ask you a question or two (typically just press Enter or Y to proceed). The last command will probably take quite some time to finish.

sudo add-apt-repository ppa:bitcoin/bitcoin
sudo apt update
sudo apt install bitcoind deborphan
sudo apt remove --auto-remove --purge libx11-.*
sudo apt remove --purge $(deborphan) #Repeat until no more is removed
sudo apt upgrade

Note: If you want to run an UASF enabled node instead of the standard Bitcoin Core software, in the first line change "ppa:bitcoin/bitcoin" into "ppa:luke-jr/bitcoin-core-bip148-unofficial-builds".

After that we will write a very short and simple configuration text file for Bitcoin Core. We will use the text based editor "nano" since it's very simple to use. Feel free to use any other text editor if you have another favourite, just as long as it writes plain txt files. These two commands create a .bitcoin directory (which is where Bitcoin Core will later put all the blockchain files) in your home directory, and then start up nano and open a new file for editing:

mkdir ~/.bitcoin
nano ~/.bitcoin/bitcoin.conf

In the nano editor, write the four following lines. Omit the last line (prune) if you are connecting the PiDrive and want to keep the full blockchain. Otherwise change 1000 into however many megabytes of the blockchain that your want to store at all times (at least 550). Keep in mind, though, that this number only controls the space that the block files themselves (the ~/.bitcoin/blocks directory) take up. Other than those, expect a couple of gigabytes or more for other data in ~/.bitcoin. Also keep thought of the space that the Ubuntu system itself takes up. And finally you will most likely be creating a swapfile that will take up around 2-3 GB. See how much free space you have on the partition that will keep the blockchain with the command "df -h ~./bitcoin" and then subtract at least 5-8 GB from that. The following should work on a 16 GB SD card.

dbcache=800
maxmempool=50
disablewallet=1
prune=5000

Hit Ctrl-X to exit nano, and then press the Y key and Enter to save the file.

In the Mate flavour of Ubuntu Linux, 64 MB of the shared memory is reserved for the GPU (graphics chip) by default. We can safely lower that to the minimum of 16 MB, thereby gaining 48 MB of additional RAM. Open the file /boot/config.txt:

sudo nano /boot/config.txt

Then scroll down to the bottom of this (rather long) text file and add the following line:

gpu_mem=16

Again, Ctrl-X and then Y + Enter saves and exits.


Getting the PiDrive ready

You can skip this entire section and go directly to "Adding a swapfile" if you go for a pruned node without additional storage!

The PiDrive comes preformatted for running on a Linux system on a Pi. Mine contained just one large "ext4" partition which is exactly what is needed.


Instead of a PiDrive you might also use any ordinary harddrive
with a powered SATA to USB adapter.

However, you might also want to use another kind of drive (external USB harddrive, an SSD, a USB pendrive, etc.), and those usually come formatted for Windows, or maybe not formatted at all. So assuming you already connected your drive to the Pi, what we need to do is remove any existing partition(s) and add one large "ext4" partition that fills out the entire drive.

In the end I will of course use my PiDrive, but for the sake of demonstration here, I connected a drive from an old Windows machine that was given to me, through a cheap externally powered SATA to USB adapter (important to get an externally powered one, as the Pi will not be able to deliver enough power for a harddrive). This drive contains several Windows partitions that I needed to remove.

First, to get an overview of the drives that are attached to the Pi, run the following command. Notice that the system also treats the SD card as a drive:

sudo fdisk -l

This gives me the following on my system with the old Windows drive connected:

The first disk, named /dev/mmcblk0 is the SD card, so don't do anything to that, or you will lose the Ubuntu system that you just set up. On my system, the next disk is /dev/sda, and you can see that it has four partitions named /dev/sda1, 2, 5 and 6 that need to be removed. Maybe you want to write down the names of the existing partitions on your harddisk, if any. Notice that ALL DATA that might already be on the harddisk will be lost, so don't use a drive containing anything you don't want to lose. Write the following command to enter the fdisk utility, changing /dev/sda into your harddisk's name if it is different:

sudo fdisk /dev/sda

Here, you will be met with a command prompt. To delete the existing partitions, type d and hit Enter, whereafter the prompt will ask you which partition number to delete, along with a list of the existing partitions numbers. Press a corresponding number and then Enter to delete the partition. Repeat until there are no partitions left.

Now, hit n to create a new partition. Then p to make it a primary partition, and 1 to give it partition number 1 (and hence the partition name /dev/sda1, if your disk name is /dev/sda). At the questions about first and last sector, just press Enter to make the partition fill out the entire harddisk.

Last, press w + Enter to save the changes to the disk and exit the fdisk tool.

To format your new partition, type in the following command (making sure to change the partition name approately, if your disk was not /dev/sda):

sudo mkfs.ext4 /dev/sda1

After a short while, you are ready to mount the partition. In the Linux world, drives don't mount to drive letters like in Windows. Instead you can mount a drive/partition to any place in the file tree. We will take advantage of this fact by mounting the harddisk partition to the ~/.bitcoin directory, which is exactly the place where Bitcoin Core will save its settings and store the blockchain. This way, the entire system is stored on the SD card (including Bitcoin Core itself), except for Bitcoin Core's data files.

To do this, you must first move the existing directory out of the way and create a new empty directory that the drive partition will be mounted to:

mv ~/.bitcoin ~/.bitcoin-old
mkdir ~/.bitcoin

Now, to tell the system to mount the drive to the newly created directory on every boot, you need to edit a short text file. Again, the text editor nano will come in handy:

sudo nano /etc/fstab

In the bottom of the file, write the following line (remember to change "sda1" if your harddrive partition was named differently, and also change "pinode" into your username):

/dev/sda1 /home/pinode/.bitcoin ext4 defaults 0 0

Then, press Ctrl-x followed by y and Enter to save the file and exit nano.

Test it out by typing sudo reboot to reboot the system. When the Pi has rebooted, log in and type df -h to see all mounted partitions. Your newly created harddisk partition should be in the list along with information that it is mounted to your .bitcoin directory:

Finally, make sure that the pinode user account (which will be running Bitcoin Core) has full access to the harddrive, and then move the bitcoin.conf file to the harddrive:

sudo chown pinode /home/pinode/.bitcoin
sudo chgrp pinode /home/pinode/.bitcoin
mv ~/.bitcoin-old/bitcoin.conf ~/.bitcoin/

Adding a swapfile

Now for an optional, but highly recommended step, namely adding a swapfile. The Pi 3 has 1 GB of RAM, which is an improvement to earlier Pi's, but still quite low for running Bitcoin Core as a full node, especially during synchronisation with the blockchain. A swapfile (or a swap partition) is a part of the harddrive (or in this case, the SD card) that acts as an addition to the RAM. The Ubuntu Mate system does not use a swapfile or swap partition by default. The reason that the step is optional is that swapfiles (and swap partitions) can be quite hard on an SD card due to the many read/write operations being performed, and SD cards only last for a certain amount of read/write operations. It's not like it will wear out in a few days or weeks or anything like that; just be prepared to only use your SD card for this project. The reason that adding a swapfile is highly recommended anyway is that chances are you will not be able to synchronise with the blockchain without it, since 1 GB RAM may simply be too little. However, we will set up the system to only use the swapfile if strictly necessary.

If you will be connecting a PiDrive, it is highly recommended to put the swapfile there instead of on the SD card. Just use /home/pinode/.bitcoin/swapfile instead of /swapfile in the rest of the commands in this section.

To set up a swapfile, perform the following commands. 4 GB RAM for a Bitcoin node should be more than fine, so given the Pi's 1 GB RAM, you can set up the swapfile to 3 GB:

sudo fallocate -l 3G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo nano /etc/fstab

The last command will open the settings file for partitions to be mounted at boot. Scroll down to the end and add the following line, then Ctrl-X, then Enter to save and exit the editor:

/swapfile none swap sw 0 0

Also, to let the system only use the swapfile if really necessary, open the following file in nano by performing:

sudo nano /etc/sysctl.conf

In the end of the file, add the following line:

vm.swappiness=1

Allowing SSH access from another computer on the network

We want to make the setup headless. This means that there are no monitors, keyboards, mice, etc. plugged into the Pi when it's running. So if we want to log in to the Pi anyway, we will make an SSH connection to it from an ordinary computer and get access to the Pi's command line this way. To allow for this, we need to get the local IP address of the Pi. Maybe you can see it in your router's user interface—otherwise start the Terminal on the Pi and run ifconfig. With a cabled connection, your IP address is in the eth0 section, with a wi-fi connection, it is in the wlan0 section.

We'll also make sure that the SSH server on the Pi is installed and started at boot:

sudo apt install openssh-server
sudo systemctl enable ssh

Now you can power down the Pi (if you are in text mode by now, you can do so using the command "sudo halt"), and you may remove keyboard, mouse, and screen. The only cables you need from now is the power and Ethernet cables, and of course the PiDrive if you're using that. If you want to log in to the Pi through SSH from another computer on the network, you may do so with (change "192.168.1.199" into your Pi's IP address, and "pinode" into the username you created on the Pi):

ssh pinode@192.168.1.199

This is assuming your other computer also runs some sort of Linux/Unix (maybe including MacOS?). If you're on Windows, you can use the PuTTY SSH client.


Letting Bitcoin Core start automatically on boot

Also, we want Bitcoin Core to start whenever the Raspberry Pi has been started up. And in addition to that, we will attempt to start it every five minutes in case it stopped for some reason. An easy way of doing that is through "crontab" which is a mechanism in any Unix/Linux system that can start programs and scripts at certain times and occasions. There may be other, more correct ways of doing this, but crontab seems to do the job. To edit the crontab settings, run:

crontab -e

If this was the first time you ran that command (it most likely was), you will be presented with a menu where you can choose a text editor to use. Again, if you have no other preferences, select "nano". Scroll all the way down in the file that nano opened for you, and add the following as the last two lines:

@reboot bitcoind -daemon
*/5 * * * * bitcoind -daemon

Open port in your router

In addition to your node making connections to other nodes, you will want other nodes to also be able to connect to your node. This is done by forwarding incoming traffic on port 8333 to the Raspberry Pi. How to do this varies from network to network, but typically involves entering the web interface of your network router or modem, and setting up "Port Forwarding" there. Typically this is done by simply adding the local IP address of the Raspberry Pi (which you found above, e.g. 192.168.1.199) along with port number 8333 to a list of forwarded ports. Refer to the documentation from your router/modem's manufacturer on how to do this exactly. Some routers even do this automatically so you don't need to do anything.

You can verify that it works by using Bitnodes' "Join the network" feature. If it returns a line that includes the word "Satoshi", you have successfully forwarded traffic on port 8333 to your Pi. Notice that while your Pi is still synchronising the blockchain, it might be lagging so much that Bitnodes will report it as unreachable. Try again after synchronising is done.


Start your new full node

Now it's time to plug your node in, and let it synchronise with the network. This will probably take several days. You can keep track of the progress by logging in (remotely or with a keyboard) and running the command:

bitcoin-cli getinfo

Compare "blocks" line with the most recent block number at your favourite block explorer (such as Blockchain.info).

Some have reported that their node stops synchronising at around block 300,000. If this is the case for you, you might want to replace your ~/.bitcoin directory with my (at the time of writing) fully synchronised directory, or even my complete SD card image file that you can download below.


Download and use my (almost) ready-to-go image instead

I've already done the work, so why should you? Well, because you might not trust me to not offer a malicious Bitcoin node that will break into your private network and steal all your Bitcoins and private data. And why should you; after all you don't know me.

Anyway, for the brave few, I've completed this guide for you, let the node synchronise with the Bitcoin network, and made an image file of my SD card available to download.

Download complete Pi 3 image

The image file is for a pruned node without any external drive (such as the PiDrive) attached, ready to be used on a Raspberry Pi 3 with an SD card of at least 16 GB:
Download my (almost) ready-to-go pruned Bitcoin node SD card image
SHA256: b9ce45dd2af0c3ab8d07ec31ea058bbaf92f641c679cc2225726113fffd5fd66
MD5: d3d33be1a85df13629c5d281bfc1c5b4

After transferring the image to your SD card, you are urged to log in (username pinode and password pinode) and change the password (you will be asked to change it when logging in for the first time) and then run the system update commands below (this will also ensure that Bitcoin Core is updated, in case a new version has been released since this guide was written):

sudo apt update
sudo apt upgrade

You also still will need to open a port in your router as described above.

Download only the blockchain for your node

You might also want to set up the system yourself and only download my (at the time of writing) fully synchronised ~/.bitcoin directory to save on sync time (which can be long). Transfer the compressed file to your Pi through SCP (file transfer over SSH), copy the directory on to your SD card through your SD card reader on your normal computer, or download it directly from the Pi:
Download pruned ~/.bitcoin directory
SHA256: dd6ffcce71590f7a72356182ff80df00c11408a54b122305969fdb2df975fe63
MD5: c36816adf5ca4fb9ea515b871e318bfa

With the compressed file in your home directory (~) on your Pi, remove your old bitcoin directory (backup your bitcoin.conf file first if you made changes to it other than described in this guide) and extract the file to the right place (might take some time) with:

cd ~
rm -rf .bitcoin
tar -zxvf bitcoindirectoryfilename.tar.gz

Thanks!

If we're to conclude anything from this, it must be that while installing and setting up a Raspberry Pi 3 system to act as a Bitcoin node is a fairly smooth procedure, the relatively low spec'ed hardware makes it quite painful to actually synchronise with the blockchain. Possible yes, but still painful. Especially the last couple of hundred blocks seem to take forever and then some. When the node is synced up, though, it seems to run without any problems.

I hope you liked this guide as much as I liked writing it. If anything is unclear, or you have any tips to share, please feel free to do so in the comments below. Also feel free (but not obliged) to donate an amount of your choice.

Disclaimer: Use this guide at your own risk. The author of this guide is in no way responsible for any loss, even when following the guide to the last comma. You're responsible for your own data and money. Don't do anything with it that you don't feel 100% comfortable with, and only test things out using small amounts.

31 comments:

  1. You are a hero. The ready 2 go image is excellent...even for me as an IT guy...I just dont feel like going through the manual. Just plug and play with the image :)

    ReplyDelete
  2. Wow! This is a great tutorial! I've read many, I'd have to say this one is the most detailed and complete.

    I already got my first node running a few weeks ago. (I sure wish this tutorial was around then.) One thing I never saw in any of the other tutorials was about the auto start and re-start every 5 minutes.

    I added this step to my Raspberry Pi3, but it doesn't seem to be working. I added your 2 line to my crontab file (there were a bunch of commented lines in there already), and I can even list the file with the 'crontab -l" command. Everything looks good, but when I reboot, btcoind -daemon doesn't run. I wait more than 15 minutes, and it's still not running. (I check for it running with "bitcoin-cli getinfo".

    Any Ideas what could be wrong? I'm using the latest Raspbian Jessie Pixel (with GUI) which boots straight to the CLI. I even did a cold boot, and still nothing. Help, please.

    ReplyDelete
    Replies
    1. Thank you for your nice words. Some questions to help me understand what happens: If you do a 'ps -ux | grep bitcoind' then what's the output? If you do a 'top' do you see bitcoind close to the top of the list at most times? If you do 'grep CRON /var/log/syslog' what's the output? Also, it might help if you copy/paste the output of 'crontab -l' here (maybe except the commented lines). Can you run bitcoind manually from the CLI, and what happens when you try?

      It *might* be that bitcoind is actually running, but in my experience, if it is synchronising with the blockchain, it will be extremely laggy to the point that it may seem completely unresponsive, even to requests from the bitcoin-cli tool. Once it has finished synchronising, at least my node acts as if running bitcoind is the easiest thing in the world.

      Delete
    2. My RP3 boots to CLI. I typically start the node with the command: bitcoind -deamon / this works fine every time. I can stop the server with bitcoin-cli stop, and start it again, no problem.

      NOTE: I did set the crontab to run bitcoind -daemon at boot and every 15 minutes instead of 5 minutes. That's the only thing I changed (the 5 to 15), but it doesn't auto start at boot, and doesn't try to restart every 15 minutes.

      It is NOT running for sure. I know to wait. I tried bitcoin-cli info 5 minutes later, 15 minutes later, 30 minutes later.... It is NOT running.

      Delete
    3. Here is the info you wanted...

      ps -ux | grep bitcoind
      pi 813 9.3 36.3 543384 362344 ? Ssl 11:04 12:37 bitcoind -daemon
      pi 1038 0.0 0.1 4276 1912 pts/0 S+ 13:19 0:00 grep --color=auto bitcoind

      If I do a "top", bitcoind is at the very top of the list almost all of the time.

      Delete
    4. grep CRON /var/log/syslog
      Apr 10 07:17:01 BN-RKY CRON[11067]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
      Apr 10 08:17:01 BN-RKY CRON[11083]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
      Apr 10 09:17:02 BN-RKY CRON[11178]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
      Apr 10 10:17:01 BN-RKY CRON[11239]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
      Apr 10 10:34:42 BN-RKY cron[401]: (CRON) INFO (pidfile fd = 3)
      Apr 10 10:34:42 BN-RKY cron[401]: (CRON) INFO (Running @reboot jobs)
      Apr 10 10:34:42 BN-RKY CRON[487]: (pi) CMD (bitcoind -daemon)
      Apr 10 10:34:42 BN-RKY CRON[428]: (CRON) info (No MTA installed, discarding output)
      Apr 10 10:45:01 BN-RKY CRON[805]: (pi) CMD (bitcoind -daemon)
      Apr 10 10:45:01 BN-RKY CRON[801]: (CRON) info (No MTA installed, discarding output)
      Apr 10 10:45:28 BN-RKY cron[400]: (CRON) INFO (pidfile fd = 3)
      Apr 10 10:45:28 BN-RKY cron[400]: (CRON) INFO (Running @reboot jobs)
      Apr 10 10:45:28 BN-RKY CRON[492]: (pi) CMD (bitcoind -daemon)
      Apr 10 10:45:28 BN-RKY CRON[428]: (CRON) info (No MTA installed, discarding output)
      Apr 10 11:00:01 BN-RKY CRON[802]: (pi) CMD (bitcoind -daemon)
      Apr 10 11:00:02 BN-RKY CRON[798]: (CRON) info (No MTA installed, discarding output)
      Apr 10 11:15:01 BN-RKY CRON[842]: (pi) CMD (bitcoind -daemon)
      Apr 10 11:15:01 BN-RKY CRON[838]: (CRON) info (No MTA installed, discarding output)
      Apr 10 11:17:01 BN-RKY CRON[852]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
      Apr 10 11:30:01 BN-RKY CRON[877]: (pi) CMD (bitcoind -daemon)
      Apr 10 11:30:01 BN-RKY CRON[873]: (CRON) info (No MTA installed, discarding output)
      Apr 10 11:45:01 BN-RKY CRON[895]: (pi) CMD (bitcoind -daemon)
      Apr 10 11:45:01 BN-RKY CRON[891]: (CRON) info (No MTA installed, discarding output)
      Apr 10 12:00:01 BN-RKY CRON[913]: (pi) CMD (bitcoind -daemon)
      Apr 10 12:00:01 BN-RKY CRON[908]: (CRON) info (No MTA installed, discarding output)
      Apr 10 12:15:01 BN-RKY CRON[930]: (pi) CMD (bitcoind -daemon)
      Apr 10 12:15:01 BN-RKY CRON[926]: (CRON) info (No MTA installed, discarding output)
      Apr 10 12:17:01 BN-RKY CRON[940]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
      Apr 10 12:30:01 BN-RKY CRON[965]: (pi) CMD (bitcoind -daemon)
      Apr 10 12:30:01 BN-RKY CRON[961]: (CRON) info (No MTA installed, discarding output)
      Apr 10 12:45:01 BN-RKY CRON[983]: (pi) CMD (bitcoind -daemon)
      Apr 10 12:45:01 BN-RKY CRON[979]: (CRON) info (No MTA installed, discarding output)
      Apr 10 13:00:01 BN-RKY CRON[998]: (pi) CMD (bitcoind -daemon)
      Apr 10 13:00:01 BN-RKY CRON[994]: (CRON) info (No MTA installed, discarding output)
      Apr 10 13:15:02 BN-RKY CRON[1013]: (pi) CMD (bitcoind -daemon)
      Apr 10 13:15:02 BN-RKY CRON[1009]: (CRON) info (No MTA installed, discarding output)
      Apr 10 13:17:01 BN-RKY CRON[1021]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

      Delete
    5. crontab -l
      # Edit this file to introduce tasks to be run by cron.
      #
      # Each task to run has to be defined through a single line
      # indicating with different fields when the task will be run
      # and what command to run for the task
      #
      # To define the time you can provide concrete values for
      # minute (m), hour (h), day of month (dom), month (mon),
      # and day of week (dow) or use '*' in these fields (for 'any').#
      # Notice that tasks will be started based on the cron's system
      # daemon's notion of time and timezones.
      #
      # Output of the crontab jobs (including errors) is sent through
      # email to the user the crontab file belongs to (unless redirected).
      #
      # For example, you can run a backup of all your user accounts
      # at 5 a.m every week with:
      # 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
      #
      # For more information see the manual pages of crontab(5) and cron(8)
      #
      # m h dom mon dow command
      #
      @reboot bitcoind -daemon
      */15 * * * * bitcoind -daemon

      Delete
    6. The 'ps' output clearly shows that bitcoind is running (as process ID 813, the first line), and you wouldn't see it in 'top' if it wasn't. Also, your crontab looke fine, and cron doesn't seem to output any errors to the syslog.

      My experience is, though, that while bitcoind is catching up with the blockchain (which can take days, or even weeks, depending on how much it has already downloaded and checked), the system can be extremely laggy, and even 'bitcoin-cli getinfo' will appear to freeze for a couple of MINUTES before giving a response and returning you to the CLI. Once bitcoind has caught up with the blockchain, the system becomes very responsive in my experience. You don't write what 'bitcoin-cli getinfo' actually outputs, just that you didn't have any luck with it. Can you copy/paste here the output of 'bitcoin-cli getinfo'? If it doesn't output anything but just refuses to let you type another command, then please wait some more minutes. It may take long before it outputs anything.

      Delete
    7. I'm sorry.... The "ps" and "top" commands only shows it running because I ran those commands AFTER I started the node from the CLI. It was NOT running because of reboot or my 15 minutes setting.

      I had this node running just fine several weeks before I found your tutorial. I only wanted to add the CRONTAB to auto-start at boot, and again every 15 minutes just it case it shut down. So the blockchain download was finished a couple of weeks ago, and "bitcoin-cli getinfo" showed all the info very quick, and the "block" number was showing the current block.

      I'm sorry for the confusion. But the must be something else wrong.

      Delete
    8. Oh... Yes, obviously the outputs show bitcoind running if you started bitcoind manually. And yes, I agree that something else much be wrong. I am not able to identify the problem from this, though. Try putting the following line in your crontab, just to test it out: "@reboot echo "Bitcoin node booted at $(date)" >> /home/username/bootlog.txt" (change "username" into your user name). This should create a file "bootlog.txt" in your home directory and add a line to it at every boot. If no file turns up after a reboot, then the problem is with cron or your crontab.

      Delete
  3. OK. I'll try this in a few days. I had a memory issue (had to make a 3GB swapfile, like you recommend) and that seems to make my Raspberry Pi 3 run a lot better. No more crashes (so far). But I had to re-download the blockchain. Looks like it should be done in 2 or 3 days. When it's done, I try the "@reboot echo" command you talked about above and let you know if it works.

    Thanks!
    NodeAdmin

    ReplyDelete
  4. Thank you for this excellent guide. I am using a Raspberry Pi 3B with a PiDrive 375GB and Ubuntu MATE. Having slavishly followed every detail, my node was building nicely for the first 18 hours and I was able to check progress intermittently with < bitcoin-cli getblockchaininfo >. At the last count I had over 270k blocks built.

    Some hours later I noticed that both red and green LEDs on the Pi were continuously on and the LED on the PiDrive was flashing in a particular pattern. Whereas it had worked during setup I could no longer connect to the Pi with ssh from my Win10 desktop using CYGWin. This situation persisted for a some hours until I suddenly noticed the green LED on the Pi was flashing and quickly managed to get a successful ssh session going.

    The Pi reported that "25 packages can be updated" and "3 updates are security updates" and "System restart required". So I did < sudo apt-get update > and when it had finished < sudo reboot >.

    On re-establishing ssh I entered < bitcoin-cli getblockchaininfo > only to be told "error code: -28" and error message: loading block index..." followed by being returned to the command prompt.

    Currently bitnodes.earn.com is reporting that the node is unreachable - which is not inconsistent with your guidance that the blockchain probably needs to be completed first.

    Any suggestions as to what caused the apparent lock-up and how I might get going again?

    Thaks
    Edward

    ReplyDelete
    Replies
    1. Hi Edward

      Thank you for the nice words. What you're experiencing sounds like what I also experienced during sync with the network. The first part of the sync probably runs without problems, but as the downloaded chain get increasingly longer, and blocks start to get larger, the sync/validation process really starts to bring the Pi to its knees. In my experience, to a degree where it will be unresponsive for whole periods of time, like you describe.

      Are you running with active cooling, i.e. a fan? For me this helped somewhat, but I still experienced the unresponsiveness.

      After rebooting, getting the "error code: -28" and error message: loading block index" is normal. This just means the bitcoin daemon can't give you stats right now since it's currently reading the parts of the blockchain that are already on the system, as part of its initialization process after being started recently. Of course, on a small Pi, "recently" will stretch quite farther in time that it would on a full desktop computer.

      You will probably benefit from doing the synchronization process on a larger computer, and then transfer the blockchain from that computer to the PiDrive and let the Pi take over from there.

      Hope that helps,

      Delete
  5. My blockchain acquisition rate is very slow on 4th day of running the program. At block 329664 the rate of acquisition has dropped below 500 blocks/hour, implying I have over 300 hours to go if the acquisition rate holds. I didn't see any code in your excellent guide to force the Pi's quad processor to use all 4 cores. Might this be a way of hurrying things along a bit? My broadband download speed is typically reported at 16Mb/s or slightly better.

    ReplyDelete
    Replies
    1. Hi Edward

      You have a good point, although I haven't tested this myself. However, as far as I know, bitcoind utilises the boost library to always use all available threads by default, but you may get different results if you run bitcoind with the "par" parametre, as e.g. "bitcoind par=4" to force 4 threads/cores.

      It may also be that your Pi is already using all cores, but is running hot and therefore throttles CPU cycles to (much) less than 100%. If you're not doing this already, you may consider adding some active cooling, at least for the synchronization process. Or to do the sync process on a full-blown computer as per my earlier reply.

      Delete
  6. Thanks for the guidance. Using the command I can see that the CPU is typically running at about 90% suggesting that it is likely to be using all cores; the SSD swap file is being used up to about 10% of the allocation and bitcoind is rightly at the top of the process list most of the time. Now that I know it is acquiring and performing to expectations I might let it catch up on it's own just out of interest (now 12 days in and 78% complete by block number). If the rate of acquisition continues to decay the import option might become compelling.

    ReplyDelete
  7. Thank you for the guide. It was the only one out of about 6 I tried that worked.
    How do I upgrade when a new core comes out? Got a guide for that?

    ReplyDelete
    Replies
    1. Hi Pete, that's actually quite easy. To upgrade the whole system, including bitcoin core, you run the following two commands, one by one:

      sudo apt update
      sudo apt upgrade

      The first command fetches info about which updates to programs are available. The second one fetches and installs the actual updates.

      Note, though, that updates to bitcoin core will often only be available for Ubuntu on the Launchpad servers some days, or even a couple of weeks, after the release. You can always for the latest version which has been packed for Ubuntu on the following webpage: https://launchpad.net/~bitcoin/+archive/ubuntu/bitcoin

      Hope this helps,

      Delete
    2. Yes it does. Thank you.
      I have read having your own node gives you a wallet, and the ability to verify my own transaction first. With no GUI is that even possible with this node?
      Thanks!

      Delete
    3. Well, technically it does give you a wallet, except if you followed this guide, you have put the "disablewallet=1" setting in your bitcoin.conf file. I'm not sure whether one is still able to handle this wallet from the command line in newer version, but I suppose so. Even so, it not recommended unless you really, really want to be able to control every technical detail of creating a transaction, etc.
      What you probably want to do instead, and this may be what you read about, is to point your regular wallet software, running on another computer, to your node, so that it contacts your node (and perhaps ONLY your node) for interaction with the Bitcoin network. How this is done (and IF it can be done) depends on your wallet software.

      Delete
    4. Gotcha! I continue to dive deeper into commands available to me. One final thing and I will stop pestering you. The code that was downloaded from the repository at this point in your guide:

      "sudo add-apt-repository ppa:bitcoin/bitcoin"

      appears to have downloaded the 0.15.1 non Segwit version. Shouldn't we all be running 0.16.0 Segwit for a healthier system.

      Delete
    5. Hmm, you're right. Checking https://launchpad.net/~bitcoin/+archive/ubuntu/bitcoin/+packages it does seem like packaging Bitcoin Core 16.0 for Ubuntu 16.04 (codename Xenial) was indeed unsuccessful for the Pi's ARM architecture ("armhf"). I wonder if this is something Matt Corallo (Ubuntu packager) is working on.

      Once a working package is uploaded to Launchpad, you will get it with the apt update/upgrade set of commands described above.

      Another option is to wait for Ubuntu (Mate) 18.04 (codename "bionic") to come out. 18.04 for normal Intel architecture computers was released a week ago, so it will probably soon be out for the Pi's ARM architecture too. It looks like packaging Bitcoin Core 16.0 for armhf on bionic was successful.

      You can check which release of Ubuntu Mate is the latest one for Raspberry Pi at https://ubuntu-mate.org/download/ . Once you see it's 18.04, you have two choices for upgrading. The safe one: Start all over with this guide(!). The more risky one, but way easier if it goes right: Running the release upgrade tool which should handle everything for you. You can run the release upgrade tool by executing the following command:

      sudo do-release-upgrade

      (or you can just run the command to see whether there's a new release ready, if not it will tell you and exit)

      Delete
    6. HELP! Everything locked up about block 384K

      I rebooted the pinode twice and now get this error:

      pinode@pi:~$ bitcoin-cli getblockcount
      error code: -28
      error message:
      Loading block index...

      Delete
    7. That is probably to be expected given the Pi's modest specifications. Syncing really takes it to its limits, and sometimes beyond, which is probably what you experienced. As someone else who just finished syncing wrote in these comments: "about 11 weeks including a handful of stalls that required restarting the program." After syncing is done, the Pi will have much more air to breathe and will not struggle to keep up.

      The error message you see is not really an error as such. Just a message telling you that the parts of the blockchain that you have already synced are being read from disk into memory. This will happen every time you start bitclin core, whereafter it will continue syncing from where it left off. In my opinion the message ought not say "error", but maybe "warning", or "info" instead.

      Delete
  8. Just reporting back that I didn't take any short cuts in the end. I just allowed my node to complete syncing in its own time - about 11 weeks including a handful of stalls that required restarting the program. Since then it has run and kept up without any problems at all. I am impressed to have achieved activation of my own full node, with your guidance, for a total capital outlay of under GBP100.

    ReplyDelete
    Replies
    1. The sync process is indeed extremely demanding of the Pi, and I am not surprised that you had to restart the bitcoind program several times. What I would really like to see from coming revisions of the Pi is more RAM, at least 2, or better yet 4 GB.

      Congratulations on successfully setting up your own node. Always very nice to hear that what worked for me also worked for others.

      Delete
  9. Update on my sync. It has been about two weeks and my pinode is up to block 469,578. I have rebooted it only 2 or 3 times. Fingers crossed it just keeps chopping wood. It definitely slows down the farther it goes.

    ReplyDelete
  10. I want to be able to upgrade to bitcoin 0.16.3 but am unable to upgrade Ubuntu Mate from 16.04 to 18.04 due to space limitation in /boot partition - only about 5MB short after removing old kernels but for some reason I can't edit COMPRESS=gzip to COMPRESS=xz in the initramfs.conf file to release more space as suggested elsewhere.
    Do you have any suggestions?

    ReplyDelete
  11. Hi again, I have been running this node since build in early spring. I ran updates and upgrades as directed but I looked at the node from the outside and it is still running 15.1 That is probably very bad. How do I upgrade the bitcoin core program? Anyone?

    ReplyDelete
  12. Im still pretty green with this stuff but if I download your full blown synced.zip file. I'm assuming I put it in a new .bitcoin folder and reboot the Pi? Or is there an install it run command? Thank you for this guide

    ReplyDelete
    Replies
    1. In the download section I list two files for download:

      1. "pruned ~/.bitcoin directory": Use this if you already have a Pi up and running with Bitcoin Core installed. The contents of the zip file does indeed go into the ".bitcoin" folder in the home directory. If this folder is not there, it will be created automatically the first time Bitcoin Core is run. Or you can create it yourself.

      2. "(almost) ready-to-go pruned Bitcoin node SD card image": This zip file contains an "img" file, ready to be written to a clean SD card. It contains a complete Linux installation, including the Bitcoin Core software and a (pruned) version of the block chain.

      PLEASE NOTE, though, that either file contains an OUTDATED version of the block chain (and the latter file also outdated versions of Linux and Bitcoin Core) since they are from when this article was written almost 2 years ago. You can, of course, still use them. They are as valid as they ever were. However, be prepared to let Bitcoin Core download the last two years worth of block chain once it's finished checking the pre-downloaded part.

      Delete