Restoring Jeedom from a Debian 8.5 to a Debian 10.3 Virtual Machine

We will here install the excellent Domotic/Home Automation system called Jeedom from an old Debian 8.5 on a fresh new Debian 10.3 Virtual Machine. Actually in this post I will not only install it, but i’ll also restore a backup of my fully functional Jeedom V3 installation that use to run on a Debian 8.5. I decided to upgrade by installing a full new system as I ran into more & more problems with Python dependencies each time I had to upgrade the packages, or some Jeedom plugins.

VM Configuration

For my Virtual machines, I run vSphere 6.5 on an Intel NUC, this is perfectly supported (except the sound card but we don’t need it for this use).

Here is the configuration for my actual Debian8.5:

First i’ll deploy the netinstall version of Debian 10.3 on my vsphere datastore. Then i create a new VM with the same capacity of the old one, it runned smoothly.

Note that i’m not running the latest vSphere update … Debian 10 is not known in the dropdown list. It’s not a problem.

I’ll use my “datastore 1” datastore, it is an SSD drive.

Same settings as my Debian 8.5 VM. Note that you need to change the dropdownlist selection inside the CD/DVD Drive, to use your Debian iso file downloaded & uploaded on your vSphere datastore earlier.

Once created, here is the list of all my VMs. I made some cleanup before. Notice I’m also upgrading my gekko trading bot VM from a Debian 9.3 to a new 10.3 VM. Now let’s start our Jeedom 10.3 VM.

Debian Installation

I dont need any graphical install … Let’s go for text install. I won’t show every screen, but basic options I choosed:

• French language, timezone, keyboard (nobody’s perfect),
• DEBIAN10-3-JEEDOM as hostname
• Domain: home
• After setting the root password, I created a new ‘jeedom’ user
• I choosed LVM to partition the disk. I did not do it on my previous 8.5 install (don’t remember if the choice existed), but it will be easier later if we need to add some disk space or change existing one.
• I choosed to have /home, /var and /tmp in separated partitions. It is much better to prevent applications in your various ~home dirs to fill all your disk space, or logs in /var/log, or various stuff in /tmp
• To be honest, after two attempts to restore Jeedom, I needed to modify the LVM volumes inside the install to delete the default home and var logical volumes, in order to recreate them and allocate 5Gb to var, as my old Jeedom database is huge and the first time it completely filled the /var directory
• Then I Next, next, next …
• The install will download & install the required packages
• In my case I didnt need a proxy server, no desktop environment
• BUT i DO want a web server and an ssh server

The install process will again download & install the required packages. I’ll choose to install GRUB on /dev/sda as it will be the only OS on this VM.

Now we reboot.

Then i’ll connect on the VM with the VMWare console, and update  the VM with:

Installing the ‘net-tools’ package will allow us to use ‘ifconfig’ to get our IP address.

Hint: this is the moment you want to define this IP address as a static one on your router so it won’t change in the future. This is a must for jeedom as much equipments or routines will refer to your box by using its IP address.

We also want to install the VMWare tools. I won’t install the ones from my vSphere installation since it is not up to date. I’ll install the Open VM Tools.

Notice the difference on your vSphere UI (maybe after the reboot we will do later):

We now see the IP address, and vSphere does see the VMWare tools installed. Perfect.

When prompted, press y to continue the installation, it will install a few dependencies, and when it’s finished, as we installed a buch of things, I suggest to reboot the VM by using:

Now you should be able to SSH to your VM. I’m using Putty from a W10 laptop.

Jeedom installation, Mysql tweaking, Apache2 configuration

What is following is extracted from various parts of the Debian installation process from Jeedom documentation (§1 & §4 + some links from it). I won’t follow everything as the Jeedom documentation (as far as I remember) will install jeedom in the default Debian /var/www directory, and I changed that on my previous installation: the whole Jeedom installation website will in fact be located in /home/jeedom (root of my jeedom user). Yes it does require a few changes in the default httpd (Apache) conf file. 

The lines above will install sudo (then add our jeedom user created during the VM installation to the sudo group) and fail2ban (to limit the rate of connexions on opened ports from the exterior) and ffmpeg as it will be needed later by Jeedom (replacement of libav-tools). Now as I dont want my sudoers to enter the root passwd, i’ll edit the /etc/sudoers file and change (or comment) this line:

With this one:

Now with the user jeedom you should be able to perform root actions by using (for example) ‘sudo ls -al /root’.

Then we will install Jeedom once before we will restore our backup.

Note #1: in case you are installing jeedom on an already existing system with a running version of mysql, you can pass arguments to the script (eg. ./install.sh -w /home/jeedom/html -m Jeedom) as I did to directly deploy it in /home/jeedom/html and avoid a few operations to move it later.

Note #2: in case you want to reset the previous Jeedom installation steps and rerun it with a clean database & target html dir, perform the following steps:

1. Remove the Mysql jeedom database & user
2. Clean the /var/www/html directory OR /home/jeedom/html in case you modified WEBSERVER_HOME in the Jeedom install.sh file

Running the install script will install a bunch of packages: mysql, php, python dependencies, etc. At the end it should display a successfull message, and you need to note the root mysql password displayed, in case you didn’t define it by passing it as seen above as an argument to the install script.

We will now change the root passwd for mysql, to restore the one used on our Debian8.5 installation (hint: you can see it from your Debian 8.5 Jeedom UI, in the configuration menu, then OS/DB tab), and give more privileges to the jeedom user (this is needed to restore some stuff during the old Jeedom database restoring).

Here you need to use the root password for the Mysql Database you defined in your install script, or noted after the Jeedom installation.

Next, I’ll want to move the default mysql database files from the default dir /var/lib/mysql to my /home/jeedom HOMEDIR, where there is more space.

First we make sure about the default mysql datadir used for storing data.

We stop mysql and check it is well stopped (look for a line “Status: “MariaDB server is down”” in the systemctl status output).

We’ll copy the existing database directory to the new location with rsync. Using the -a flag preserves the permissions and other directory properties, while-v provides verbose output so you can follow the progress. And we rename the old database with an extension, so that we we ensure mysql can not use this one anymore. We will remove it after.

Now I’ll edit the /etc/mysql/mariadb.conf.d/50-server.cnf file (Debian specific) to change the datadir directive and point it on my new location, and edit /etc/apparmor.d/tunables/alias to create a pointer between mysql old dir and new dir.

And we restart AppArmor.

Before restarting mysql we will need to create a basic directory structure in the old files directory as it checked by the default mysql startup scripts, and we will need to authorize Mysql/MariaDB to be run from the /home directories by addind an authorization.

Then we need to reload systemctl configuration files.

And we restart Mysql, and check it is running well.

OK there are some complains about the mysql update, but it’s launched and working. Now a quick sanity check inside mysql to verify it is using the new datadir.

Perfect. We can now remove our old database directory.

Now we will connect to our fresh Jeedom UI (by using http://<our vm ip>), and go for the graphical installation (sorry my screenshots are in French). Default Jeedom login/pass is admin/admin.

Next we change the default password. I’ll reuse the one i’m using on my other Jeedom installation. And in next screen i’ll tell Jeedom what is my login/pass for Jeedom’s market. If you don’t have one yet there is a link to create one.

After validating the account, Jeedom will display its start page, very empty but that’s normal.

So, at this stage, we have a working standard Jeedom V4 installation, listening on port 80 of our VM. Fine but not enough.

Now, i’ll tweak the Apache2 configuration to reach /home/jeedom/html as default directory, enable HTTPS and use personal TLS certificates bought on Gandi.

Note that if we want to get more info in Apache2 log files, we will need to edit the /etc/apache2/apache2.conf “LogLevel warn” line to “LogLevel debug”.

In /etc/apache2/apache2.conf, I need to add the following lines:

I check /etc/apache2/ports.conf to verify the default HTTPS listener port.

I’ll modify the ErrorLog directive in /etc/apache2/conf-available/security.conf so that it will log in /var/log/apache2, not in /var/www/html/logs.

Now I modify /etc/apache2/sites-available/000-default.conf to use /home/jeedom/html as root dir and modify the log files.

I modify /etc/apache2/sites-available/default-ssl.conf to configure our HTTPS listener with personal TLS certificates bought on Gandi for my fqdn.

In this file notice the use of a few specific files for my installation:

Those are the files i’ll later need to backup on my Debian8.5 apache2 dir and restore on my Debian10.3 apache2 dir.

We’ll also enable SSL and make the default-ssl.conf file loadable by Apache2 by linking it from sites-available to sites-enabled:

I backup the SSL certificates used on my previous Jeedom installation, copy it on my new VM, and restore it. On the old Debian 8.5 VM:

On the new Debian10.3 VM:

Then I check Apache configuration, and I restart Apache.

Now i’ll check if my ports 80 and 443 are opened:

The output is good:

Now let’s try to reach Jeedom UI (by using http://192.168.1.82):

And now by HTTPS (by using https://192.168.1.82):

This warning is normal as i’m reaching the UI by using the VM’s IP, not the FQDN defined in the TLS server certificate I told Apache2 to use in my default-ssl.conf file. When I validate the warning I can access Jeedom’s UI this is perfect.

Now i’ll want to backup & restore the little script used to update my (unfortunately) dynamic public IP address on Internet so that I can permanently reach Jedom from the outside, and other small stuff.

On the Debian 8.5 box:

On the Debian 10.3 box, i’ll copy it with scp, untar it, and install a required python package to make it work.

Then I need to add a crontab job to check my ipaddress & eventually update it:

Then add this line:

Zwave stick & Jeedom Backup

Now I will backup my Aeotec Zwave USB stick, and my Debian 8.5 Jeedom installation.

About the Aeotec ZWave stick (thanks to Nechry an active contributor for Jeedom about his article):

• Windows 10 will already have the driver or it will be downloadable automatically, for other OS or to get the inf files manually check here
• you’ll need the Network Key for the stick, it will probably be the default key, but in doubt in my old Jeedom installation i’ll check it in resources/openzwaved/ozwave/manager_utils.py, it will be a suite of 16 hexa values « 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10 » which will need to be converted as « 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 » in the backup utility

On my W10 laptop, on which I never plugged this USB stick, Windows automatically configured the device, and then I launch the zwave_500.exe utility downloaded from Aeotec’s website.

I can check the stick is well detected because it displays a COM port in the title bar, and a “Load Zstick Success” in the status bar at the bottom of the window.

Then I need to check « Enable Security » and enter my network/security key and click “set”.

Then I click on “Read Eeprom” button and choose where to save the backup file. And get a coffee …

Jeedom backup

Next I need to save my whole functional Jeedom, through its integrated backup functionality.

Jeedom restauration

First, i’ll go back in vSphere administration UI to reallocate my Aeotec Zstick, and bluetooth module to the new VM. This is really simple, just unallocate them from the old DEbian8.5 VM, and reallocate them as a new USB device to the new VM.

I copy the backup on my Debian 10.3 Box, in the standard backup directory of Jeedom.

Then i’ll restore it through my new Jeedom’s UI (sorry this is a Jeedom v3 configuration > backups UI screenshot, forgot to take the one from the fresh V4 … But it’s almost the same).

It will be a long task, and the UI may not update well as files will be deleted and restored. You should be able to follow the restauration process through internal jeedom log file:

OK so obvisouly there was at least 1 error, but the result is still OK … Dont’t know if this is a big problem or not. First let’s try to reboot the VM and access Jeedom’s UI. First i’ll shut off the old Jeedom on my Debian 8.5.

After a sanity reboot, I reconnect on Jeedom’s UI and It is running, I can see my dashboard.

The problems I quickly found are:

1. Check Jeedom > Health menu, it will probably complain about the external network configuration. This is normal, my backup restored the network config made to be reached from the outside, but it’s still pointing on my old VM. I’ll need to assign the IP address of my old VM to the new one on my DHCP server (in my case my internet box) OR modify the external & Internal IP in Jeedom.
2. In Jeedom > Update Center, it complains that I have too much Jeedom’s declared on the Jeedom market. This is normal i’m using the “free” version, and the same login to access the market. I just connect on the Market website, check my declared boxes, and remove the old one.
3. Some modules were disabled. Not important ones (forecast.io plugin to display weather, etc.) but they are disabled and can’t manage to get them back.
4. You need to reinstall the dependancies for all your modules that need some in their configuration, this is VERY important: Bluetooth Advertisement, ZWave, KRoomba, etc. By using this Debnian 10.3 installation, I had absolutely NO problem with dependencies building, the modules were working well after.

Other than that, all my scenarios, seetings, were perfectly restored … I spent a few days on making this tutorial and finding the good VM parameters & restauration process, but it was definitely needed for the health of my home automation.

Hope it can help some people.