Posts tagged with “misadventures”

rsnapshot Debian how-to

**rsnapshot** is a filesystem backup utility for Unix-like systems that uses rsync and hard links to efficiently create incremental snapshots. It stores backups in dated directories, minimizing disk usage by linking unchanged files. Designed for automation, it supports local and remote backups via SSH.

apt install rsnapshot

/etc/rsnapshot.conf

config_version  1.2
# executables paths on your system
cmd_cp          /bin/cp
cmd_rm          /bin/rm
cmd_rsync       /usr/bin/rsync
cmd_ssh 		/usr/bin/ssh
cmd_du          /usr/bin/du
# type and number of snapshots to execute
retain  alpha   7       #7 snaps, one per day
retain  beta    2       #2 snaps, one per week
retain  gamma   1       #1 snap montly
verbose         2
loglevel        3
logfile 		/var/log/rsnapshot.log
lockfile        /home/roughnecks/rsnapshot.pid
# where to save your snapshots
snapshot_root   /mnt/backup/
# rsnapshot won't create the "root" folder of your backups. Create it yourself
no_create_root  1
# global short rsync args (feel free to adjust)
rsync_short_args        -avz
# uses hard links (keep this)
link_dest       1

# what, where and how to backup
# --rsync-path="sudo rsync" is only needed if you're backing up directories which need root access on the remote host
# --exclude-from=/etc/rsnapshot/exclude_path.list is a list of paths never to be backed up, relative to this backup only

backup  		debian@remote_host:/path/   local_path/   +rsync_long_args=--rsync-path="sudo rsync" --exclude-from=/etc/rsnapshot/exclude_path.list

# global exclusion list (directories never to be backed up)

exclude /proc/
exclude /sys/
exclude /dev/
exclude /run/
exclude /tmp/

First things first: I suggest to always use a non-root user (not even sudo, because that would be the same, right?) to execute rsnaspshot.

Make sure your user can write the lockfile and also the logfile (create paths where necessary)

Create the root dir (where to store your snapshots) with root, if necessary - (maybe inside /mnt/ ) - but give your user permissions to read and write afterwards.

Decide the number and frequency of your backups (daily, weekly and monthly is a sane default) - don't change the names “alpha, beta and gamma” in the config, because, as it is written in the comments: “Must be unique and in ascending order”

A simple backup line would be like:

backup  /var/www/       local_path/

  • “backup” is the logical name
  • “/var/www” is the source directory
  • “local_path/” is the destination directory inside your snapshot_root

If you're backing up a direcory on a remote host, use the same syntax as the one for rsync. I also suggest making use of the ~/.ssh/config file to configure and shorten your ssh destinations (name, ports, keys).

Last piece I added in the backup line above in configuration file is used when you need to backup a directory with root permission. You can give the “normal” user “sudo” rights to use rsync command without entering a password on the remote host and then you can keep running rsnapshot locally without root

backupuser ALL=(ALL) NOPASSWD: /usr/bin/rsync

One big advice, if I may. Be sure to make an exclusion for the following directories when backing up a /home/user dir

*/go
*/node_modules
*/.cache
*/.npm
*/.nvm
*/.rustup

especially go/pkg directory creates a big mess because all of the files of the modules have no “write” (+w) permission set - for whatever reason they did it (I really don't understand this choice).. So after the first successful snap, on the second one, when the software will move e.g. alpha.0 to alpha.1, it won't have the permission to do so. It will fail and you'll need to fix it.

Useful commands

rsnapshot help

rsnapshot <alpha | beta | gamma> # manually execute one of the rsnapshots

rsnapshot configtest # syntax check

rsnapshot du # show disk usage for your backups

scheduling

I'm using crontab to schedule rsnapshot, like this:

# Run rsnapshot alpha every day at 2:00 PM
0 14 * * * /home/roughnecks/scripts/rsnapshot-alpha.sh

# Run rsnapshot beta every monday at 3:00 PM
0 15 * * 1 /home/roughnecks/scripts/rsnapshot-beta.sh

# Run rsnapshot gamma the first of every month at 4:00 PM
0 16 1 * * /home/roughnecks/scripts/rsnapshot-gamma.sh

I actually run those mini shell scripts because I'm also sending notifications about successful/failed backups via ntfy

Here's one of them, the other ones are copy-pasted with different frequency:

#!/bin/bash

rsnapshot alpha

if [ $? -eq 0 ]; then
    curl -d "rsnapshot alpha successful! 😀" ntfy.myserver.tld:port/tag
else
    curl -d "rsnapshot alpha failed?! 🤔" ntfy.myserver.tld:port/tag
fi

That would be all. Hope it's been useful

gemlog


self hosting is a serious matter - but fun

I've had my hands full in the last few weeks.. let's see what's new for @wpn

  1. First of all, I removed some #DNS records (and related services), namely:

laltrowiki (our (me and old friends) wiki didn't really fit with @wpn - I'm keeping it for myself)

trilium (note-taking app which I kept on @wpn, but was only personal.. Now it's hosted at home)

grocy (pantry/shopping aid app, same as trilium above)

  1. Have upgraded plugins for roundcube #webmail and FreshRSS, #RSS reader. Main services are always up-to-date, while these were some additions due from time to time.

  2. Have done some tinkering with this blog's and xmpp-it homepage's #sqlite DBs: if you're interested, follow here

  3. Have installed and then removed 2 #gemini to #html #proxies, because I believe the one I've been (and still am) using so far, is the best one. I also tweaked its stylesheet a bit. The two proxies I tried are september and kineto, while the one currently in use is loxy

  4. I have first upgraded uptime-kuma to V2 Beta but then backed it up and moved to gatus, which lacks some features but it's also lightweight and straightforward. During the process we lost webhooks live notifications (about services status) for XMPP chatroom, so I have also installed ntfy server and related Android app, but, obviously, I'm currently the only one to see those. If any of @wpn users is interested, I can share info about the "topic", server's "address:port" and such. For the time being everything is in plain and without authentication.. Don't think that @wpn's services status notifications are so "sensible" info to require encryption, so I didn't even bother 😀

  5. It's "6" already.. wow!

transmission-daemon was replaced by good-old rtorrent and that also got a web-ui, which only myself can access so far.

All of the aforementioned changes and fixes were mainly meant to reduce the load on the machine, in order to provide a better experience for everyone. Things still look complicated though, because even if there was a significant decrease in RAM usage, CPU, on the other side, seems almost worsened - and I still can't explain that, apart from thinking contabo maybe doing oversubscription of resources on their host.

Last 2 things, then I'll shut-up! 😁

  • Some of the users' home directories (only the ones which belongs to me or are system-related users account) are now backed-up via rsnapshot to my #homelab. I intend to write an how-to on that topic, further ahead in time, because I'm still testing/figuring it out.

  • The onboarding tool has got a new checkbox for (legal) age verification.

That's it. Feel free to tale a "tour" if you're new or haven't had the chance yet: https://woodpeckersnest.space/

See you soon!


Pimped up!

It's officially pimped!!

Can't believe the time I spent fixing "homarr" progressive layout, almost breaking at each smallest change.

New @wpn's home is running "glance" and the whole site is just 1 .yml file, plus some custom CSS rules and assets (images). Took me ~2 evenings to set it up and it's looking quite good, in my opinion. It's light (actually it's dark-only 😀 ), fast, fully featured.

Unfortunately, you always have to touch it first hand to understand the wrong choices and remedy them; better late than never though!


Busy Days, Exciting Progress

Busy Days, Exciting Progress 🚀

It’s been a couple working months, busy and sometimes hard but in the best way possible!

For starters, today the "GD" website went officially online, after we completed its migration from ZOPE to Wordpress and I couldn't be happier 😉

Another thrilling update is with COMMutiny, our social network project. The converse.js anonymous chat, originally set up for @wpn’s support room, has now found its own space within COMMutiny. This is a big step forward, and we’re inviting everyone interested in #XMPP, gaming, and social networks to join the conversation, anonymously or not.

On top of that, we’ve launched a brand-new freelance website, powered by Hugo. It’s sleek, fast, and everything we hoped it would be — hooray for #Hugo!

If you’re passionate about decentralized communication, gaming communities, or just want to connect with like-minded folks, we’d love to see you in COMMutiny! See you soon 👋


Some news for XMPP: support MUC and anonymous login

Simple instructions on how to join @wpn's XMPP server  anonymously

More news on the previously announced bridge for support MUC: we got another bridge set up. This time around, IRC joins the ballet, so matterbridge is now bridging @wpn support MUC for 3 different protocols.. XMPP and MATRIX being the previous other two.

https://health.woodpeckersnest.space/

On this matter I was thinking about making a "tombstone" of the current XMPP MUC, which is called status and move to wpn - In fact that's the name of the MATRIX and IRC rooms. This is not really something I want to do soon, though, because it implies changing several things.

Last, but not least, I have also set up an anonymous VirtualHost in Prosody. You can now login to anon.woodpeckersnest.space with a disposable account (whose data gets deleted from the server as soon as the account is not connected anymore) and participate/discuss in @wpn MUCs - access to external servers is not permitted to anonymous users, for obvious security concerne.

You can follow the steps in this GIF to connect anonymously with Gajim; Dino should also support it but I don't know how it's done.. Finally if you want to join via Android, these are the instructions provided by Daniel Gultsch, from Conversations IM:

you can just add the account something@anon.woodpeckersnest.space with an empty password to #Conversations_im and it will login anonymously. Anonymous logins don't require registration. When I say 'something@anon...' you can use anything as that username. Doesn't matter (it's only used to get through the client side jid validation check)

TIP: If the server tells you can't join a MUC, make sure you have added a nickname in your profile's details (Android).


Downtimes

It's been a few days now that I'm experiencing downtimes at night, early mornings.

When I wake up, connect to the VPS and attach to tmux, I am welcomed by these messages in console:

        Message from syslogd@pandora at Nov 3 05:37:13 ...
        kernel:[1586232.350737] Dazed and confused, but trying to continue

        Message from syslogd@pandora at Nov 3 05:37:24 ...
        kernel:[1586235.049143] Uhhuh. NMI received for unknown reason
        30 on CPU 1.

        Message from syslogd@pandora at Nov 3 05:37:24 ...
        kernel:[1586235.049145] Dazed and confused, but trying to continue

        Message from syslogd@pandora at Nov 3 05:37:55 ...
        kernel:[1586273.642163] watchdog: BUG: soft lockup - CPU#2 stuck
        for 27s! [dockerd:526408]

        Message from syslogd@pandora at Nov 3 05:38:00 ...
        kernel:[1586278.545172] watchdog: BUG: soft lockup - CPU#1 stuck
        for 24s! [systemd-journal:257]

        Message from syslogd@pandora at Nov 3 05:38:02 ...
        kernel:[1586281.187611] watchdog: BUG: soft lockup - CPU#3 stuck
        for 35s! [lua5.4:1702]

There's no need to say that when this happens, the server is completely frozen and doesn't respond to anything.

I already contacted support, but they didn't investigate at all, I believe. They manually restarted my VPS once and did some pings and connection tests (VNC, SSH) afterwards.. "everything is working fine!"

This last Saturday I was up when it happened, so I did a mtr from my PC to the VPS's IP and logged it, then I sent another email with the output to support.. Still waiting for them to reply, I guess tomorrow (Monday).

Friends like lorenzo and shai are having difficulties too, with the same provider, so I'm not imagining things.

Well, that's all I got to say, will keep you posted if any news.


@wpn gemini capsule changes home

Hello,

just a brief update on gemini here at @wpn.

I switched TLD from ".eu" to ".space": seemed more appropriate for gemini.

gemini://woodpeckersnest.space/

gemlog


Summer Recap at WPN

I'm always a bit busy when it comes to pandora (the VPS running WPN: woodpeckersnest.space/eu). I like experimenting new things, fixing/improving existing things.. I cannot stay still 😀

After migrating the homepage to homarr - which took really no time for the initial setup, but a lot of work afterward to fix layouts for mobile devices and non-full-hd screens for desktop PCs - I started messing up with a brand new toy: gemini!!

Not even a week since I installed molly-brown, the actual gemini server, to today, I can count lots of improvements..

  • Installed terminal gemini browser client amfora for wpn's shell users and also gtl, a tinylog reader, always for the shell.
  • Configured a local tinylog which groups together all wpn's capsuleer tinylogs, so it's easy to follow all of the local server users in one single place; the log is generated by gtl itself, refreshed and published every 5 minutes: can't miss a thing!
  • Initially configured gemlog mentions starting from a script by @bacardi55, who is the author of many gemini-related things, like the before mentioned gtl software. When I realized it lacked multi-capsules support, I started modifying it and came up with some spaghetti code, which is working surprisingly well and it was deployed earlier today.

gemini@wpn

Onboarding on WPN didn't go as well as I thought, but at least the first user (hey, Mario, I'm looking at you! :) registered and, I believe, everything is working fine for them! On this topic, the onboarding page was migrated from PHP and Email to Python and XMPP, thanks to my friend Schimon! He also kept the UI pretty much intact, so I think most people who looked at it before and after, wouldn't even notice the changes under the hood.

https://hello.woodpeckersnest.space/

Something else I've been doing was setting up: https://invite.woodpeckersnest.space/

which is a landing page to allow people to join an XMPP MUC or add an XMPP contact from a web interface, which will also guide them in choosing a client for their platform. It's rather simple but very useful at the same time.

The chatmail server was upgraded (more or less) at the beginning of August and running smoothly so far; it got some cool new improvements like automatic account deletion after #amount of days from last login and lots of fixes. Total number of registered accounts, so far, is 117.

https://chatmail.woodpeckersnest.space/

Services which I dismissed include:

  • Jitsi Meet (wasn't really using it and it was wasting quite a lot of resources just to be running)
  • Isso comments service, which powered the old homepage contact section and also a shaarli instance, which is still running but it's more a private thing, rather than a public one.

One more proposition: from now on, I will be publishing these (B)log posts in both protocols, HTTP here as you're reading and gemini on roughnecks' gemlog. I will be probably publishing less often than usual though, at least in this format, and send more status updates through the tinylog on WPN, the microlog at Station and my fediverse account.

In the next days I will be monitoring how everything goes and relax a bit, if I manage.. Today I didn't feel so good after a few stressful days, too much computing and too less sleep hours - it's 01:40 AM right now, so yeah, tomorrow will be another of “those” days, I guess.

gemlog


re: CardDAV Plugin for Roundcube Installed

Continuing from the previous article..

Today while trying to install yet another plugin (Calendar this time), I had a lil incident and destroyed everything 😃

Some hours later I restored a backup and we're up again. BUT! In the process I discovered some SQL errors which I believe were there since a lot ago, always gone unseen.

To make a long story short, I had to disable the standard "Personal Address Book" for everyone, because it was impossible to save any contact in there anyway.. And we are now relying on CardDAV, which is way better.

At one point I had the Calendar plugin working too, alongside CardDAV, but I had (wrongfully) installed it as a local one, so no sync to the cloud with CalDAV; it was later that I tried the CalDAV way by changing the config and shit got me.

Now I asked the people of libera.chat about the plugin, to see if it really supports any CalDAV implementation or not - and then I'll try again :) Feel free to check it out and leave a comment if you know better than me..

https://git.kolab.org/diffusion/RPK/browse/master/plugins/calendar/

and here's the configuration: https://git.kolab.org/diffusion/RPK/browse/master/plugins/calendar/config.inc.php.dist$28

I believe I'm done for today tho.. Looked like a full day's job. Ooh, yes.. Was already forgetting. I also updated the services blob in my website with all the new stuff.


More on WebDAV - Connecting a remote WebDAV folder in Windows

After some failed attempt at this, I think I found the right way to "mount" a remote WebDAV folder under Windows' Explorer.

Initially my baby steps took me here: https://note.woodpeckersnest.space/share/0TJT81fgI8Jy

After following that tutorial I didn't succeed, so I investigated further. I can say that everythig looks correct until you get to point 9.

The address they tell you have to enter isn't correct in my experience and they aren't even using https for the URL. What worked for me was instead something like:

\\webdav.woodpeckersnest.space@SSL\folder

You have to input the network-path-stile address which is common in Windows, as in: double backslash, FQDN of your WebDAV server, "@SSL" and then the path (folder) where you have access to files in your WebDAV server, with a backslash preceding it.

That's it, a prompt will ask for username and password and then a new Network Path (WebFolder) will be connected in Explorer, just below your local drives.

You can then browse, copy, upload, delete (and so on) whatever content you like.

EDIT: Just found out I couldn't rename files/folders from Windows or Total Commander (Android)

Fixed by setting nginx virtualhost like this:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name  webdav.woodpeckersnest.space;


    # HTTPS configuration
    ssl_certificate /etc/letsencrypt/live/webdav.woodpeckersnest.space/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/webdav.woodpeckersnest.space/privkey.pem;

    access_log /var/log/nginx/webdav/access.log;
    error_log /var/log/nginx/webdav/error.log;

  location / {
    set $destination $http_destination;

    if ($destination ~* ^https(.+)$) {
         set $destination http$1;
    }

    proxy_set_header   Destination $destination;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   Host $host;
    proxy_pass         http://127.0.0.1:17062/;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection "upgrade";
  }

  client_max_body_size 0;

}

Now I'm quite happy 😀