An average guy's IT (mis)adventures

I've had my hands full in the last few weeks.. let's see what's new for @wpn

1. First of all, I removed some #DNS records (and related services), namely:

laltrowiki (our (me and old friends) wiki didn't really fit with @wpn - I'm keeping it for myself)

trilium (note-taking app which I kept on @wpn, but was only personal.. Now it's hosted at home)

grocy (pantry/shopping aid app, same as trilium above)

2. Have upgraded plugins for roundcube #webmail and FreshRSS, #RSS reader. Main services are always up-to-date, while these were some additions due from time to time.

3. Have done some tinkering with this blog's and xmpp-it homepage's #sqlite DBs: if you're interested, follow here

4. Have installed and then removed 2 #gemini to #html #proxies, because I believe the one I've been (and still am) using so far, is the best one. I also tweaked its stylesheet a bit. The two proxies I tried are september and kineto, while the one currently in use is loxy

5. I have first upgraded uptime-kuma to V2 Beta but then backed it up and moved to gatus, which lacks some features but it's also lightweight and straightforward. During the process we lost webhooks live notifications (about services status) for XMPP chatroom, so I have also installed ntfy server and related Android app, but, obviously, I'm currently the only one to see those. If any of @wpn users is interested, I can share info about the "topic", server's "address:port" and such. For the time being everything is in plain and without authentication.. Don't think that @wpn's services status notifications are so "sensible" info to require encryption, so I didn't even bother 😀

6. It's "6" already.. wow!

transmission-daemon was replaced by good-old rtorrent and that also got a web-ui, which only myself can access so far.

All of the aforementioned changes and fixes were mainly meant to reduce the load on the machine, in order to provide a better experience for everyone. Things still look complicated though, because even if there was a significant decrease in RAM usage, CPU, on the other side, seems almost worsened - and I still can't explain that, apart from thinking contabo maybe doing oversubscription of resources on their host.

Last 2 things, then I'll shut-up! 😁

• Some of the users' home directories (only the ones which belongs to me or are system-related users account) are now backed-up via rsnapshot to my #homelab. I intend to write an how-to on that topic, further ahead in time, because I'm still testing/figuring it out.

• The onboarding tool has got a new checkbox for (legal) age verification.

That's it. Feel free to tale a "tour" if you're new or haven't had the chance yet: https://woodpeckersnest.space/

See you soon!

It's been a few days now that I'm experiencing downtimes at night, early mornings.

When I wake up, connect to the VPS and attach to tmux, I am welcomed by these messages in console:

        Message from syslogd@pandora at Nov 3 05:37:13 ...
        kernel:[1586232.350737] Dazed and confused, but trying to continue

        Message from syslogd@pandora at Nov 3 05:37:24 ...
        kernel:[1586235.049143] Uhhuh. NMI received for unknown reason
        30 on CPU 1.

        Message from syslogd@pandora at Nov 3 05:37:24 ...
        kernel:[1586235.049145] Dazed and confused, but trying to continue

        Message from syslogd@pandora at Nov 3 05:37:55 ...
        kernel:[1586273.642163] watchdog: BUG: soft lockup - CPU#2 stuck
        for 27s! [dockerd:526408]

        Message from syslogd@pandora at Nov 3 05:38:00 ...
        kernel:[1586278.545172] watchdog: BUG: soft lockup - CPU#1 stuck
        for 24s! [systemd-journal:257]

        Message from syslogd@pandora at Nov 3 05:38:02 ...
        kernel:[1586281.187611] watchdog: BUG: soft lockup - CPU#3 stuck
        for 35s! [lua5.4:1702]

There's no need to say that when this happens, the server is completely frozen and doesn't respond to anything.

I already contacted support, but they didn't investigate at all, I believe. They manually restarted my VPS once and did some pings and connection tests (VNC, SSH) afterwards.. "everything is working fine!"

This last Saturday I was up when it happened, so I did a mtr from my PC to the VPS's IP and logged it, then I sent another email with the output to support.. Still waiting for them to reply, I guess tomorrow (Monday).

Friends like lorenzo and shai are having difficulties too, with the same provider, so I'm not imagining things.

Well, that's all I got to say, will keep you posted if any news.

Last evening, at around 22:00 CET, Contabo migrated my VPS from old Hub at Nuremberg to the newly built "Hub Europe".

They rebooted it and everything came back up as usual.. I wasn't at home when it happened, didn't even notice anything until I connected and found no tmux session running. Even my phone which was connected to wireguard, quietly continued working and sending notifications 😎

Meanwhile, my friends and I watched "The Creator" on Disney+

Can't really say I do recommend it, but for some casual evening/entertainment, it wasn't half bad.

gemlog

I'm always a bit busy when it comes to pandora (the VPS running WPN: woodpeckersnest.space/eu). I like experimenting new things, fixing/improving existing things.. I cannot stay still 😀

After migrating the homepage to homarr - which took really no time for the initial setup, but a lot of work afterward to fix layouts for mobile devices and non-full-hd screens for desktop PCs - I started messing up with a brand new toy: gemini!!

Not even a week since I installed molly-brown, the actual gemini server, to today, I can count lots of improvements..

• Installed terminal gemini browser client amfora for wpn's shell users and also gtl, a tinylog reader, always for the shell.
• Configured a local tinylog which groups together all wpn's capsuleer tinylogs, so it's easy to follow all of the local server users in one single place; the log is generated by gtl itself, refreshed and published every 5 minutes: can't miss a thing!
• Initially configured gemlog mentions starting from a script by @bacardi55, who is the author of many gemini-related things, like the before mentioned gtl software. When I realized it lacked multi-capsules support, I started modifying it and came up with some spaghetti code, which is working surprisingly well and it was deployed earlier today.

gemini@wpn

Onboarding on WPN didn't go as well as I thought, but at least the first user (hey, Mario, I'm looking at you! :) registered and, I believe, everything is working fine for them! On this topic, the onboarding page was migrated from PHP and Email to Python and XMPP, thanks to my friend Schimon! He also kept the UI pretty much intact, so I think most people who looked at it before and after, wouldn't even notice the changes under the hood.

https://hello.woodpeckersnest.space/

Something else I've been doing was setting up: https://invite.woodpeckersnest.space/

which is a landing page to allow people to join an XMPP MUC or add an XMPP contact from a web interface, which will also guide them in choosing a client for their platform. It's rather simple but very useful at the same time.

The chatmail server was upgraded (more or less) at the beginning of August and running smoothly so far; it got some cool new improvements like automatic account deletion after #amount of days from last login and lots of fixes. Total number of registered accounts, so far, is 117.

https://chatmail.woodpeckersnest.space/

Services which I dismissed include:

• Jitsi Meet (wasn't really using it and it was wasting quite a lot of resources just to be running)
• Isso comments service, which powered the old homepage contact section and also a shaarli instance, which is still running but it's more a private thing, rather than a public one.

One more proposition: from now on, I will be publishing these (B)log posts in both protocols, HTTP here as you're reading and gemini on roughnecks' gemlog. I will be probably publishing less often than usual though, at least in this format, and send more status updates through the tinylog on WPN, the microlog at Station and my fediverse account.

In the next days I will be monitoring how everything goes and relax a bit, if I manage.. Today I didn't feel so good after a few stressful days, too much computing and too less sleep hours - it's 01:40 AM right now, so yeah, tomorrow will be another of “those” days, I guess.

gemlog

It's been a week or so that I started using Wireguard on my desktop too, browsing the Internet and doing the usual stuff I do, but this time connecting both via IPv4 and IPv6 through my VPS.

Results? I've already been banned (or to better state it, my VPS's IPv4 has) from 3 popular hosts: reddit, imgur and alienwarearena. Reason? I don't really know, looks like everyone doesn't like VPNs.

For the time being I resorted to replace reddit.com with old.reddit.com (even in my SearxNG instance) to be able to browse that shit, which unfortunately is sometimes useful. "imgur" is even more trickier, since I was still able to upload images (and also display them) via their API on Glowing-Bear.. But if I try to curl imgur.com from my VPS shell I get this:

{"data":{"error":"Imgur is temporarily over capacity. Please try again later."},"success":false,"status":403}

"Over capacity", yeah, but it's a 403, you liar!

So, a few moments ago I set my Wireguard service in Windows to manual start, stopped it and now I'm back with Hurricane Electric IPv6 tunnel - I would like to avoid being banned from the rest of the world, if possible.

Thanks for all the fish.

I recently moved from OVH to Contabo for my VPS: since setting up everything from scratch looked like a hard challenge, I moved everything using rsync.

Firstly, some preparations:

• installed the same kernel I had on OVH
• rebooted Contabo
• installed rsync on Contabo

Then I switched to the OVH shell:

• created a exclude.txt file with all the directory and file I didn't want to move:

/boot/
/etc/modules
/etc/fstab
/etc/mtab
/etc/netconfig
/etc/networks
/etc/network/
/etc/ssh/
/etc/cloud/
/etc/cloud-release
/etc/grub.d/
/etc/host.conf
/etc/hostname
/etc/hosts
/etc/hosts.allow
/etc/hosts.deny
/etc/init.d/cloud-config
/etc/init.d/cloud-final
/etc/init.d/cloud-init
/etc/init.d/cloud-init-local
/etc/initramfs-tools/
/etc/default/grub
/etc/default/grub.d/
/etc/kernel/
/etc/kernel-img.conf
/lib/modules/
/lost+found/
/sys/
/proc/
/dev/
/tmp/
/var/cache/
/var/log/journal/
/mnt/
/root/.ssh/

• moved it to /root/exclude.txt

• stopped all the running services

  systemctl stop <service>

• now we can begin rsync-ing: you have to have access to the root user on destination VPS

  rsync -avzP --exclude-from=/root/exclude.txt / root@CONTABO_IP:/

• after that, I edited /etc/fstab to add support for quota, like I had on OVH

• then I searched and grepped /etc for my OVH IP address, because I knew it was set somewhere on some config file

  grep -r “OVH IP” /etc/*

• and substituted it with the new Contabo IP, where necessary.

Finally, I could reboot Contabo:

reboot

Once up & running again, I changed all DNS entries from OVH to Contabo IP.. And waited 😀

• Last but not least, I edited etc/hosts manually to reflect the new Domain Name address and also set up the hostname

  hostnamectl set-hostname <new-hostname>

Done!

EDIT! A detail I missed to mention is that your destination host's root password will be changed, after rsync-ing, to the one of the source host!!