An average guy's IT (mis)adventures

I am most productive at night, when there is finally silence - apart from background music. The last couple hours of this Wednesday brought some nice additions to the @wpn support MUC:

• chat is not XMPP only anymore, but it got bridged to Matrix, via "matterbridge".
• a bot is now sending "BOFH excuses" to the chat, once a day.

XMPP MUC can be reached from: https://invite.woodpeckersnest.space/#status@chat.woodpeckersnest.space?join

Matrix room can be reached from: https://matrix.to/#/#wpn:matrix.org

Bridged chat means that everything being written in any of those rooms, gets relayed to the other by bots: there are in fact 2 bots connected to the 2 rooms, one is an XMPP bot and the other is a Matrix one, operated by "matterbridge", as I said earlier.

About "BOFH excuses", they are some fortune cookies from the relative Linux package, which are all about "Bastard Operators From Hell" excuses.. They are some short funny sentences about IT which an imaginary Operator (sysop, sysadmin, you name it) could say as excuse to justify issues.

Example?

BOFH excuse #266:
All of the packets are empty.

I have the same script running on gemini, at gemini://woodpeckersnest.space/cgi-bin/bofh

Basically it's some python spaghetti which runs the fortune command, does some formatting to the resulting text and sends it over gemini, or in the case of @wpn chat, over webhook to one other bot, which then posts its content to XMPP. It's more complicated to explain than do it 😀

P.S. @wpn support MUC is not publicly listed, so even if I'm posting links which may be used to join it, it should be treated as limited to @wpn users only.

gemlog

store /etc in git, mercurial, brz or darcs

The etckeeper program is a tool to let /etc be stored in a git, mercurial, brz or darcs repository. It hooks into APT to automatically commit changes made to /etc during package upgrades. It tracks file metadata that version control systems do not normally support, but that is important for /etc, such as the permissions of /etc/shadow. It's quite modular and configurable, while also being simple to use if you understand the basics of working with version control.

https://packages.debian.org/stable/etckeeper

root@pandora:/etc# git log --oneline mumble-server.ini
c1e2238 daily autocommit
2e66cf6 daily autocommit
76fd073 daily autocommit
baaff1c saving uncommitted changes in /etc prior to apt run
9980c10 daily autocommit
830fa84 daily autocommit
0cc8fd9 saving uncommitted changes in /etc prior to apt run
4e7b545 saving uncommitted changes in /etc prior to apt run
20d692a daily autocommit
c05a405 saving uncommitted changes in /etc prior to apt run
3a00635 daily autocommit
62e0e1c Initial commit

root@pandora:/etc# git diff c1e2238 2e66cf6 mumble-server.ini
diff --git a/mumble-server.ini b/mumble-server.ini
index 3332154..d91f520 100644
--- a/mumble-server.ini
+++ b/mumble-server.ini
@@ -236,8 +236,8 @@ logdays=5
 ; If you only wish to give your "Root" channel a custom name, then only
 ; uncomment the 'registerName' parameter.
 ;
-registerName=SpaceNest
-registerPassword=sekret
+registerName=SpaceNest[EN/IT]
+registerPassword=newsekret
 registerUrl=https://woodpeckersnest.space/
 registerHostname=voice.woodpeckersnest.space
 registerLocation=IT

I'll update this post whenever I see fits, A.K.A. when I have other bits of information which don't require a whole post.

Operations on files and directories

Move files and directories to the current/parent directory in Linux

Current dir

find . -type f -exec mv {} . \;

Parent dir

find . -type f -exec mv {} .. \;

Recursive chmod on files and dirs

To change all the directories to 755 (-rwxr-xr-x):

find /var/www/blog -type d -exec chmod 755 {} \;

To change all the files to 644 (-rw-r--r--):

find /var/www/blog -type f -exec chmod 644 {} \;

Certbot common commands

Register single domain:

certbot certonly --standalone -d domain.tld --dry-run

Renew single domain:

certbot renew --cert-name domain.tld --dry-run

• Remove “–dry-run” when ok.

Revoke certificate:

certbot revoke --cert-path /etc/letsencrypt/archive/${YOUR_DOMAIN}/cert1.pem

Check certificate's expiry date:

cat /etc/letsencrypt/live/domain.tld/cert.pem | openssl x509 -noout -enddate

GIT

To reset your git repository to given commit id, do:

git reset --hard <commit-id>
git push origin master --force

Ignoring files that are already tracked

git update-index --assume-unchanged <your file here>

Delta Chat Mail sieve

require ["fileinto"];
# rule:[DeltaChat]
if header :contains "Chat-Version" "1.0"
{
	fileinto "DeltaChat";
	stop;
}

Postfix mail queue

To view postfix mail queue in case of problems and remove a particular message from it, do as follows:

# mailq

-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
6DCF32201B*    4824 Thu Oct 19 22:54:44  roughnecks@woodpeckersnest.eu
                                         debian@spacenet.it

-- 4 Kbytes in 1 Request.

# postsuper -d 6DCF32201B

postsuper: 6DCF32201B: removed
postsuper: Deleted: 1 message

To delete all emails in the queue, use this command:

# postsuper -d ALL

Check Preferred Outgoing IP (when multiple are set on <interface>)

curl ifconfig.me

The correct command to add a new Linux user (in this case without a shell)

adduser --shell /usr/sbin/nologin <username>

I always forget which command is the complete one, useradd or adduser

Several months ago I was running a "matrix-docker-ansible" playbook on OVH, but limited resources led me to take it down.

More than one time I thought about reliving that server (which I have backups of), but reading on the changes that the playbook got so far and the difficulties in running a Matrix environment without exposing web ports (80 and 443), I thought to better give up.

Yesterday I looked into Dendrite (a Matrix server written in GO), which looks a lot simpler than the whole lot the docker-ansible playbook offers, so I went ahead and installed it the manual way with nginx as reverse proxy. Easy peasy - documentation is quite good.

Today I also installed "Cinny" as a web client and I'm done for the moment... Federation works, registrations are closed (can invite friends though) and apart from RAM peaks and quite long waiting times when joining rooms (up to 2K members), the server is doing quite well: started with 700MB of RAM (at joining time) and went down to less than 300 in normal usage. CPU gets high spikes when sending messages but other than that is just a few points %. Just don't even think to join the official #matrix room or shit will happen 😀

My new Matrix username is: @roughnecks:woodpeckersnest.space, feel free to add me.

I recently moved from OVH to Contabo for my VPS: since setting up everything from scratch looked like a hard challenge, I moved everything using rsync.

Firstly, some preparations:

• installed the same kernel I had on OVH
• rebooted Contabo
• installed rsync on Contabo

Then I switched to the OVH shell:

• created a exclude.txt file with all the directory and file I didn't want to move:

/boot/
/etc/modules
/etc/fstab
/etc/mtab
/etc/netconfig
/etc/networks
/etc/network/
/etc/ssh/
/etc/cloud/
/etc/cloud-release
/etc/grub.d/
/etc/host.conf
/etc/hostname
/etc/hosts
/etc/hosts.allow
/etc/hosts.deny
/etc/init.d/cloud-config
/etc/init.d/cloud-final
/etc/init.d/cloud-init
/etc/init.d/cloud-init-local
/etc/initramfs-tools/
/etc/default/grub
/etc/default/grub.d/
/etc/kernel/
/etc/kernel-img.conf
/lib/modules/
/lost+found/
/sys/
/proc/
/dev/
/tmp/
/var/cache/
/var/log/journal/
/mnt/
/root/.ssh/

• moved it to /root/exclude.txt

• stopped all the running services

  systemctl stop <service>

• now we can begin rsync-ing: you have to have access to the root user on destination VPS

  rsync -avzP --exclude-from=/root/exclude.txt / root@CONTABO_IP:/

• after that, I edited /etc/fstab to add support for quota, like I had on OVH

• then I searched and grepped /etc for my OVH IP address, because I knew it was set somewhere on some config file

  grep -r “OVH IP” /etc/*

• and substituted it with the new Contabo IP, where necessary.

Finally, I could reboot Contabo:

reboot

Once up & running again, I changed all DNS entries from OVH to Contabo IP.. And waited 😀

• Last but not least, I edited etc/hosts manually to reflect the new Domain Name address and also set up the hostname

  hostnamectl set-hostname <new-hostname>

Done!

EDIT! A detail I missed to mention is that your destination host's root password will be changed, after rsync-ing, to the one of the source host!!

Thanks to Zash and Jcbrand in the "Converse" MUC and a bit of hacking I was able to set up a few community plugins for my conversejs install as a prosody module.

Here's the configuration in /etc/prosody/prosody.cfg.lua:

conversejs_resources = "/usr/local/lib/prosody/modules/mod_conversejs/dist"
conversejs_tags = {
        -- Load favicon
        [[<link rel="shortcut icon" href="https://woodpeckersnest.space/images/converse-js.ico">]];
        -- Load libsignal-protocol.js for OMEMO support (GPLv3; be aware of licence implications)
        [[<script src="https://cdn.conversejs.org/3rdparty/libsignal-protocol.min.js"></script>]];
        -- Load community plugins
        [[<link type="text/css" rel="stylesheet" media="screen" href="conversejs/dist/plugins/search/search.css" />]];
        [[<script src="conversejs/dist/plugins/actions/actions.js"></script>]];
        [[<script src="conversejs/dist/plugins/search/search.js"></script>]];
        [[<script src="conversejs/dist/plugins/search/jspdf.debug.js"></script>]];
        [[<script src="conversejs/dist/plugins/search/jspdf.plugin.autotable.js"></script>]];
        [[<script src="conversejs/dist/plugins/toolbar-utilities/toolbar-utilities.js"></script>]];
        [[<script src="conversejs/dist/plugins/screencast/screencast.js"></script>]];
}

conversejs_options = {
        locked_domain = "woodpeckersnest.space";
        auto_focus = true;
        view_mode = "fullscreen";
        allow_registration = false;
        auto_reconnect = true;
        reuse_scram_keys = true;
        muc_clear_messages_on_leave = true;
        clear_cache_on_logout = false;
        play_sounds = true;
        whitelisted_plugins = {"actions", "search", "toolbar-utilities", "screencast"};
}

You'll have to copy the plugins directories (actions, search etc..) in this path:

/usr/local/lib/prosody/modules/mod_conversejs/dist/plugins/

Then reload configuration and conversejs module or restart prosody.

Already found a bug in "toolbar-utilities" and haven't still had a chance to try the screencast plugin, but they look good for the most part.

Maybe I will add Jitsi Meet or Voice Chat at some point.. Not now though. ¹

EDIT: screencast is working alright, but not in the way you'd expect it. It's not a live streaming, instead it's a recording of your screen which gets uploaded once you stop the cast.. I wouldn't say it's perfect but not even bad.

¹ I've added them 😛

Remember: don't ever use special characters in your DB's users password!!

I have been troubleshooting a migration from mariadbd to pgsql for several hours, until I tried changing my password, which, weirdly enough, was working to connect roundcube but gave errors with pgloader. Once I successfully migrated using the same password with those special characters, roundcube wouldn't connect with pgsql, so I made another fighting round! Finally changed my pgsql password to some letters and numbers only and BAM!, it worked.

Then I stopped mariadbd and disabled the service; now I have more than 200MB RAM freed¹.

¹ See previous post.