A small side-effect of my community and blogging work is, that I occassionaly (a few times per month) have people reach out to me asking for advice on a variety of topics, or for help if they are in a pickle. The lastest case was a guy from Czechia (Let’s call him Mr. D), asking for help with his pwnd web/mail server. It had a multiple glaring issues and even after my intervention is probably a minefield of vulnerabilities and backdoor. Since I am in the last year (hopefully) of my PhD studies and am currently swamped by my regular job as well, I didn’t have time to spend properly investigating and fixing all the issues in a manner that I would feel comfortable getting paid for, I’ve decided to perform the bare minimum to get the site up and running “pro bono”.

Information gathering

Mr. D first reached out to me a few months ago with an issue that surfaced after upgrading from Ubuntu 18.04 to 20.04. I however wasn’t available to help at that time and not really confident dealing with LAMP (instead of my preferred LEMP) server on such short notice, so I’ve recommended that he tried looking for help from somebody else in our local Ubuntu community Telegram group. Somebody probably helped him as he next messaged me in late August for an advice and lastly in a couple of weeks ago after his long suffering server has finally keeled over and started serving the dark lords of the Internet script kiddies hunting for low hanging fruit. His mail server was up and running, but wasn’t able to deliver any email. He’s sent me a screenshot of his mail.err log file.

The first kind of error that hits your eyes is the repeated line:

mail dovecot: lmtp .... Error: SSL context initialization failed, disabling SSL: Can't load DH parameters: error 1408518:SSL routines:ssl3_ctx_ctrl:dh key too small

Even without engaging my google-fu (or rathe duckduckgo-fu, since I prefer the fruity overlord that at least pretends to care about my privacy), we can dissect the error message and gain valuable intel. First of all, we know the issue is with dovecot, our usually reliable IMAP server, and specifically in lmtp. LMTP is a transfer protocol often used to bridge postfix and dovecot, otherwise known as the peanut butter and jelly of email services. The error message itself informs us that we won’t be able to use SSL, because the DH key is too small. We can still access IMAP without encryption, but that really isn’t optimal. It’s better to fix it, even though it’s not the cause of the unusable mail server. Copy-pasting the error message into our favourite avian-themed search engine leads us to find out, that this error is pretty common occurence when upgrading Ubuntu to the latest LTS (as of writing, Ubuntu 20.04), especially when also running some “user-friendly” server managment tools such as ISPConfig. The solution is to generate a new 4096 bit SSL.

After some time (an hour or two on a powerful machine, a night on the VPS), I’ve copied the resulting file to /etc/dovecot/private, edited the config file at /etc/dovecot/conf.d/10-ssl.conf and added this line:

ssl_dh = </etc/dovecot/private/dh.pem

After that I’ve restarted the dovecot service and waited a few minutes to see if I get any more SSL errors. Since the error log has been clean since than, I can assume it worked. The rest of the issues I decided to resolve later, since I needed to get the mail running first. Interestingly, the rest of the issues eventually disappeared on their own, probably as a side effect of my other actions. If they resurface, I’ll deal with them and update the post accordingly.

After dealing with this lower hanging fruit, I went and checked on the running services. Nothing really seemed that strange, apart from the holy trinity of PHP versions, specifically 5.6, 7.2 and 7.4. From chat with D, it seems the 5.6 is necessary for some legacy intranet website, 7.4 runs both of his WordPress instances and ISPConfig and 7.2 runs… Cthulhu knows what, I don’t.

Restoring mail capability…

There have been a multiple indicators that D’s server has been pwned. His mail queue was constantly being filled so that proper email couldn’t get through and mail logs showed spam being sent to a multitude of emails. Whenever I dropped the entire queue with

sudo postsuper -d ALL

it was filled up again immediately. I’ve shut down all three versions of PHP and confirming my suspicion, the after once again deleting the whole queue, it remained empty allowing the test emails to pass freely. Unfortunately, I wasn’t able to locate the spam script, since it didn’t run again after launching all three PHP versions and the sheer number of php files modified in September alone is in thousands.

… just to have it fail again along with database, web and everything.

A few days passes and D has reached out to me once again, this time with more panic in his words. Unfortunately what I feared to be true ( that is, the attacker had unrestricted root access to the OS) proved to be very likely. His mariadb woudn’t run, because of errors in InnoDB and his database root user was deleted. In order to fix what was left of his database, I was forced to delete ib_logfiles risking corrupting some data and try to save what was possible to fix. Then, I stopped the mysql service and logged in while skipping grant tables:

sudo rm /var/lib/mysql/ib_logfile*
sudo mysqlcheck --repair --all-databases
sudo systemctl stop mysql
sudo mysqld_safe --skip-grant-tables &

Then I was able to finally login to mysql and create a new root user. Annoyingly, when running mysql in this regime, you are unable to use some of the more userfriendly features of the database (no CREATE user command, no GRANT privileges, etc.). Instead, you have to do what you’re almost never supposed to do and edit the inner mysql tables manually.

INSERT INTO mysql.user  SET user = 'root',  host = 'localhost',      password = Password('password'),      Select_priv = 'y',     Insert_priv = 'y',     Update_priv = 'y',     Delete_priv = 'y',     Create_priv = 'y',     Drop_priv = 'y',     Reload_priv = 'y',     Shutdown_priv = 'y',     Process_priv = 'y',     File_priv = 'y',     Grant_priv = 'y',     References_priv = 'y',     Index_priv = 'y',     Alter_priv = 'y',     Show_db_priv = 'y',     Super_priv = 'y',     Create_tmp_table_priv = 'y',     Lock_tables_priv = 'y',     Execute_priv = 'y',     Repl_slave_priv = 'y',     Repl_client_priv = 'y',     Create_view_priv = 'y',     Show_view_priv = 'y',     Create_routine_priv = 'y',     Alter_routine_priv = 'y',     Create_user_priv = 'y',     Event_priv = 'y',     Trigger_priv = 'y',   Create_tablespace_priv = 'y', ssl_cipher = '', x509_issuer = '', x509_subject = '', authentication_string = '';

The monstrosity above has done the trick and I was finally able to kill this unholy mysql session, restore the regular one and login as root. Then I changed the password, and altered the configs of WordPress and Phpmyadmin accordingly. Lastly, I’ve sent a message to D, explaining that this is just a temporary solution and that he should migrate to a new clean machine and my further assistance would come with my regular friendly fee.