We had an issue where one of our email accounts got hacked (this can happen from time to time). The password was relatively secure but it got hacked anyway. As it was an unmonitored email we were unaware that it was being used to send spam far and wide on a routine basis.
Eventually our sender reputation with the larger providers ended up in the gutter and as a result we got blocked by google and every message to hotmail ended up in the user’s spam folder.
Once we identified the offending account (which was done through the logs)
tail -f /var/log/mail.log
we simply deleted the email account, but that was not the end of it.
Postfix maintains a queue of deferred emails that it will keep trying to send over and over again in perpetuity until it is flushed. Even if you have resolved the issue by deleting the account any of these deferred emails that are stuck in the loop will remain there and keep trying to send, giving the appearance that the issue has not been resolved (this took a while to find)
Postfix has a couple of utilities to resolve the problem called postqueue and postsuper. First use postqueue to list the emails that are stuck in the queue.
sudo postqueue -p
and then postsuper to delete them. Our server only looks after a limited number of accounts so we just purged the queue completely however in larger organisations it may be necessary to be a little more careful. I would recommend reading the man pages for both postqueue and postsuper in such circumstances. If you just want to go ahead and purge the queue then it is simply a matter of:
sudo postsuper -d ALL
That should clear the queue and leave you with a clean server however you will still need to regain your sender reputation, and this takes time.
Monitoring
A useful utility to monitor your server statistics is pflogsumm. It does this by parsing the log file and is available in the Debian repository:
apt install pflogsumm
Using it is not entirely intuitive as you need to tell it where the log file is so the command involves some piping:
sudo cat /var/log/mail.log | sudo pflogsumm | less
will get you the results you want.
NOTE: If you want to reset the statistics then simply clear the log file (you will need to do this as root or a super user – sudo will not work),
echo > /var/log/mail.log
When our server was hacked it delivered 22,000 emails before the log had recycled when ordinarily our organisation only sends out around 50 a day at the most so it is a useful monitor.
Once Bitten
So as with most things monitoring is key, but often forgot so we decided it would be wise to create a cron job to mail us our email statistics for the previous day.
Here is the command to send the email using mutt
cat /var/log/mail.log | pflogsumm -d yesterday | mutt -s "Mail Statistics" myemail@mydomain.com
So now we can email the statistics all we need to do is set up a job to do this automatically. This used to be done as a cron job but as systemd has pretty much taken over everything we are going to use that instead by first creating a service and then a timer.
Navigate to /etc/systed/system and create a file called mail-monitor.service. Next edit the file and add the following (Requires Mutt to be configured already – Instructions on how to do that are on the bacula configuration page)
# This service is used to email mail statistics [Unit] Description=Send an email statistics report to the assigned recipient Wants=mail-monitor.timer [Service] ExecStart=/bin/sh -c 'cat /var/log/mail.log | pflogsumm -d yesterday | mutt -s "Postfix mail statistics" me@mydomain.com' [Install] WantedBy=multi-user.target
Save the file and you should be able to test the service by using.
systemctl start mail-monitor
If all is well you should have received an email of the report, if something is wrong then have a look at the systemd journal to see what the problem is:
journalctl -f
Now all you need to do is create the timer. In the same directory create a file called mail-monitor-timer. Now edit that file as follows
# This timer unit is used for periodic sending of postfix email statistics [Unit] Description=Periodically send postfix statistics to recipient Requires=mail-monitor.service [Timer] Unit=mail-monitor.service OnCalendar=*-*-* 23:00:00 [Install] WantedBy=timers.target
This will run the job daily at 11:00pm. For other settings have a look at https://opensource.com/article/20/7/systemd-timers. Now you can enable the timer and start it.
systemctl enable mail-monitor.timer systemctl start mail-monitor.timer
That should be it; you should get an email on start and then periodically according to your timer (This is very similar to creating a systemd daemon).
You can make sure that your timer is active by
systemctl status mail-monotor.timer
And it should tell you when it will next run:
● mail-monitor.timer - Periodically send postfix statistics to recipient
Loaded: loaded (/etc/systemd/system/mail-monitor.timer; enabled; preset: enabled)
Active: active (waiting) since Sat 2023-10-28 19:29:40 BST; 17min ago
Trigger: Sat 2023-10-28 23:00:00 BST; 3h 12min left
Triggers: ● mail-monitor.service
To help prevent being brute force hacked in the first place you should set some limits in /etc/postfix/main.cf.
# RATE THROTTLING smtpd_client_connection_rate_limit = 20 smtpd_error_sleep_time = 17s smtpd_soft_error_limit = 3 smtpd_hard_error_limit = 5
This should at least slow them down, albeit it could slow everything down if the settings are excessive.
Another; probably better way of preventing hackers is to utilise fail2ban. Navigate to /etc/fail2ban/jail.d and create a file called postfix.conf. Add the following to that file and restart fail2ban and it should now monitor the postfix logs once restarted.
[postfix] enabled = true
Don’t forget to make sure that fail2ban is enabled and started. You can check that all is well by looking at the fail2ban log /var/log/fail2ban.log which should show something like
fail2ban.jail [39743]: INFO Creating new jail 'postfix'
fail2ban.jail [39743]: INFO Jail 'postfix' uses pyinotify {}
fail2ban.jail [39743]: INFO Initiated 'pyinotify' backend
fail2ban.filter [39743]: INFO maxRetry: 5
fail2ban.filter [39743]: INFO findtime: 600
fail2ban.actions [39743]: INFO banTime: 600
fail2ban.filter [39743]: INFO encoding: UTF-8
fail2ban.filter [39743]: INFO Added logfile: '/var/log/mail.log' (pos = 0, hash = 476995ef3f25e2b7bc61d4695b1787f3dbbe72e4)
fail2ban.jail [39743]: INFO Jail 'postfix' started
There is a utility called fail2ban-client that you can use to check the status:
fail2ban-client status postfix
With any luck this should help prevent something like this happening again.
Recent Comments