{"id":484,"date":"2023-10-28T19:49:54","date_gmt":"2023-10-28T18:49:54","guid":{"rendered":"https:\/\/blog.inplico.uk\/?p=484"},"modified":"2023-10-29T10:38:54","modified_gmt":"2023-10-29T10:38:54","slug":"email-hacking","status":"publish","type":"post","link":"https:\/\/blog.inplico.uk\/?p=484","title":{"rendered":"Email Hacking"},"content":{"rendered":"<p>We had an issue where one of our email accounts got hacked (this can happen from time to time).\u00a0 The password was relatively secure but it got hacked anyway.\u00a0 As it was an unmonitored email we were unaware that it was being used to send spam far and wide on a routine basis.<\/p>\n<p>Eventually our sender reputation with the larger providers ended up in the gutter and as a result we got blocked by google and every message to hotmail ended up in the user&#8217;s spam folder.<\/p>\n<p>Once we identified the offending account (which was done through the logs)<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">tail -f \/var\/log\/mail.log<\/pre>\n<p>we simply deleted the email account, but that was not the end of it.<\/p>\n<p>Postfix maintains a queue of deferred emails that it will keep trying to send over and over again in perpetuity until it is flushed.\u00a0 Even if you have resolved the issue by deleting the account any of these deferred emails that are stuck in the loop will remain there and keep trying to send, giving the appearance that the issue has not been resolved (this took a while to find)<\/p>\n<p>Postfix has a couple of utilities to resolve the problem called <strong>postqueue<\/strong> and <strong>postsuper<\/strong>.\u00a0 First use postqueue to list the emails that are stuck in the queue.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">sudo postqueue -p<\/pre>\n<p>and then postsuper to delete them.\u00a0 Our server only looks after a limited number of accounts so we just purged the queue completely however in larger organisations it may be necessary to be\u00a0 a little more careful.\u00a0 I would recommend reading the man pages for both postqueue and postsuper in such circumstances.\u00a0 If you just want to go ahead and purge the queue then it is simply a matter of:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">sudo postsuper -d ALL<\/pre>\n<p>That should clear the queue and leave you with a clean server however you will still need to regain your sender reputation, and this takes time.<\/p>\n<p>&nbsp;<\/p>\n<h3>Monitoring<\/h3>\n<p>A useful utility to monitor your server statistics is pflogsumm.\u00a0 It does this by parsing the log file and is available in the Debian repository:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">apt install pflogsumm<\/pre>\n<p>Using it is not entirely intuitive as you need to tell it where the log file is so the command involves some piping:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">sudo cat \/var\/log\/mail.log | sudo pflogsumm | less<\/pre>\n<p>will get you the results you want.<\/p>\n<p><strong>NOTE:<\/strong>\u00a0 If you want to reset the statistics then simply clear the log file (you will need to do this as root or a super user &#8211;<strong> sudo will not work<\/strong>),<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">echo &gt; \/var\/log\/mail.log<\/pre>\n<p>When our server was hacked it delivered 22,000 emails before the log had recycled when ordinarily our organisation only sends out around 50 a day at the most so it is a useful monitor.<\/p>\n<p>&nbsp;<\/p>\n<h3>Once Bitten<\/h3>\n<p>So as with most things monitoring is key, but often forgot so we decided it would be wise to create a cron job to mail us our email statistics for the previous day.<\/p>\n<p>Here is the command to send the email using mutt<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">cat \/var\/log\/mail.log | pflogsumm -d yesterday | mutt -s \"Mail Statistics\" myemail@mydomain.com<\/pre>\n<p>So now we can email the statistics all we need to do is set up a job to do this automatically.\u00a0 This used to be done as a cron job but as systemd has pretty much taken over everything we are going to use that instead by first creating a service and then a timer.<\/p>\n<p>Navigate to <strong>\/etc\/systed\/system<\/strong> and create a file called <strong>mail-monitor.service<\/strong>.\u00a0 Next edit the file and add the following (Requires Mutt to be configured already &#8211; Instructions on how to do that are on the bacula configuration page)<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\"># This service is used to email mail statistics\r\n\r\n[Unit]\r\nDescription=Send an email statistics report to the assigned recipient\r\nWants=mail-monitor.timer\r\n\r\n[Service]\r\nExecStart=\/bin\/sh -c 'cat \/var\/log\/mail.log | pflogsumm -d yesterday | mutt -s \"Postfix mail statistics\" me@mydomain.com'\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n<\/pre>\n<p>Save the file and you should be able to test the service by using.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">systemctl start mail-monitor<\/pre>\n<p>If all is well you should have received an email of the report, if something is wrong then have a look at the systemd journal to see what the problem is:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">journalctl -f<\/pre>\n<p>Now all you need to do is create the timer.\u00a0 In the same directory create a file called <strong>mail-monitor-timer<\/strong>.\u00a0 Now edit that file as follows<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\"># This timer unit is used for periodic sending of postfix email statistics\r\n\r\n[Unit]\r\nDescription=Periodically send postfix statistics to recipient\r\nRequires=mail-monitor.service\r\n\r\n[Timer]\r\nUnit=mail-monitor.service\r\nOnCalendar=*-*-* 23:00:00 \r\n\r\n[Install]\r\nWantedBy=timers.target\r\n<\/pre>\n<p>This will run the job daily at 11:00pm.\u00a0 For other settings have a look at <a href=\"https:\/\/opensource.com\/article\/20\/7\/systemd-timers\">https:\/\/opensource.com\/article\/20\/7\/systemd-timers<\/a>. Now you can enable the timer and start it.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">systemctl enable mail-monitor.timer\r\nsystemctl start mail-monitor.timer<\/pre>\n<p>That should be it; you should get an email on start and then periodically according to your timer\u00a0 (This is very similar to creating a systemd daemon).<\/p>\n<p>You can make sure that your timer is active by<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">systemctl status mail-monotor.timer<\/pre>\n<p>And it should tell you when it will next run:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-highlight=\"4\">\u25cf mail-monitor.timer - Periodically send postfix statistics to recipient\r\n     Loaded: loaded (\/etc\/systemd\/system\/mail-monitor.timer; enabled; preset: enabled)\r\n     Active: active (waiting) since Sat 2023-10-28 19:29:40 BST; 17min ago\r\n    Trigger: Sat 2023-10-28 23:00:00 BST; 3h 12min left\r\n   Triggers: \u25cf mail-monitor.service\r\n<\/pre>\n<p><span style=\"color: #999999;\">To help prevent being brute force hacked in the first place you should set some limits in \/etc\/postfix\/main.cf.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\"># RATE THROTTLING\r\nsmtpd_client_connection_rate_limit = 20\r\nsmtpd_error_sleep_time             = 17s\r\nsmtpd_soft_error_limit             = 3\r\nsmtpd_hard_error_limit             = 5\r\n<\/pre>\n<p><span style=\"color: #999999;\">This should at least slow them down, albeit it could slow everything down if the settings are excessive.<\/span><\/p>\n<p>Another; probably better way of preventing hackers is to utilise <strong>fail2ban<\/strong>.\u00a0 Navigate to <strong>\/etc\/fail2ban\/jail.d<\/strong> and create a file called <strong>postfix.conf<\/strong>.\u00a0 Add the following to that file and restart <strong>fail2ban<\/strong> and it should now monitor the postfix logs once restarted.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">[postfix]\r\nenabled = true<\/pre>\n<p>Don&#8217;t forget to make sure that fail2ban is enabled and started.\u00a0 You can check that all is well by looking at the fail2ban log <strong>\/var\/log\/fail2ban.log<\/strong> which should show something like<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">fail2ban.jail           [39743]: INFO    Creating new jail 'postfix'\r\nfail2ban.jail           [39743]: INFO    Jail 'postfix' uses pyinotify {}\r\nfail2ban.jail           [39743]: INFO    Initiated 'pyinotify' backend\r\nfail2ban.filter         [39743]: INFO      maxRetry: 5\r\nfail2ban.filter         [39743]: INFO      findtime: 600\r\nfail2ban.actions        [39743]: INFO      banTime: 600\r\nfail2ban.filter         [39743]: INFO      encoding: UTF-8\r\nfail2ban.filter         [39743]: INFO    Added logfile: '\/var\/log\/mail.log' (pos = 0, hash = 476995ef3f25e2b7bc61d4695b1787f3dbbe72e4)\r\nfail2ban.jail           [39743]: INFO    Jail 'postfix' started\r\n<\/pre>\n<p>There is a utility called fail2ban-client that you can use to check the status:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">fail2ban-client status postfix<\/pre>\n<p>With any luck this should help prevent something like this happening again.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We had an issue where one of our email accounts got hacked (this can happen from time to time).\u00a0 The password was relatively secure but it got hacked anyway.\u00a0 As it was an unmonitored email we were unaware that it was being used to send spam far and wide on a routine basis. Eventually our [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[],"class_list":["post-484","post","type-post","status-publish","format-standard","hentry","category-debian-server"],"_links":{"self":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts\/484","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=484"}],"version-history":[{"count":7,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts\/484\/revisions"}],"predecessor-version":[{"id":492,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=\/wp\/v2\/posts\/484\/revisions\/492"}],"wp:attachment":[{"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=484"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=484"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.inplico.uk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=484"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}