remove_duplicate_imap_messages

Requirements

Installation

Linux

sudo python3 setup.py install

Windows (from PowerShell, untested)

& $(where.exe python).split()[0] setup.py install

Usage

Usage: remove_duplicate_imap_messages.py [OPTIONS]

Options:
-d, --debug-level [CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
                                Set the debug level for the standard output.
-l, --log-file TEXT             File to store all debug messages.
-n, --dummy                     Don't do anything, just show what would be
                                done.
-s, --imap-server TEXT          IMAP server
-p, --imap-port INTEGER RANGE   IMAP server port  [1<=x<=65535]
-u, --imap-user TEXT            User name to use for the connection to the
                                IMAP server  [required]
-P, --imap-password TEXT        Password to connect to the IMAP server.
                                Warning! Use a configuration file to avoid
                                revelaing your passwords.  [required]
-S, --ssl BOOLEAN               Whether to use a secure connection or not.
-m, --mailbox TEXT              IMAP mailboxes to check. Will compare
                                messages between all mailboxes. Default:
                                INBOX
-m, --method [full|headers]     Method to decide messages are duplicated.
                                The 'full' method will check that the whole
                                message and headers are the same. The
                                'headers' method will check that selected
                                headers (with --header) are the same.
-f, --header TEXT               Fields to compare when method is 'headers'
--config FILE                   Read configuration from FILE.
--help                          Show this message and exit.

The 'full' method.

This method will take all information of the message (headers, dates, recipients, sender, subject, body, ...) and create a code (SHA256 hash), if the code is the same for another message then they are consider duplicates.

The 'header' method

This method will take only the select headers (at least one) and create a code (SHA256 hash), if the code is the same for another message then they are consider duplicates. Some common headers are:

• Subject
• Body
• To
• From
• Reply-To And there is a RFC (Request For Comments with more) https://www.rfc-editor.org/rfc/rfc2076 but ultimately you can add any custom header to a message with the right client or server tools.

Security

• Try first with the --dummy run to see how many messages will be deleted, because there won't be any prompts and messages get deleted (almost) inmediately.
• Don't pass the password in the command line, use the --config file to store parameters and protect it with the right permissions.
• Use SSL or your messages and credentials will be visible in your network (and any networks you need to go throw to get to your mail server).