iMessage.to

Separate the Email Message Sending Pools

james — Thu, 05 Aug 2010 07:18:59 +0000

Previously, iMessage.to kept all the about to be sent and sent messages in a common message pool. The only thing that differentiates between these 2 types of messages is a STATUS FLAG.

Previous Architecture : Message Pool comprises of both Sent and About to be Sent Messages

Well, this architecture worked fine when the amount of messages in the pool are small. But as the messages accumulate to the millions… the Email Sending Thread started to encounter performance issues. To sieve out the messages to be sent amidst a mountain load of messages in a pool consumed virtually 100% of a server’s CPU.

It was time for an architecture revision. For a start, we decided to split the messages into 2 separate pools. A MessageQueue for messages to be sent an a MessageArchive for messages that have been sent. By doing it this way, the MessageQueue will always be small in size and will allow the Email Sending Threads to retrieve the relevant messages to be dispatched quickly.

Sent messages will need to be kept in the MessageArchive to ensure that Bounce, Unsubscription, Update and Statistics Tracking operations can be performed on the dispatched email.

New Architecture : The messages are split into separate pools for better performance

Glad that a simple architectural change has greatly improved the performance of the product!

Sender Policy Framework – Version 1

james — Fri, 30 Jan 2009 07:15:15 +0000

Well, if you have read the previous post of Reverse DNS Issue, you will realize that computers which have been compromised by viruses and trojan horses have kind of been weeded out of the email spamming industry.

You would have think that we have finally won the war over the hackers and spammers. Truth be told… that was only the beginning…. As hackers found that compromised drones with dynamic IP addresses were no longer effective in their email spamming efforts, they turned their attention to …

servers with static IPs.

Problem and Solution

What happened was that there were servers idling in the internet space which are not fully hardened. These servers became prime time target. Once compromised, the servers could be harnessed to send their emails. To combat against such activities, ISPs started to implement the Sender Policy Framework.

Technical Tit-Bits

EHLO xtremax.com
MAIL FROM:

A reverse DNS on xtremax.com will pass. However… when the SMTP server seeing that the mail from actually comes from a different domain name, they will do a NSLOOKUP to check SPF record. If xtremax.com is authorised to send email on behalf of wierdcompany.com, the mail server will approve the email.

More often than not, the spf record will look like:

v=spf1 a mx –all

This means that emails can only be sent from wierdcompany.com to be valid. To allow xtremax.com to send the email, the SPF record will be like:

v=spf1 a mx xtremax.com -all

Reverse DNS Issue

james — Fri, 30 Jan 2009 06:46:41 +0000

The Origin

A couple of years ago, it did not really mattered how DNS was configured. Mails just went through. Those were the easy days… before SPAM came into the whole.

What happened was that some brilliant hacker discovered that sending unsolicited emails pushing drugs like Viagra, Enlargement drugs, etc… was very profitable business even if the response rate was dismal. They worked by the laws of large numbers…. Spammers understood that even with very low response rates, they just need to send larger and larger quantities of emails to generate the necessary responses and $$$.

Soon, hackers began running their home office like data centers. They have several computers with a high bandwidth. These computers served only one person – to dish out emails at an ever more ferocious rate….

However that was not fast enough yet. The numbers had to get larger… and that is when they decided to leverage on the resources of computers not belong to them. Trojan horses and viruses started to come into the picture… these viruses upon inflicting hapless computers will render these computers at the will of the spammer… to send emails! In fact, one prominent hacker at a time had over half a million drones at his control. By issuing a few simple commands, he could totally flood the network of his victims. This flooding of a victim’s network became known as a Denial-of-Service attack.

Most of the time though, the hacker will use these drones as a way to dispatch more emails to anonymous strangers. That prompted ISPs to look into using reverse DNS to authenticate the origin of emails….

The Solution

One of the major problems of using drones to send these emails are that drones are generally computers of home users with dynamic IPs. Doing a reverse DNS on dynamic IPs will reveal nothing, while doing a reverse DNS on a static IP will show the domain belong to a real server. By implementing a simple reverse DNS algorithm in their mail servers, ISPs could suddenly weed out the large volumes of spam from compromised computers….

Technical Tit-Bits

Sending an email occurs over a SMTP process – a protocol in which computers talk amongst themselves. They will first say hello by:

EHLO xtremax.com or
EHLO 192.168.1.4

If a domain name is given, the mail server will do a DNS query to get the IP. Once the IP is gotten, the mail server will issue a reverse DNS query. If the IP is not tied to any domain name… the email will be regarded as SPAM…

IP Address

james — Tue, 09 Sep 2008 05:29:33 +0000

When reporting SPAM problems for whitelistings, they will often require you to specify your IP address.
This is how you can check your imessage.to IP address

Click on configuration > settings

Under software info there is a line called IP Address. That is your IP address.

Understanding the delivery status

james — Fri, 05 Sep 2008 08:57:10 +0000

When your email is sent out there is a few kind of situation, imessage.to will indicate the status in the messaging queue.

How do you check the delivery status in imessage.to?
Click onto the “messaging queue” link of the newsletter that you have sent out, you will see a ‘Status’ column.

In this article, we are going to explain the different kinds of status and how we should go about handling them.

1) Sent - Message is sent across without any rejection message from the SMTP server
If customer comes back to say that they have not received any E-mails, but in the system, status is sent, 99% of the time, the email is sent across has been categorised by the internal SMTP server as a spam.
You should get your customer to contact their SMTP provider to white list the emails. The SMTP providers will normally ask you for the time the email was sent and the email address.

The time sent can be check in the messaging queue email (roll over the sent status)
The email address and ip address can be found under the configuration section.

2) Hard bounce - Email is invalid
Our system will allow a resend up to 3 times before we convert this status to Failed.

3) Soft bounce - Receipt’s mail box is full
Our system keep will keep resending.
The email will get through once the customer’s email inbox is cleared.

4) Temporary bounce - network traffic is too busy
Our system keep will keep resending.

5) Public block - the public server that blocks your IP
We will be notified of this situation and we will look into it promptly

6) Private block - the private server that blocks your IP
We will be notified of this situation and we will look into it promptly

7) Failed - this email is failed after being on hard bounce status for 3 times. The next time a newsletter is sent out, it will skip this email.

Any of the status, you can edit it manually in the customer information page.

Newly Implemented Function! “Paste from Words”

james — Mon, 01 Sep 2008 07:54:02 +0000

Paste from words - an add on function to the editor function.The editor functions allows the easy editing of words directly on our system into the newsletter template.
Knowing the number of customer who draft out their newsletter contents with the Microsoft Office Word, we have implemented this new function.
Simple and fast.
All you have to do is Copy (ctrl+c) your contents from the words document, open up the editor, click on ‘paste from words’ button and (ctrl+v) into the editor. It will automatically update the newsletter and the newsletter will be ready to go with your contents !

Sending an email directly from your Command Prompt

james — Mon, 11 Aug 2008 08:15:09 +0000

Did u know that you could send off emails directly from your Windows Command Prompt without any email clients like Microsoft Outlook or Outlook Express?

Well, the email clients do not have any additional functionality other than providing a nice user interface to a backend smtp server. Accessing your emails involve a 2 part process:

Identifying the SMTP server
Sending email through the SMTP server

Before doing anything else, start your command prompt by going to the Windows Start menu and clicking on run. After that type cmd. You will see the Windows Command Prompt.

Identifying the SMTP server

nslookup
set type=mx
xtremax.com

Sending email through the SMTP server

telnet mail.xtremax.com 25
mail from:
to:
DATA
blah blah blah
.

Note.. the fullstop at the end of the command is to signify the end of the message data.

SPF Record and SenderID

james — Fri, 11 Jan 2008 09:45:58 +0000

SPF stands for Sender Policy Framework, it is an extension to the SMTP protocol. Though it is still at Experimental stage, most of the major email providers have already adopted it (Hotmail, a MS product, is using SenderID). While SenderID can be considered an extension of the SPF framework, in fact, the keyword it uses in the record is “spf2.0”.

How it works

Sender email address present in the SMTP session

There are several sender email addresses present in a normal SMTP session.

The most important one is the “mail from” or Return Path or “mfrom” in the senderID framework. it is the email address provided by the sending server to the receiving server in the first SMTP command after the handshaking “HELO/EHLO” command. 2.
The “From” header field. 3.
The “Sender” header field (in the case of multiple email present in the “From” header field).
Other header fields has the sender email information including “Resent-From” “Resent-Sender”

The first one is normally the most important, it is where the email will be returned if it cannot be delivered. In fact, SPF framework only checks the “mail from”.

For sender ID frame work, it extract out another sender email other than “mail from” from item 2, 3, 4 listed above. Being called “PRA” (Purported Responsible Address), it is being selected in the following order:

From
Sender
Resent-From
Resent-Sender

Work flow of SPF and SenderID

Once a SMTP session starts, the receiving server will obtain the sender email address from the SMTP session.

Basically, a SPF / SenderID checking is checking if the current connecting sending server is a match with the server of the sender email address in the message. The workflow is illustrated in the figure below.

The major difference between the two is SPF only checks the email address provided by the “Mail From” SMTP command, but SenderID also checks the PRA obtained from the data of the email.

SPF / SenderID Record format

SPF /SenderID record are actually DNS records. It can be added in the DNS management page.

The spf record and senderID record share a similar format. It’s content will be illustrated by examples below.

1.SPF Record

example.org. IN TXT "v=spf1 a mx 123.45.67.89 -all"

For this line, it means for domain “example.org” there is a “TXT” record in the DNS record. The value of the “TXT” record is “v=spf1 a mx -all”.

While the line “v=spf1 a mx –all” means:

“v=spf1” Version 1 of SPF framework record.
“a mx”, if the current connecting server matching the “A” record of the server from the “mail from” command, it is considered passed. If it does not match, it would check against the “MX” record, if it matches it is a pass as well.
“123.45.67.89” is actually an IP address, if the current connected sending server matches this IP address, the test is passed.
“-all” Means if all the previous tests fails, for all the rest will results in a fail status in the SPF check.

2.SenderID record

example.org. IN TXT "spf2.0/mfrom,pra a mx 123.45.67.89 -all"

We can see it is very similar to SPF record except “v=spf1” is being replaced by “spf2.0/mfrom, pra”.

“spf2.0” is the version, and “mfrom, pra” means the receiving server should not only check the “Mail From” email address, but also the PRA email address obtained form the email data. While the checking procedure is the same as SPF record.

It is possible to spicify only checking the “Mail From” email, then the line would be:

example.org. IN TXT "spf2.0/mfrom a mx 123.45.67.89 -all"
And if the sending server want only PRA be checked the line should be similar to:
example.org. IN TXT "spf2.0/pra a mx 123.45.67.89 -all"

Spam Control Bill - Parliament

james — Wed, 21 Feb 2007 05:12:21 +0000

Recently the Singapore Government finally passed the Spam Control Bill - after a lot of consultation and reviews. The bill will have little effect on the SPAM situation in Singapore since most of the SPAMs originates from overseas, marketing products like viagra, nigerian scams, etc….

However it is a good first step. But how will this affect legitimate local SMEs who would like to leverage on email marketing to promote their products to the mass audience?

For those of you who likes reading legal bills, here you go, a link to the Spam Control Bill. Honestly, it was a tough read. After quite a bit of struggling with the legal jargon, I picked up the phone and give Boon Kiat of IDA a call at 62111907 to clarify the few remaining issues.

For the rest of you lazy buggers, read on:

Email harvesting softwares - where robots are sent to comb web pages to extract valid email addresses are considered harmful to society in nature as they consume bandwidth. Avoided as all costs!
SPAM is defined as anything that is unsolicited and commercial. So mails containing only information like exchange rates will not be considered as SPAM. However, if they have links to other products the company is marketing, that will be another issue.
Even if a consumer gives the retailer/merchant their email address, it is still considered SPAM as long as he/she did not explicity state that they will like to receive emails from your organisation.
Database companies that sell email lists? Definitely SPAM

So is SPAM illegal? Well, they are not as long as:

Your data was not taken from data harvesting softwares. This effectively means that even if you bought your data from a database firm, you can still send off your marketing messages… as long as comply with the below guidelines!
Append an ADV in the subject title of your email.
Provide an unsubscribe link that allows the user to unsubscribe.
Comply will all unsubscription requests within 10 days.

Below are some snapshots from excerpts of the Bill.

Hard Bounce & Soft Bounce Detection

james — Sun, 19 Nov 2006 22:06:44 +0000

Recently, a prospect asked if our system supports both hard bounce and soft bounce detection. Well, the answer is a definite yes! However, we would just like to take this opportunity to educate our users on the difference between these 2 types of bounces.

Hard Bounce occurs when a user sends an email to an invalid email address. Soft Bounce occurs when a user sends an email to a server which is currently not available for receiving mails. In the case of Hard Bounces, we need to immediately stop sending to these recipients. ISPs have been known to blacklist people who send their mailers to a large proportion of invalid email addresses (So for people who go about acquiring databases.. be forewarned!).

Hard Bounce occurs for a variety of reasons:

data entry error, i.e. keying in wrong email address
expired email addresses, i.e. customers giving up their singnet, singtel account
expired domain names

Soft Bounce occurs when:

a SMTP server requests for a resend at a later time. this can be due to either greylisting or just the server’s inate way to keep up with mail overload.

EMAIL SENDING CYCLE

The client SMTP Server contacts the Domain Name Server to resolve the email address’ domain name. If the email address is abc@yahoo.com, the SMTP server will basically try to resolve yahoo.com and find out the IP address of the server.
If the domain name is not registered, the email address is immediately treated as a hard bounced. In the case of a mistype email, i.e. abc@yahood.com, the domain name yahood.com not be registered.
The client SMTP Server then tries to contact the recipient’s server to deliver the message.
Here, the recipient’s server might choose to report that the email address is invalid immediately.
Alternatively, the recipien’s server might choose to send an email a few days later telling the sender that it is invalid.

Most systems only detect up to direct hard bounce, i.e. Step 4. They do not detect Step 5 as that requires direct access to your sender’s POP mailbox to parse through the bounced messages.

We understand the importance of keeping the privacy of your POP accounts. As such, we did a nifty implementation on our side to

direct all your bounced messages to a virtual mailbox on our side while keeping your customers’ replies directed back to you.

In this way, our client can track all bounces without getting their mailboxes flooded with an onslaught of bounced mails!