Integrate Amazon SQS and SendGrid with SpringBoot Application

Abstract:

In this blog, I will introduce how to integrate Amazon SQS and SendGrid with SpringBoot Application. SendGrid is a popular email service platform, we will use one of its java libraries to integrate SendGrid service and use its dynamic email template to send system transactional email. As I choose Amazon SQS as the message queue in my app, I will also explain how to bind SendGrid with Amazon SQS through the spring-boot application.

Problem & Solution:

Problem:

In a web application, we always need to send notifications to users through email. For example, when a new user finishes the registration, we should send a welcome email; when a user sends a friend request to another, we should send an email to the second user to notify there is a pending invitation.

At the very beginning, emails with plain text might be OK. However, as the service grows more sophisticated and user number grows larger, we must make our notifications more readable and elegant. What’s more, different kinds of notifications should be applied to different email templates. In my scenario as mentioned above, we need a template for registration and a template for friend invitation. SendGrid is just a good choice because it offers easy-to-use APIs and supports user-defined dynamic email templates for different uses.

Scalable Async Solution:

Meanwhile, compared with other requests which require real-time responses to improve user experience, requests to send emails are not time-sensitive. Thus, processing those requests is highly suitable for a producer-consumer model based on a message queue and scalable worker threads. Also, such a design can fully utilize the advantages of microservice over monolithic service.

Implementation:

SendGird & AWS Dependency Installation:

First of all, we should check the details of the SendGrid and AWS Java sdk on Maven repository and add the dependency to our Maven pom.xml for later use.

<dependencies>
    ......
    <!-- SendGrid - Email with HTML Template -->
    <dependency>
        <groupId>com.sendgrid</groupId>
        <artifactId>sendgrid-java</artifactId>
        <version>4.4.1</version>
    </dependency>
    <!--aws-sdk-->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.11.360</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>amazon-sqs-java-messaging-lib</artifactId>
        <version>1.0.4</version>
        <type>jar</type>
    </dependency>
</dependencies>

SpringBoot Worker Implementation:

Java Message Service Configuration:

I will skip the Producer implementation in my Spring MVC module and only focus on the Consumer(worker) part.

First, we should build a JmsListenerContainer to manage the interaction between the worker and our message queue. Specifically, the listener will listen on a designated message queue and drag messages from it as long as there are messages available in the queue. Below is the content of my JmsConfig.java file.

@Configuration
@EnableJms
public class JmsConfig {
    @Value("${aws.region}")
    private String region;

    @Bean(name="connectionFactory")
    public SQSConnectionFactory getSQSConnectionFactory(){
        AmazonSQS amazonSQSClient = AmazonSQSClientBuilder.standard().withCredentials(new DefaultAWSCredentialsProviderChain()).withRegion(region).build();
        SQSConnectionFactory factory = new SQSConnectionFactory( new ProviderConfiguration(),amazonSQSClient);
        return factory;
    }

    @Bean
    public JmsTemplate getJmsTemplate(@Autowired SQSConnectionFactory connectionFactory){
        JmsTemplate jmsTemplate = new JmsTemplate(connectionFactory);
        return jmsTemplate;
    }

    @Bean
    public DynamicDestinationResolver getTopicDynamicDestinationResolver(){
        return new DynamicDestinationResolver();
    }

    @Bean(name="jmsListenerContainerFactory")
    @DependsOn("connectionFactory")
    public DefaultJmsListenerContainerFactory getDefaultJmsListenerContainerFactory(@Autowired SQSConnectionFactory connectionFactory,@Autowired DynamicDestinationResolver dynamicDestinationResolver){
        DefaultJmsListenerContainerFactory jmsListenerContainerFactory = new DefaultJmsListenerContainerFactory();
        jmsListenerContainerFactory.setSessionTransacted(false);
        jmsListenerContainerFactory.setConnectionFactory(connectionFactory);
        jmsListenerContainerFactory.setDestinationResolver(dynamicDestinationResolver);
        jmsListenerContainerFactory.setConcurrency(“1”);
        jmsListenerContainerFactory.setSessionAcknowledgeMode(Session.AUTO_ACKNOWLEDGE);
        return jmsListenerContainerFactory;
    }
}

Another implementation without using Java Spring can be found here.

Message Consumer:

Next, let us have a look at how to use this JmsListenerContainer. In the code below, we can designate a message queue to the listener by passing the Amazon SQS queue name to the JmsListener annotation. The JmsListenerContainer will automatically resolve the name, bind a listener to the designated message queue, and start listening. Normally in the producer part, we should send well-organized JSON Strings containing all the necessary information to the message queue for our worker to parse. Here I just hard-coded a message for demo purpose.

@Service
public class ProcessService {
    private final Logger logger = LoggerFactory.getLogger(getClass());

    @Autowired
    private SendGridEmailService sendGridEmailService;

    @JmsListener(destination = MQ_NAME)
    public void processMessage(String msg) throws IOException {
        logger.debug("Processing Message: " + msg);

        Map<String, String> fakeRequest = new HashMap<>();
        fakeRequest.put("username", SRC_USERNAME);
        fakeRequest.put("avatar", AVATAR_URL);
        fakeRequest.put("attached_text", ADDITIONAL_TEXT);
        fakeRequest.put("request_link", REQUEST_LINK);

        Map<String, Object> fakeMessage = new HashMap<>();
        fakeMessage.put("subject", "Friend Invitation");
        fakeMessage.put("from", "test@example.com");
        fakeMessage.put("to_emails", Arrays.asList(EMAIL_1, EMAIL_2));
        fakeMessage.put("to_usernames", Arrays.asList(DST_USERNAME1, DST_USERNAME2));
        fakeMessage.put("request", fakeRequest);

        String fakeMessageJson = new ObjectMapper().writeValueAsString(fakeMessage);

        sendGridEmailService.sendEmail(fakeMessageJson);
    }
}

Once there are messages available, the worker will get the message and process it. Here I just call the sendFridEmailService to send an email to notify the target user there’s a pending friend invitation from the source user.

SendGrid Email Template:

First, let us create some dynamic email HTML templates here and copy the template_ids. In a template, we can put some placeholders between . Later when we use the template, we can pass JSON data to it and SendGrid will replace all placeholders with specific values according to relevant placeholders. An example template is shown below:

<img class="header-icon" src="" alt="" align="left" />
<a href="" rel="noopener"></a>
<div style="margin-top: auto; margin-bottom: auto;">
    <p>Hi ,</p>
    <p> has sent you a friend request, Please check.</p>
    <p style="font-weight: bold;">Attached text:</p>
    <p></p>
</div>

More example templates can be found here here.

SendGrid Email Service Integration:

Let us move to the email service integration. I use Jackson to do JSON parsing and object mapping. Notice that during the first mapping, all the nested JSON objects will be converted to its exact type, so I just make some explicit type casting on those objects, as shown below (requestInfo, toEmails, and toUsernames).

@Service
public class SendGridEmailService {
    private SendGrid sg;
    private String friendInvitationTemplateId = "xxx";
    private String eventInvitationTemplateId = "xxx";
    private String registrationTemplateId = "xxx";

    public SendGridEmailService(@Autowired SendGrid sendGrid) {
        this.sg = sendGrid;
    }

    public void sendEmail(String emailJson) throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        Map<String, Object> emailDetails = mapper.readValue(emailJson, new TypeReference<Map>(){});

        Mail mail = new Mail();
        String subject = emailDetails.get("subject").toString();

        if (subject.equals("Friend Invitation")) {
            mail.setTemplateId(friendInvitationTemplateId);
        } else if (subject.equals("Event Invitation")) {
            mail.setTemplateId(eventInvitationTemplateId);
        } else if (subject.equals("Registration Confirmation")) {
            mail.setTemplateId(registrationTemplateId);
        } else {
            return;
        }

        mail.setFrom(new Email(emailDetails.get("from").toString()));

        Map<String, String> requestInfo = (Map<String, String>) emailDetails.get("request");
        List<String> toEmails = (List<String>) emailDetails.get("to_emails");
        List<String> toUsernames = (List<String>) emailDetails.get("to_usernames");

        Personalization personalization = new Personalization();

        for (String to : toEmails) {
            personalization.addTo(new Email(to));
        }

        personalization.setSubject(subject);
        personalization.addDynamicTemplateData("username", toUsernames.get(0));
        personalization.addDynamicTemplateData("request", requestInfo);

        mail.addPersonalization(personalization);
        Request request = new Request();
        try {
            request.setMethod(Method.POST);
            request.setEndpoint("mail/send");
            request.setBody(mail.build());
            Response response = sg.api(request);
            System.out.println(response.getStatusCode());
            System.out.println(response.getBody());
            System.out.println(response.getHeaders());
        } catch (IOException ex) {
            throw ex;
        }
    }
}

In the code above, we first generate a Mail object which is a helper class to send emails through SendGrid. Then we set the “from” address of this email and the template we are going to use by passing the templated_id. Next, we need to construct our personalized data for the email. Notice that there are several fixed properties in the personalization object, such as the “to” property which is a list, we can add multiple dest email addresses by calling addTo() method in a loop.

For the data we need to fill in the dynamic template, all we need is simply adding (key, value) pairs to the DynamicTemplateData property. The “key” should be exactly the same as the placeholders in the template. Here in the example, I add an object called requestInfo containing the information of a friend invitation (src_username, avatar_url, attached_text, and invitation_link) to simulate the scenario when someone sends a friend invitation to another person.

Unit Test with Mockito:

Mock Configuration:

Before testing, we need to tell Spring which configuration we want to use by setting “unit” profile. In this config file, I create two beans to return fake SQS and SendGrid clients by calling Mockito.mock() method. Because I am testing my sendEmailService(String msg) method, I need to tell the fake SendGird client what to return when its api() method is invoked. More information about Mockito.when.then syntax can be found here

@Configuration
public class MockConfig {
    @Bean
    @Profile("unit")
    public AmazonSQS getAmazonSQSBean() {
        AmazonSQS sqsFake = Mockito.mock(AmazonSQS.class);
        return sqsFake;
    }
    @Bean
    @Profile("unit")
    public SendGrid getSendGridBean() {
        SendGrid sendGridFake = Mockito.mock(SendGrid.class);
        Response fakeResponse = Mockito.mock(Response.class);
        Mockito.when(sg.api(any(Request.class))).thenReturn(fakeResponse);
        return sendGridFake;
    }
}

sendEmailServieTest:

Below is the real Unit Test for my sendEamilService. I hard-code an email request, convert it to a JSON string, and call the method. After calling it, I tell Mockito to verify the invocation times of the api() method of my fake SendGrid client.

@RunWith(SpringRunner.class)
@SpringBootTest
@Import(MockConfig.class)
@ActiveProfiles("unit")
public class SendGridEmailServiceTest {
    @Autowired
    SendGrid sg;
    @Autowired
    SendGridEmailService sendGridEmailService;

    @Test
    public void sendEmailTest() throws IOException {
        Map<String, String> fakeRequest = new HashMap<>();
        fakeRequest.put("username","Ryo");
        fakeRequest.put("avatar","https://easy-planner.s3.amazonaws.com/Wechat_Icon.jpg");
        fakeRequest.put("attached_text", "Hi, I'm the founder of ASCENDING LLC.");
        fakeRequest.put("request_link", "");

        Map<String, Object> fakeMessage = new HashMap<>();
        fakeMessage.put("subject", "Friend Invitation");
        fakeMessage.put("from", "test@example.com");
        fakeMessage.put("to_emails", Arrays.asList("zhangxwsavvy@gmail.com", "xinwei.z@columbia.edu"));
        fakeMessage.put("to_usernames", Arrays.asList("SavvyZ", "Xinwei"));
        fakeMessage.put("request", fakeRequest);

        String fakeMessageJson = new ObjectMapper().writeValueAsString(fakeMessage);

        sendGridEmailService.sendEmail(fakeMessageJson);

        Mockito.verify(sg,  Mockito.times(1)).api(any(Request.class));
    }
}

Now everything is set. We can start up our worker program and send a message to the message queue it is listening. Once the message is sent, our worker will immediately consume the message and extract the information from the message, and send a templated email to all destination users.

Cheers!

Author:

Source Code:

https://github.com/Saewo0/easy-planner-spring-mvc