Phase 3
This article follows on from part two which you can read here: http://dale-dev.blogspot.com/2019/01/holiday-price-scraper-serverless_10.html, in phase 3 I want to accomplish the following things...
- AWS fails to send emails via Gmail, I have allowed insecure apps in Gmail enabled and still email fails to send, will setup sending emails using SES.
- Implement email templates to use when sending an email with all the data that I would like to see.
- Handle when the holiday is no longer available.
- Complete README.md file
AWS Failing to send email
Looking online there were a few guides on how to send emails using Amazons SES. I refactored my code to allow me to provide a different implementation of a mail sender interface, I wrote the following code to perform the sending of the email
AwsSesConfig.java
@Configuration
public class AwsSesConfig {
@Bean
public AmazonSimpleEmailService client() {
return AmazonSimpleEmailServiceClientBuilder.standard()
.withRegion(Regions.EU_WEST_1).build();
}
}
SesMailService.java
@Service
public class SesMailService implements MailService {
private final Logger log = LoggerFactory.getLogger(this.getClass());
private AmazonSimpleEmailService client;
/**
* @param client
*/
public SesMailService(AmazonSimpleEmailService client) {
this.client = client;
}
@Override
public void sendEmail(String to, String subject, String html, byte[] chart) {
log.debug("Attempting to send email via Amazon SES client : {}", client);
SendEmailRequest request = new SendEmailRequest();
request.setSource("Holiday Price Scraper <daledev.pricescraper@gmail.com>");
request.setDestination(new Destination(Collections.singletonList(to)));
request.setMessage(createEmailMessage(subject, html));
log.debug("Send email request : {}", request);
client.sendEmail(request);
}
private Message createEmailMessage(String subject, String html) {
return new Message()
.withSubject(new Content().withCharset("UTF-8").withData(subject))
.withBody(new Body().withHtml(new Content().withCharset("UTF-8").withData(html)));
}
}
When running the code I got the following error:
"errorMessage": "User `arn:aws:sts::675239832622:assumed-role/MyPriceHistoryDynamoDB/holidayPriceChecker' is not authorized to perform `ses:SendEmail' on resource `arn:aws:ses:eu-west-1:675239832622:identity/daledev.pricescraper@gmail.com' (Service: AmazonSimpleEmailService; Status Code: 403; Error Code: AccessDenied; Request ID: b120b161-176b-11e9-888e-8ddbf253bcdf)",
This was just a matter of using IAM to grant the following permission "AmazonSESFullAccess" to my role that the Lambda function uses.
Registering the sending email address
Email address is not verified. The following identities failed the check in region region: identity1, identity2, identity3—You are trying to send email from an email address or domain that you have not verified with Amazon SES. This error could apply to the "From", "Source", "Sender", or "Return-Path" address. If your account is still in the Amazon SES sandbox, you also must verify every recipient email address except for the recipients provided by the Amazon SES mailbox simulator. If Amazon SES is not able to show all of the failed identities, the error message ends with an ellipsis.
I followed the instructions and registered my email address but I kept getting an error about an unverified email address, the email address is reported in the error was actually the recipient, nowhere in the above-quoted text did it mention registering recipient email addresses and that would not make any sense to have to do so to me, however I tried it and it worked which was a bit of a head-scratcher. Further searching I found the explanation for this here: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/request-production-access.html ...
To help prevent fraud and abuse, and to help protect your reputation as a sender, we apply certain restrictions to new Amazon SES accounts.
We place all new accounts in the Amazon SES sandbox. While your account is in the sandbox, you can use all of the features of Amazon SES. However, when your account is in the sandbox, we apply the following restrictions to your account:
You can only send mail to verified email addresses and domains, or to the Amazon SES mailbox simulator.
You can only send mail from verified email addresses and domains.
Note
This restriction applies even when your account is not in the sandbox.
You can send a maximum of 200 messages per 24-hour period.
You can send a maximum of 1 message per second.
When your account is out of the sandbox, you can send email to any recipient, regardless of whether the recipient's address or domain is verified. However, you still have to verify all identities that you use as "From", "Source", "Sender", or "Return-Path" addresses.
I wish they made some reference to that in the registering email addresses section, would have saved a little pain but I got it working in the end.
Email Templates
public interface MailService {
void sendPriceDropMail(PriceHistory priceHistory);
void sendPriceIncreaseMail(PriceHistory priceHistory);
}
I decided to play safe here and go with a framework I was already familiar with, Apache Velocity. What I wanted to be displayed in the email is a little summary explain that the price has gone up or down, a table showing the history of prices and a line chart. The velocity templates I create for these emails
price-increase.vm
<html>
<head>
#parse ("static/style.vm")
</head>
<body>
<h2>$data.criterion.description</h2>
<b>Price Increased : £$data.lastRecordedPrice.price</b>
#parse ("static/price-history-table.vm")
<p>
<img src="$reportImageSrc"/>
</p>
</body>
</html>
price-drop.vm
<html>
<head>
#parse ("static/style.vm")
</head>
<body>
<h2>$data.criterion.description</h2>
<table>
<tr>
<td>Price Dropped</td>
<td style="font-weight: bold;">£$data.lastRecordedPrice.price</td>
</tr>
<tr>
<td>Cheapest ever captured?</td>
<td style="font-weight: bold;">
#if($data.currentPriceCheapestCaptured)
Yes
#else
No
#end
</td>
</tr>
</table>
#parse ("static/price-history-table.vm")
<p>
<img src="$reportImageSrc"/>
</p>
</body>
</html>
price-history-table.vm
<h2>Price History</h2>
<table>
<tr style="font-weight: bold;">
<td width="150">From</td>
<td width="150">To</td>
<td width="150">Depart Date</td>
<td>Price</td>
</tr>
#foreach($history in $data.history)
<tr>
<td>$date.format('dd/MM/yyyy HH:mm', $history.firstRetrievedDate)</td>
<td>$date.format('dd/MM/yyyy HH:mm', $history.lastRetrievedDate)</td>
<td>$date.format('dd/MM/yyyy (EEE)', $history.departDate)</td>
<td>£$history.price</td>
</tr>
#end
</table>
I used a library called JFreeChart to generate the chart from the data in the Price History Entity. Embedding the chart into the email turned out to be a bigger pain in the ass than I ever realized, there is a nice article detailing different problems with the different ways to embed an image https://sendgrid.com/blog/embedding-images-emails-facts/ so what I decide to do was to upload the generated image to Imgur and embed the URL tot he image uploaded there which seems to work.
This is the kind of email I have ended up with...
As you can see the has taken a dive the last week, about £100 less than I paid 😬
Handle when the holiday is no longer available
public class FirstChoiceFetchUrlBuilder {
private static final String SCRAPE_URL = "https://www.firstchoice.co.uk/holiday/packages";
private String airportCodes;
private String accommodationRef;
private String date;
private String until;
private int flexibleDays = 0;
private int numberOfAdults = 0;
private int[] childrenAges = new int[0];
private int durationCode;
private String searchRequestType = "ins";
private String searchType = "search";
private boolean sp = true;
private boolean multiSelect = true;
private String room;
private boolean isVilla = false;
private FirstChoiceFetchUrlBuilder() {
}
public static FirstChoiceFetchUrlBuilder get() {
return new FirstChoiceFetchUrlBuilder();
}
/**
* @return
*/
public String build() {
StringBuilder url = new StringBuilder(SCRAPE_URL);
url.append("?airports[]=").append(airportCodes);
url.append("&units[]=").append(accommodationRef);
url.append("&when=").append(date);
url.append("&until=").append(blankStringNulls(until));
url.append("&flexibility=").append(flexibleDays > 0);
url.append("&flexibleDays=").append(flexibleDays);
url.append("&noOfAdults=").append(numberOfAdults);
url.append("&noOfChildren=").append(childrenAges.length);
url.append("&childrenAge=").append(IntStream.of(childrenAges).mapToObj(String::valueOf).collect(Collectors.joining("|")));
url.append("&duration=").append(durationCode);
url.append("&searchRequestType=").append(searchRequestType);
url.append("&searchType=").append(searchType);
url.append("&sp=").append(sp);
url.append("&multiSelect=").append(multiSelect);
url.append("&room=").append(blankStringNulls(room));
url.append("&isVilla=").append(isVilla);
return url.toString();
}
public FirstChoiceFetchUrlBuilder withAirports(Airport[] airports) {
this.airportCodes = Stream.of(airports).map(Airport::getCode).collect(Collectors.joining("|"));
return this;
}
public FirstChoiceFetchUrlBuilder withAccommodationRef(String accommodationRef) {
this.accommodationRef = accommodationRef;
return this;
}
public FirstChoiceFetchUrlBuilder withDate(String date) {
this.date = date;
return this;
}
public FirstChoiceFetchUrlBuilder withUntil(String until) {
this.until = until;
return this;
}
public FirstChoiceFetchUrlBuilder withFlexibleDays(int flexibleDays) {
this.flexibleDays = flexibleDays;
return this;
}
public FirstChoiceFetchUrlBuilder withNumberOfAdults(int numberOfAdults) {
this.numberOfAdults = numberOfAdults;
return this;
}
public FirstChoiceFetchUrlBuilder withChildrenAges(int[] childrenAges) {
this.childrenAges = childrenAges;
return this;
}
public FirstChoiceFetchUrlBuilder withDuration(int durationInDays) {
if (durationInDays == 14) {
this.durationCode = 1413;
} else if (durationInDays == 10 || durationInDays == 11) {
this.durationCode = 1014;
} else {
this.durationCode = 7114; // 7 days
}
return this;
}
public FirstChoiceFetchUrlBuilder withSearchRequestType(String searchRequestType) {
this.searchRequestType = searchRequestType;
return this;
}
public FirstChoiceFetchUrlBuilder withSearchType(String searchType) {
this.searchType = searchType;
return this;
}
public FirstChoiceFetchUrlBuilder withSp(boolean sp) {
this.sp = sp;
return this;
}
public FirstChoiceFetchUrlBuilder withMultiSelect(boolean multiSelect) {
this.multiSelect = multiSelect;
return this;
}
public FirstChoiceFetchUrlBuilder withRoom(String room) {
this.room = room;
return this;
}
public FirstChoiceFetchUrlBuilder withVilla(boolean villa) {
isVilla = villa;
return this;
}
private String blankStringNulls(Object value) {
return value == null ? "" : value.toString();
}
}
This simplified the First Choice DAO code. TheFirstChoicePriceScraper,java code required the changes when the holiday is no longer available there
private JSONArray getPriceJsonArray(String json) {
try {
JSONObject jsonObject = new JSONObject(json);
// searchResult > holidays > dateSliderViewData
if (jsonObject.has("searchResult")) {
JSONObject searchResultJsonObject = jsonObject.optJSONObject("searchResult");
return searchResultJsonObject.optJSONArray("dateSliderViewData");
} else {
log.debug("Holiday result JSON contains no search results");
return new JSONArray();
}
} catch (Exception e) {
log.error("Could not extract price from search result JSON", e);
return new JSONArray();
}
}
The JSON holding price data contained searchResult attribute, the text displayed to the user when the holiday is no longer available is also extractable from a different portion of the response HTML, I added the following method to extract that.
private HolidayQuoteResults discoverNoPriceReason(Document htmlDocument) {
HolidayQuoteResults results = new HolidayQuoteResults();
Element holidayRemoveBannerElement = htmlDocument.getElementById(HOLIDAYS_ALL_GONE_ELEMENT_ID);
if (holidayRemoveBannerElement != null) {
log.debug("Holiday all gone element found : {}", holidayRemoveBannerElement.text());
results.setStatus(QuoteResultStatus.ALL_GONE);
}
return results;
}
With this information, I now make this DAO return a response object which includes the fetch status instead of just prices so that I can inform the calling DAO that the holiday is no longer available
Source Code
What's Next in Part 4
- Deploy in Azure
- Support TUI Scraping
