Phase 2
So this article follows on from part one which you can read here: http://dale-dev.blogspot.com/2019/01/holiday-price-scraper-serverless.html, in phase 2 I want to accomplish the following things...
- The function in AWS is not working when running when triggered by the timer, this needs resolving.
- The function in AWS is not persisting the history of retrieved prices, I will be creating a DynamoDB DAO to persist this data.
- Search criteria to be persisted as saved searches rather than hard coded.
AWS Trigger issue
The timer job I set up in AWS which triggers the function fails, the error from the log shows the following...
REPORT RequestId: db8ec301-0fbe-11e9-902c-ad0f3698404b Duration: 442.69 ms Billed Duration: 500 ms Memory Size: 512 MB Max Memory Used: 98 MB
START RequestId: db8ec301-0fbe-11e9-902c-ad0f3698404b Version: $LATEST
An error occurred during JSON parsing: java.lang.RuntimeException
java.lang.RuntimeException: An error occurred during JSON parsing
Caused by: java.io.UncheckedIOException: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
at [Source: lambdainternal.util.NativeMemoryAsInputStream@78b729e6; line: 1, column: 1]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
at [Source: lambdainternal.util.NativeMemoryAsInputStream@78b729e6; line: 1, column: 1]
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:857)
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:62)
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1511)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1102)
END RequestId: db8ec301-0fbe-11e9-902c-ad0f3698404b
From that, it's showing that there is a problem de-serializing the input from the trigger. I didn't actually configure an input, I assumed a null value would be passed and everything would be ok but I assumed wrong (What's that they say about assumptions?).
So going back into AWS, I can see there is an option to configure the input passed from the cloud event so I made the following change
DynamoDB Persistence
Setup
Under DynamoDB, I have selected "Create Table" and enter the following...
| Table Name : | PriceHistory |
| Primary key : | searchUUID |
New DynamoDB DAO
Following this guide: https://www.javaworld.com/article/3248595/application-development/serverless-computing-with-aws-lambda-part-2.amp.html I created the following code
pom.xml
DynamoDBPriceStoreDao
Got the following error when running the function in AWS ...
pom.xml
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>1.11.475</version>
</dependency>
DynamoDBPriceStoreDao
package com.daledev.holidaypricescrapper.dao.aws;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.document.*;
import com.daledev.holidaypricescrapper.dao.PriceStoreDao;
import com.daledev.holidaypricescrapper.domain.PriceHistory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Repository;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
/**
* @author dale.ellis
* @since 04/01/2019
*/
@Repository
public class DynamoDBPriceStoreDao implements PriceStoreDao {
private final Logger log = LoggerFactory.getLogger(this.getClass());
@Override
public List<PriceHistory> getPriceHistories() {
List<PriceHistory> histories = new ArrayList<>();
log.debug("Scanning For all PriceHistory items");
ItemCollection<ScanOutcome> scanResults = getHistoryTable().scan();
for (Item item : scanResults) {
log.debug("Item {} from results returned in results", item);
PriceHistory priceHistory = PriceHistoryMapper.mapToPriceHistory(item);
if (priceHistory != null) {
histories.add(priceHistory);
}
}
log.debug("Returning {} price histories", histories.size());
return histories;
}
@Override
public PriceHistory getPriceHistory(String searchUuid) {
log.debug("Attempting to retrieve PriceHistory with ID : {}", searchUuid);
Item persistedHistory = getHistoryTable().getItem(PriceHistoryMapper.PRIMARY_KEY_NAME, searchUuid);
if (persistedHistory != null) {
log.debug("History found in DynamoDB. {}", persistedHistory.toJSONPretty());
return PriceHistoryMapper.mapToPriceHistory(persistedHistory);
}
return new PriceHistory();
}
@Override
public void storePrice(PriceHistory priceHistory) {
log.debug("Storing history with ID : {}", priceHistory.getSearchUUID());
Item item = PriceHistoryMapper.mapFromPriceHistory(priceHistory);
PutItemOutcome putItemOutcome = getHistoryTable().putItem(item);
log.debug("Create response : {}", putItemOutcome);
}
private Table getHistoryTable() {
log.debug("Creating Table API object for table : {}", PriceHistoryMapper.TABLE_NAME);
DynamoDB dynamoDB = getDynamoDB();
return dynamoDB.getTable(PriceHistoryMapper.TABLE_NAME);
}
private DynamoDB getDynamoDB() {
log.debug("Creating AmazonDynamoDB client");
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.defaultClient();
log.debug("AmazonDynamoDB client : {}", client);
return new DynamoDB(client);
}
}
Got the following error when running the function in AWS ...
User: arn:aws:sts::675239832622:assumed-role/holidayCheckRole/holidayPriceCheck is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:eu-west-2:675239832622:table/PriceHistory (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: K8T3CA4G8RMTBINN546UDMI5RFVV4KQNSO5AEMVJF66Q9ASUAAJG): com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::675239832622:assumed-role/holidayCheckRole/holidayPriceCheck is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:eu-west-2:675239832622:table/PriceHistory (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID: K8T3CA4G8RMTBINN546UDMI5RFVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4192)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4159)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeGetItem(AmazonDynamoDBClient.java:2028)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.getItem(AmazonDynamoDBClient.java:1995)
at com.amazonaws.services.dynamodbv2.document.internal.GetItemImpl.doLoadItem(GetItemImpl.java:77)
at com.amazonaws.services.dynamodbv2.document.internal.GetItemImpl.getItemOutcome(GetItemImpl.java:40)
at com.amazonaws.services.dynamodbv2.document.internal.GetItemImpl.getItemOutcome(GetItemImpl.java:99)
at com.amazonaws.services.dynamodbv2.document.internal.GetItemImpl.getItem(GetItemImpl.java:111)
at com.amazonaws.services.dynamodbv2.document.Table.getItem(Table.java:624)
at com.daledev.holidaypricescrapper.dao.aws.DynamoDBPriceStoreDao.getPriceHistory(DynamoDBPriceStoreDao.java:28)
at com.daledev.holidaypricescrapper.service.PriceRetrieverServiceImpl.priceCheck(PriceRetrieverServiceImpl.java:82)
at com.daledev.holidaypricescrapper.function.PriceCheckFunction.apply(PriceCheckFunction.java:30)
at com.daledev.holidaypricescrapper.function.PriceCheckFunction.apply(PriceCheckFunction.java:15)
at org.springframework.cloud.function.core.FluxFunction.lambda$apply$0(FluxFunction.java:47)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107)
at reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:99)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:156)
at reactor.core.publisher.BlockingIterable$SubscriberIterator.onSubscribe(BlockingIterable.java:214)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)
at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70)
at reactor.core.publisher.FluxMapFuseable.subscribe(FluxMapFuseable.java:63)
at reactor.core.publisher.Flux.subscribe(Flux.java:6877)
at reactor.core.publisher.BlockingIterable.iterator(BlockingIterable.java:80)
at org.springframework.cloud.function.adapter.aws.SpringBootRequestHandler.result(SpringBootRequestHandler.java:54)
at org.springframework.cloud.function.adapter.aws.SpringBootRequestHandler.handleRequest(SpringBootRequestHandler.java:49)
The lambda function required the role is assigned to have AmazonDynamoDBFullAccess granted, I added this in the AWS IAM (Identity and Access Management) area.
Persist Search Criteria
I modified the JSON document to also hold the search criteria as well as the history of price changes, example content... {
"searchUUID" : "495f1431-5526-4083-997d-f91849be966b",
"criterion" : {
"description" : "Holiday Village in KOS 1 week in May-June 2019",
"startDate" : "2019-05-01T00:00:00.000+0000",
"endDate" : "2019-06-30T00:00:00.000+0000",
"airports" : [ {
"code" : "STN",
"uniqueId" : null
} ],
"duration" : 7,
"accommodationRef" : "047406:HOTEL",
"numberOfAdults" : 2,
"childrenAges" : [ 2 ]
},
"history" : [ {
"firstRetrievedDate" : "2019-01-06T21:35:59.700+0000",
"lastRetrievedDate" : "2019-01-06T21:53:00.319+0000",
"cheapestQuotes" : [ {
"airport" : {
"code" : "STN",
"uniqueId" : null
},
"date" : "2019-05-08T00:00:00.000+0000",
"price" : 613.0
} ]
}, {
"firstRetrievedDate" : "2019-01-07T14:16:22.610+0000",
"lastRetrievedDate" : "2019-01-07T17:47:00.076+0000",
"cheapestQuotes" : [ {
"airport" : {
"code" : "STN",
"uniqueId" : null
},
"date" : "2019-05-08T00:00:00.000+0000",
"price" : 611.0
} ]
}, {
"firstRetrievedDate" : "2019-01-08T12:34:23.896+0000",
"lastRetrievedDate" : "2019-01-08T12:34:23.896+0000",
"cheapestQuotes" : [ {
"airport" : {
"code" : "STN",
"uniqueId" : null
},
"date" : "2019-05-07T23:00:00.000+0000",
"price" : 613.0
} ]
} ],
"subscribers" : [ "dale.ellis@netcall.com", "daleellis1983@gmail.com" ],
"active" : true
}
Other Changes
I did a pretty major refactor of my project, I split my code into a multi-module project, the issue I was having was that I was bundling the controller code for running the app locally in the AWS build, when running locally, the way I had it setup wouldn't compile as there were AWS dependencies that where mandatory which weren't included in a local build so I was getting errors.
New structure...
| Module Name | Description |
| holiday-price-scraper-aws | DynamoDB implementation for persistence, Spring cloud function adapter for AWS. |
| holiday-price-scraper-boot | Main application, business logic |
| holiday-price-scraper-local-rest | File system implementation for persistence, REST API to test locally |
| holiday-price-scraper-persistence | Contains domain objects and DAO interfaces |
Source Code
What's Next in Part 3
The desire is to be able to deploy this to other FaaS providers, but before I tackle Azure, I would like to tidy up some loose ends first...
- AWS fails to send emails via Gmail, I have allowed insecure apps in Gmail enabled and still email fails to send, will setup sending emails using SES.
- Implement email templates to use when sending an email with all the data that I would like to see.
- Handle when the holiday is no longer available.
- Complete README.md file
