PZ Cussons had a wealth of data in on-premise systems and in other system managed by third parties in their overseas subsidiaries. Additionally, being an FMCG company PZ Cussons consume considerable amounts of third-party market data to better understand their customers and markets.
PZ Cussons has a requirement to prove value in quickly ingesting and joining data, enabling the cross-analysis between their own internal data and that sourced from their retail partners and market analysis businesses through multiple Proof of concepts.
Why Amazon Web Services
PZ Cussons opted for a solution powered by AWS for their proof-of-concept because it had all the services and providers needed to execute the first tasks as well as additional capabilities to support future data driven use cases.
Seeking an optimised environments strategy, PZ Cussons also required that their environments to be fully scripted from end to end to allow repeatable patterns. Terraform was used to create all the native AWS services and was extended to create all database objects too.
To enable maximum automation, flat file imports would have the row counts, column counts, and data types verified using AWS Lambda functions. Once verified, the ETL environment would be initiated using SQS; this orchestrated the instantiation of Matillion as well as other tasks. SES (Simple Email Service) was also used to ingest email attachments.
Outside of the native AWS tooling, partner technologies running on AWS were adopted including
- Matillion for ETL
- Snowflake for the data warehouse
- Tableau Online for reporting and visualisation
Crimson Macaw were able to bootstrap a production, test and development environments quickly enabling the proof-of-concept project to ingest data feeds, transform the ingested data and create reports after just a few days. This enabled PZ Cussons to speed up delivery whilst still having certainty over their environments and testing. Being Scripted, PZ Cussons knew exactly what was being deployed right down to warehouse, databases and tables, users access was also provisioned through a configuration in Terraform ensuring access was always provisioned as needed.
Data was ingested from multiple sources including zipped flat files, databases and even files attached to emails which were ingested via SES (Simple Email Service) and then the attachments scanned and extracted to S3 before triggering the ingestion into the data warehouse. By validating these files as part of an automated ingestion process, Crimson Macaw could prevent ‘bad’ data from being processed into the warehouse.
PZ Cussons were able to use Snowflake and Tableau to give them to new insights, replacing existing Excel graphs and charts with a new suite of Tableau dashboards which not only automated much of the manual work, saving time and money, but also enhanced existing analyses used by the category team to supply better metrics and insights into both their data and their key competitors.
By using Tableau Online they were able to create and share new and easy to use reports and data visualisations, with a host of slice and dice features to get different teams excited and interested in their data, as well as supplying published data sets to give users the power to create their own insights, the project was able to quicky increase company engagement in analytics and insight.
Alex McCallum, Data Strategy Consultant said,
Crimson Macaw stood-up a proof-of-concept analytics stack using AWS, Snowflake and Tableau and supplied experienced and knowledgeable architecture advisory, engineering and visualisation resource. Importantly, Crimson Macaw were able to provide consistency of named resource on a flexible basis which fitted PZ Cussons evolving budget as they grew out their proof-of-concept data platform and analytics capability. As such, Crimson proved themselves to be a great partner for an organisation, greenfield in analytics that is evolving this new insight and automation capability at speed. Crimson’s approach enabled PZ Cussons to minimise the upfront investment and risk associated with building out what can be a cost hungry capability with sometimes elapsed time for return as a Business takes time to absorb the new capability into their ways of working, make improved decisions and act differently.
Abby Adderley, UK IT Director said,
I was very impressed at how quickly Crimson was able to stand up an environment for PZC to ingest and store its data, and even more impressed at the level of expertise applied to the initiative to help us demonstrate the power and business value the data insights could bring to help define our strategy. I have seen these projects over the years absorb significant sums of money and effort upfront with lengthy complex projects that can take many months/years before the business start to really feel the value, this project was a refreshing change to this.