Mana, a London based ed-tech startup, was stepping into the next phase of its growth journey. The client's platform provides a curated directory of experts which mainly includes tech leaders, career advisors, lifestyle gurus and teachers. It allows the users to bring content from Youtube, Instagram, Udemy, TikTok and other platforms in a portfolio for their followers.
Led by visionaries and backed up by a passionate team of technologists, the company's main focus remains on boosting the quality of engagement for the user and keeping up with the evolving trends. Behind a neat looking user interface, a massive amount of data gets generated that has significant potential to provide actionable insights for the user and the business.
The data analysts were struggling with the quality of data, at the moment even the aggregated data in RDS contained usernames that could be changed an infinite number of times unlike the static user id's, this would result in incorrect data for analysis.
The client needed a more efficient way to analyse and gain insights from its log data. It is imperative that business has the right count of profile views, count of content/session views and referral sources.
Colibri was approached by the client for an architecture review and data lake implementation
After a few meetings and a discovery call with the CEO and the Director of Engineering, our architects performed a thorough assessment of the existing stack and made recommendations and suggestions to revamp the architecture and build a smart data pipeline leveraging DataOps.
Working closely with the client's developers and ML engineers we made a list of data points captured from the web app. Our lead cloud engineer carefully crafted the data schema and data capture design for data collection in Dynamo DB and provisioned an S3 data lake bucket hooked to AWS lambda functions connected with Amazon SQS and AWS Kinesis. After the client's approval on using Terraform for infrastructure deployment, Python Lambdas were implemented to consume data from the app. Each implementation task included; unit testing, CICD, logging and alerting.
At the time of receiving the data, we directly joined the click event data with the client's RDS database to get the user_id for that user and then stored that in the raw layer of the S3 datalake, allowing the aggregation to now be done against the userid, which is static.
Meilisearch was also productionised, leveraging ECS, Fargate and EFS and configured to pick up data for incremental loads on RDS for enhanced search.
The new data architecture has boosted innovation, the client has an agile architecture that supports evolving business and technical needs. The data analysts have accurate data at their disposal for prescriptive and predictive analysis, providing them with a precise view of the user's journey, even if a user changes their name, the clicks would carry over. They're able to track time spent on each card, tap interactions, likes, top searches and build custom metrics, based on which the users have clear insights enabling them to enhance their profile engagement and make learning obsessive for their followers.
With the IaC approach, the data engineering team can be more nimble and equipped to adapt to the changes quickly in a secured manner using Terraform to deploy workloads. Allowing them to explore more possibilities, and not only to keep the platform functional but also introduce new features, providing a beautiful experience to the user.
The all new search service now allows for a dramatically enhanced user experience - with features like type-ahead completion and multi-faceted queries. This new, decoupled solution enables search indexes and search data to scale and grow independently of other application databases.
Retaining the creative edge for continuous product development, the Mana team is ready to onboard more users and their followers.
Colibri supported us in our critical transition from a SaaS tool into a consumer learning marketplace, building the data architecture to enable world-class search, user analytics and recommendations. Their expertise was invaluable for our long-term success.
James Lo
Co-founder & CEO, Mana