Identifying the Problem
Dataninja, a retail analytics consulting firm, was building powerful analysis and forecasting tools using Excel for suppliers to the world’s largest retailer. A handfull of home grown software scripts for fetching retail reports kept their business manageable, but when a partnership opportunity presented itself to increase their client base five-fold there simply weren’t enough hours in the day to satisfy demand.
Dataninja had a vision for improvements to their workflow, but needed technical assistance on the leap from local machine to cloud based software. Executed properly, Dataninja envisioned their in-house tools being expanded to a software as a service solution for other firms as a new source of revenue.
Lofty Labs investigated Dataninja’s workflow and built a distributed application to manage their reporting pipeline, using Microsoft Azure as a cloud provider.
Leveraging tools like Celery and PostgreSQL, Lofty consultants facilitated concurrency in report retrieval and processing with multiple servers simultaneously on highly configurable scheduled intervals. This ingest layer was connected to configurable output interfaces for maximum flexibility on how to post-process reports.
Finall, Lofty consultants implemented a REST API on top of the new data warehouse, removing a layer of complexity where each customer’s data was stored in an isolated relational database. The API layer moved significant amounts of non-testable proprietary database logic into unit testable Python code which could be adapted to any database system.
Increased Operational Capacity
Dataninja grew their consultancy from 9 to over 50 clients on the new application infrastructure, and reduced the number of managed databases from 9 to 1. This simplified architecture meant customer onboarding was reduced from a matter of days to just minutes.
The configurable pipeline output enabled Dataninja to fork processing: Internal client data could be post processed and stored for analysis but external organizations could receive raw reports via FTP or directly to attached storage devices.
The ability to push reports became the basis for a licensed software product, enabling a new type of customer to leverage Dataninja’s report automation and scheduling in their own analytics workflows. Dataninja began piloting this product with customers just one month into its engagement with Lofty Labs.