From a Software Project to a Cutting-Edge Tech Business. A Developer’s Experience
The beginnings: feeling lost
Today, the Optimal Shelf Availability Hybrid Platform (OSA HP) is an AI-based service that ensures optimal on-shelf availability of goods in retail stores in real time. Sounds impressive now, but here’s how it all started.
It was back in December 2014, on a chilly day when I arrived into our half-empty Kyiv office and met my new team – a local development group of 5 people only 3 of whom were actual IT guys. We didn’t have any system requirements or specifications, nor did we understood the functionality of the end product. So it was up to us what exactly to do, using what technologies, and when the deadlines would be. Naturally. We accepted the challenge.
As everybody was the most comfortable with PHP, we didn’t have to think much about what programming language to use. Picking the framework was also straightforward – we anticipated large amounts of data and high loads so we went with Phalcon PHP which, thanks to its architecture, was the fastest PHP framework available. So we really only had to think hard of what to do about the database. We still hadn’t seen the actual raw data that would be coming from the clients, but we knew that we’d need to frequently and quickly integrate large amounts of data of unknown structure and high cohesion. We tested MySQL, PostgreSQL, and MongoDB, and ended up choosing the latter as it showed amazing insert speed, had flexible schema and could do MapReduce.
I’m gonna skip over the drudgery of initial architecture design and the first development stages. In the beginning, both the data transporting and analytic components were implemented in PHP which was later augmented by Gearman, Redis cache, and was migrated to Amazon’s cloud. Next, the transporting component spun off into its own project which is now under the management of the data wrangling team, while the analytical component transformed into a fully-fledged data science module worked on by another team.
Certainty that brings the need for deeper knowledge
Then we were given the task of further developing the analytical module – do rocket science analytics based on receipts register coming in real-time from the retailers. It became clear that we needed to have somebody with a very deep corresponding knowledge and so data scientists came on board and started working on algorithms and models taking us into the world of Python and R.
This research was separate from the development, and so the latter was still done in PHP for a while. In the meantime, the data science team was beating the records daily detecting more and more accurately the instances of out-of-shelf. By that measure, we were ahead of everybody else.
We faced our first real roadblock when the demo was already up and running. The IT director of a client thought we needed a reality check and said there’s no way he’d have a non-relational database and asked to change it to something more traditional. The happy MongoDB chapter of our company ended and a hectic migration to PostgreSQL begun. In the end though, we did succeed.
In the meantime, the development team was growing and at some point, the data transfer and data analytics parted into separate Python projects led by different teams. AWS was getting prohibitively expensive and we started bringing in our own dedicated servers. A DevOps engineer joined the team trying to integrate us with Docker containers. We eventually had to scrap that idea but it was a good experience.
Regrouping before charging further ahead
This is where we are. The technology selection for the project has been established, for now. The development itself has been spurring internal evolution that introduced us to Cassandra, Spark, Scala, Couchbase, ClickHouse, Yii2, and SOA. In addition, a new architecture was born, one able to take in lots of data and therefore more clients. But that’s a whole different story.
In the next post – testing, epically failing in databases, algorithms, and programming languages, and finally succeeding, facing challenges we couldn’t imaging before the real players picked up interest in us. At any rate, we made all the mistakes required of building an IT business that includes developing innovative software products under uncertainty. The most rewarding was ultimately transforming a software project into a profitable cutting-edge technology business.
To be continued