“icitizen is an award-winning polling platform that connects people and decision-makers to make their community a better place through online polling.”
DevOps Manager
“icitizen is an award-winning polling platform that connects people and decision-makers to make their community a better place through online polling.”
icitizen’s award-winning civic engagement platform was running on Google Container Engine (GKE), a powerful cluster manager and orchestration system for running Docker containers built on Kubernetes. Their in-house team built a significant amount of infrastructure and automation as an early adopter to GKE.
As with many small startups, attrition is an ever-present risk, and several of those most intimate with the Ops infrastructure left over time. Many similar events always pile up, where Murphy’s Law is in play; the perfect storm of a combination of events at nearly the same time overwhelmed the existing / remaining team, taking long hours and making them longer. Resolutions were difficult as folks were migrating to speed on several aspects of the platform and several items of interdependence were causing issues all at once.
Named a 2016 IDC Innovator for its groundbreaking polling and data analytics app, icitizen needed these critical deployment problems fixed as quickly as possible.
icitizen needed a different solution fast, as issues included parts of development and staging being down and concerns with getting new code into prod while not incurring downtime. Google pointed them to ReactiveOps as the right resource to assist the team, and in August 2016 a contractual partnership was reached with RO and icitizen.
RO was asked to assist in triage; assess and prioritize problems as a co-member of the icitizen team and be ready to lay out what the future might hold for ops. There was some redesign, re-implementation and deployments to again move forward. As part of the plan to move forward, RO was enlisted to assist in reducing complexity, to continue to support the technological stack ‘as-is’ and to assist in a planned move to streamline and follow more common practices while moving toward a best-practices scenario.
As time has progressed, teams from RO and icitizen have aligned to further enhance and simplify icitizen’s complex pipeline into a more manageable and maintainable environment that is more stable, simpler and where the platform is behaving in a predictable fashion.
The immediate goal was to get them up and running; the long-term goal was to reinforce their infrastructure, assist in its simplification while maintaining availability and features, to add redundancy and reliability, lower costs and expedite product delivery. Early in the engagement, RO brought assistance with immediate issues, then assisted with planning. Now both teams are looking to next steps to further enhance the platform.
In transitioning icitizen to the ReactiveOps way, RO has assisted with icitizen’s production efforts, enabling their in-house operations to dramatically downsize. The knowledge share from RO has been a success point to the icitizen team as well. ReactiveOps support runs dramatically less than the cost of the company’s former in-house team and adds the 24/7 monitoring and late-night response the team was missing in the past and where resources are not available to fill in for the future. In assisting with operations, RO also gave the company’s software engineers a window into operations and helped with minimizing their struggle with understanding how to make the most of the resources they have by focusing less on ops tedium.
Moving forward, the icitizen / RO teams look forward to reducing additional complexity in the pipeline and having a bulletproof system that has less fragility in getting the code in front of the users as their application.
“ReactiveOps has assisted us with filling the gaps in our team so we can shore up technology debt, has brought additional value in knowledge transfer and has been a critical piece of our monitoring solution for 24/7 coverage.”