In March of 2023, the influential online conference, DevOps fwdays’23, was dedicated to sharing best practices and tools in the DevOps sector. The lineup featured speakers hailing from renowned tech companies such as Softserve, Spotify, Luxoft, Snyk, Solidify, Zencore, Mondoo, and P2H, just to name a few.
Mykyta Savin, DevOps Infrastructure Architect at P2H, presented “How We Block Production: Triangulating Issues, Solutions, and Postmortems,” highlighting the impact of minor errors on production and providing strategies for resolution.
While we’re eager to share this valuable insight, we hope you won’t need it.
Project Overview:
P2H spearheads the development of an E-Government platform meticulously addressing the intricacies of integrating with government systems. This platform streamlines interactions with the country’s labor market for both citizens and businesses. Embracing an asynchronous architecture, the platform continually evolves with new services, leveraging a robust tech stack and meticulous processes.
Tech Stack and Processes:
- Microservice architecture
- Front end: Vue.js, React.js
- Back end: Ruby, Ruby on Rails, Java, PHP
- Message broker: RabbitMQ
- Global cache: Elasticsearch
- Infrastructure: Docker
- Monitoring, observation, and tracing: Grafana, Grafana Loki, Grafana Tempo, Prometheus, OpenTelemetry, Vector
- Integrations: IBM APP Connect, IBM API Connect, Absher, Unifonic, Mada, SADAD, and more
The project employs a microservice architecture, boasting over a hundred microservices currently in production, predominantly developed in Ruby, with Java powering the latest additions. Adopting the Enterprise Service Bus (ESB) pattern and leveraging RabbitMQ as the message broker facilitates seamless handling of the project’s asynchronous operations.
Elasticsearch and PostgreSQL form the robust storage layer, complemented by Docker and Docker Compose for streamlined infrastructure management. To ensure compliance with government regulations, an internal provider from Saudi Arabia is utilized, addressing data locality requirements.
Monitoring is facilitated through the Grafana Stack, integrated with various ministries and private institutions. RabbitMQ operates as a resilient cluster of four nodes, accessible via the prod-rabbit-new-lb load balancer.