A collection of arcane, and perhaps at times the most inane of my thoughts.

Deployment and validation woes

Had made a minor addition to the existing API to prevent duplicate payments - a point of trouble that came up a month back. Since we had to bear the cost of duplicate payments, decided we'll add a check to check bills for markers like date, biller, and such.

After a couple of iterations, made a final change in the morning - json cast data does remind me of the older days of working with embedded objects in Mongodb.

Had setup an alert with Metabase to track any failed cases where the fix didn't work as intended. Three entries of this kind is low considering the 1500 odd bills paid today, but that's nonetheless, a painful rabbit hole.

Since the queries when run gave the intended results, was flummoxed to be right. Thankfully, have smart people to help with - team work does win. Sat with a colleague to track things down and a few good query changes, new repository tests, and replacing the select with a select exists.

And then the bulb started glowing when they asked the question of when deployment happened. We're on ECS and had a ~3x container count for about an hour post the CI's release workflow. Appears, there has been a mix of old and new deployments during the timeframe when the 3 failures happened.

Does raise a few questions in my mind about how this works and the need to take a bit more time and sit with - to understand these beyond what I do now.

August 2021 (3)July 2021 (1)June 2021 (1)May 2021 (4)April 2021 (6)March 2021 (6)February 2021 (1)November 2020 (11)