Chewy out

[infryq]

Yesterday the aggregations for F were extremely late due to the continued problems with the API. I went to the big group meeting and delivered our update, then spent the rest of the day monitoring the continuing work fixing the API, running off some gold results for A’s work on the validation system, fixing the aggregations, and scheduling an OS upgrade for W’s GPU machines. Re the API, there are two problems: direction calculations were using SQL too advanced for our database version, and metadata calculations we’re running out of storage. For directions, the darn thing got three hours into a test job to fix it and then dropped everything on the floor and had to be restarted, so that was not ideal. Meanwhile, we added more storage to the production server and tried the meta job again, and it took 5 hours but ultimately succeeded...except one of the changes we’d neglected to test broke the map. I generated a hot fix for B to apply, and a PR for a permanent fix, and the hotfix worked so I merged the PR and we set up automation to only run the meta updater once a day so it can run overnight.

Dinner, crossword, noodled around on Netflix and found a season of Lucifer I hadn’t seen, so started rewatching the end of season 3 to refresh my memory.

Snack, sleep. Dreamt of weird reunions, social dance competitions, architecture based on logic puzzles, accidentally wrecking a wooden ladder in an attempt to fix it (thanks, subconscious, I’m aware of that), giving and getting fantastic advice.