Privacy, Duplicati, Timestamps, OVH

Watched a few privacy documentaries with friends. Sure, all seeing eye, panopticon and total surveillance state are here to stay. Of course we've setup means to communicate securely in case, where we can safely assume (as usual) that all communications are being monitored. Everyone should be able to run secure convert communication network with flexible routing.
Duplicati - I'm wondering if there is anyway to make partial or limited compact? In some cases the data set is so huge, that running full compact will take too long. It would be much better to have the compaction to run more often, but to limit it to partial data set. As example many database systems allow operations like this, limiting the time and resources the data compaction takes.
Duplicati - Did some additional data integrity tests and concluded that Duplicati repair is absolutely broken. It says everything is good, list-broken-files, purge-broken-files and repair are ok. But when you try to restore it says files are missing and restore completely fails. - This is very bad. Really huge ticking time bomb. - It seems that the repair process is fatally flawed. It's not that complex after all. Check that data is good, and if not, get rid of broken data and only broken data and then continue. But that's not working. It's not getting rid of all broken data, which prevents restoring completely and repair functionality doesn't detect that the data set is broken. What a nightmare. - I've tested that at least with four completely separate independent data sets, and the result is always the same. It's also unfortunately safe to assume, this is quite common issue. Hidden corruption, worst nightmare of all. Everything seems to be ok, until you'll try to do full restore, which then completely fails and let's you down with data loss. - I think I'm forced to full restore all backups to verify integrity, which is of course very time and bandwidth consuming process. But without that, there's no integrity guarantee whatsoever. - Even worse, the status says restore was successful, when in reality it completely failed (no files whatsoever were restored at all). Ouch!
In one project there was interesting a timestamp issue. At seemingly random intervals, transactions came out of order through the processing. But when they sent logs and data, I started to analyze the situation. Let's see what the developer has written in the source code. And yep, there it was. First they took time for date (yyyy/dd/MM) from the system clock, and then they took in separate call time for time (HHmmss) and finally time from milliseconds (fff). Ok, now I know why this code sucks. How about getting the timestamp once, and using it to provide that information. If time is read in several separate calls for each of the segments, date, time or the milliseconds can roll-over while the code is running. This is perfect example of extremely lame code, which mostly seems to work, but is inherently botched and flawed. - Thank you for that, who ever wrote that bleeping code. Someone can claim that the code works, but fact is that it was never working correctly, when edge cases are included. It just seems to work temporarily, if you test it lightly. Oh yeah, I thought that for a while, that I could have just added let's say a two second delay between each processing step. It should make it less probable that the milliseconds wrap around causing ordering issues. That would have been a true engineering fix. Fix bad code, by making it kind of silly and slightly less likely to malfunction, but leaving it still absolutely flawed. Because it's still possible that the date rolls over between the date / time fetches. Duh! But sure, that's kind of trolling bad engineering. Or I could have checked after fetching the milliseconds, that time without milliseconds and date are still the same, and if that's not true then retry. Lol. Sometimes I just wonder how much time developers use, so that they can come up with so poor code. That could have been awesome. Giving the WTF effect for anyone looking the code later. - Hey, but at least it would actually work reliably.
OVH server KVM is so often broken, that it's getting seriously ridiculous. At least ILO works. But what if ILO isn't enough, then you're stuck. Potentially for days. This is just so awesome. - Thank you for that. - Everyone can choose freely what kind of words should be used to describe such situation in production.

2019-09-29