SQLite3, Unicode, httpbin, SLO/SLA, HTTP Pipelining, Android slow(er) WiFi

Post date: Sep 12, 2017 4:52:31 PM

Multi-Threaded SQLite3 without the OperationalErrors - Yet another totally awesome post by Charles Leifer. - Thanks! - Thoughts: I've had pretty similar thoughts. Yet as said, I often use bit different approach to keep the writes as short as possible. I read data, process it and then check if the record getting updated is still the same. So processing data might take 5 seconds, but at the end, updates happen in very short block. Of course this might lead to situation where value has changed and there's need for retry. Aka opportunistic locking. So this isn't perfect solution for very volatile data. With some of the most volatile data, which doesn't require absolute consistency I've got very different approach. I'll update in RAM counters, and then just flush updated values to disk, ever 15 seconds, or 15 minutes, whatever seems to be the reasonable interval. On graceful shutdown data is flushed to disk. This allows 'very high transaction rates'. Also individual processes maintain own counters, so there's no memory lock contention either. About Charles approach, it's very nicely wrapped and pretty obvious. Just use single thread for writes -> removes all locking issues. It can also cause bottle neck, but actually in this case the bottle neck already exists and it's the SQLite's single active writer limitation. About the gotcha, that's obvious. As many other performance improving techniques, it requires some planning and writing program logic in a way that it works efficiently with the underlying system. I'm waiting for the SqliteQueueDatabase. I've got a few projects which really could use it. - fopen, ha. Yeah. Sure. - I'm using several different databases for different projects. PostgreSQL, Raima SQL, Microsoft SQL Server. Yet for most of projects with small requirements I do choose SQLite3. It's good enough, and doesn't require additional setup, planning, discussion possibly expensive licenses etc.
Had some Unicode play. Usually Unicode is so transparent you don't even need to think about it. But now I had an issue. I had to work with Unicode Mahjong tiles. Now I know what's the difference between \u (16 bit) and \U (32 bit) and that chr() integer -> code point can be also used with Unicode nicely. It's all so simple and easy and too just a couple of minutes to figure out. But I haven't had any reason to do it earlier, because most of programs just handle Unicode without any issues. Also with Python 3 the ord() -> code point to integer works with Unicode seamlessly too. Awesome. Some people complain about Python 3 Unicode. But I think it's actually just the way it should be and that's awesome.
Here's a nice Unicode password generator. I thought I could make one, but now I don't need to.
Another interesting observation is that when OVH delivers Windows Servers, the Windows Firewall is disabled by default. Meaning that all 'basic windows network services' are accessible from whole Internet. Maybe this is the reason why they're blocking SMB/TCP445 on their network edge? Maybe, maybe not. I don't know. But that's one thing which has been also baffling me for quite a while. (Btw. They have fixed this while this post was in backlog). But they're still blocking SMB/TCP445.
About great tools, httpbin - HTTP Request & Response Service is totally essential service. Especially if doing something quickly and ad hoc. Otherwise you could setup your own test systems, but for quick fixes testing against something online is just way to get it done now.
Service Level Objective (SLO) key component of Service Level Agreement (SLA).
A very nice post about HTTP/1.1 and Pipelining requests. Using Japronto Python HTTP server and micro framework. Lovely optimizations, can't say anything else.
Just suspecting that Android 6.0 update made 5 GHz WiFi slower on one old device. Earlier WiFi Speeds were over 90 Mbps, but after upgrade those have dropped to around 30 in the very same environment. Got any scientific proof. Nope, got adequate sample size, nope. But this just happened at the same time when Android was upgraded. Why I'm posting this? To see if anyone else has had similar experiences.