SQLite3, Junk, TSE&RTM, Python: functools.lru_cache, Linux 3.6, misc

Post date: Oct 7, 2012 4:36:30 PM

  • Studied SQLite3 internals in detail [1,2,3], tested and performed performance tests with normal and WAL mode. etc. Now I'm very happy with SQLite3, I would use it for any projects requiring database without high parallelism. Integrations and tasks I usually do aren't usually running in parallel at all, so SQLite3 is perfect solution for my needs. It's also very fast. I have been experimenting with multiple different databases with my use cases and I'll be writing more about these options in future.
  • Dovecot / Evolution - I really don't know what the real problem is, but I have twice tried to move folder using Evolution and results has been catastrophial. For some reason whole folder simply disappears with it's content. Very very bad stuff. I have had to restore my backups twice to get messages back for some pretty important folders. Not a good thing at all.
  • Evolution leaves leave incredible amount on junk files and folders in file system. I had to do extensive manual cleanup. (Over 30k files deleted from several years!)
  • Roundcube leaves junk in database, had to implement automated (crontab) cleanup.
  • Read more about Transactional Synchronization Extension (TSE) and Restricted Transactional Memory (RTM). This is very interesting topic. But if everything works like it should be, in best case developer tools can utilize these techniques allowing parallelization so that programmer using high level programming language like Java, Python, Ruby or Lua doesn't even need to know anything about the parallelization that happens transparently.
  • I have often written about caching. But Python is making it almost too easy! Simply add @functools.lru_cache(entries_to_cache_int) decorator to any of your slow running functions and N return values of function are automatically cached. I also added decorator option to my own pure Python version of the CAR cache implementation. This is especially great with database related functions. Based on my tests and production experience with databases, using internal cache instead of databases page cache can easily improve performance by multiple orders of magnitude. If you program with Python and aren't familiar with decorators or lru_cache fuction, just check it out, right now. Super easy and yet very powerful feature.
  • Linux 3.6 brings new stuff: I was earlier familiar with TCP Small Queues (TSQ), CoDel buffer management and TCP Fast Open (TFO). I also have studied multiple TCP congestion control algorithms, but I wasn't aware of Proportional Rate Reduction for TCP (PRR/TCP) and I need to read the full white paper. TCP Blind Reset Attack fix, Hybrid Sleep, SMBv2 support - SMB2 specifications: chattiness (roundtrips, small packets), pipelining (proceeding with new requests without waiting responce for previous requests) SMB2.1 also provies opportunistic locking mechanism.
  • I also transferred Google Spanner (Globally-Distributed Database) white paper to my Kindle.
  • I'm also planning to play a little with MySQL and PostgreSQL.