Test project: DNS-Hosts, Bottle.py, Apache2, PostgreSQL

Post date: Apr 5, 2014 5:47:35 AM

DNS-Hosts is a very simple project which generates hosts file with precomputed list of DNS names. List maintenance is crowdsourced. There's also project roadmap about the features I'm willing to add if there's some user base after all. It was quite clear that this is marginal requirement in the very beginning. So therefore I just made it as my first practice project with PostgreSQL and Bottle Micro Web Framework. Of course I were familiar with all related things. But reading theory and actually doing something is bit different.

Project contains 6 database tables and three indexes for most commonly used data. There's automatic abuse detection and blocking, statistic tables for daily usage statistics, daily usage reports by email, logging tables, Automated database cleanup monthly etc. So if I want to I could extend this platform to do what's required. I have the basic capability to build systems on demand. I've done similar practice projects on Google App Engine too. If I need solution which is able to scale, a lot. I got three templates, one css file, favicon.ico etc.

As you can see from my previous blog post, everything went really easily and smoothly until I had to actually deploy the script to the server. Where all the problem begun, because I was trying to run it as normal CGI script using ScriptAliasMatch, but in matter of fact, it required different configuration using RewriteEngine and DirectoryIndex options to get it completely working. I think that was the hardest part of whole project.

Project naturally uses HTTP etags, cache-control and browser caches, and server side internal caching of precomputed data correctly and efficiently.

I also tried installing it in WSGI mode, but because it's written in Python 3, I found out to be in such a niche quite quickly that it would have taken too much time. Because latest versions of mod_wsgi should work with Python 3, but it was quite hard to find any sufficient information how to configure it out. Probably getting that to work, would have taken more time than the whole project so far altogether.

I've written similar kind of applications earlier using Google App Engine, and the most delightful feature was speed of the local database, compared to Google App Engines Datastore. So maybe cloud isn't the perfect solution every time. With sharded counters, optimistic concurrency control, etc. Of course local database got it limitations, but as far as there's no problem with it, it's very nice and fast solution.

This was also good test, because I got some ideas, and if those ideas require execution, I must have existing working technology stack, so I don't run into sudden problems like described above.

I would also use probably use Bottle, if I needed to write extensive RESTful integration packet. But I'll blog more about that later. Anyway, utilizing same backed for any kind of integration stuff with authentication JSON / XML etc, is simply trivial. As you can see from my sites, I'm not a web designer. But I'm sure that part can be arranged with some interesting side project if required.