Small file handing & file queues (Linux vs Windows)

Post date: Feb 7, 2015 7:24:23 AM

Got bit annoyed due to poor disk I/O performance with one of my projects when using Windows. I just synced one of my git repository with about 2000 files and it seemed to take along time.

I found out that it's really true. Even server with SSD gives 3x worse disk I/O performance that el cheapo VPS. Even bigger the difference is when we compare results to Linux server with SSD. Test generates files, gets list of those files and then delete files by name, just to test out file queue (in directory) processing speeds, using Python 3.4 64bit and standard library.

10k files 4k / file chart, lower is better.

I often develop stuff on Linux systems, but production runs Windows. Production systems run on Windows servers, but when I'm bored I often develop stuff at home and I'm not going to use Windows at home. Funny thing is that really slow el cheapo USB flash stick gives you better actually performance on Linux than server SSD gives when using Windows. Yeah, that's life.You might be wondering the results, but the point is that the test app does NOT force fsync with disk as most of programs won't do.

This leads to situation where same program is working lot faster on Linux platform than on Windows. Of course this might be a problem if you assume that all data would be persisted on disk without using fsync. I've written about this topic earlier.

Raw data and extended tests with larger files, times are in seconds required to run the loop with code.

Windows Server (SSD) NTFS:

Create 3.185

Delete 0.955

Linux Server (SSD) ext4:

Create 0.276

Delete 0.051

Linux Server (HDD using Writeback) ext4:

Create 0.270

Delete 0.054

Linux Server (Slow (4MB/s write) USB Flash) NTFS:

Create 1.735

Delete 0.393

Linux VPS 1 ext4, el cheapo, minimal resources:

Create 1.061

Delete 0.324

Linux VPS 2 ext4, el cheapo, minimal resources:

Create 0.916

Delete 0.279

Same tests with 100x 16M / file:

Windows server (SSD) NTFS:

Create 10.359

Delete 0.124

Linux Server (SSD) ext4:

Create 2.416

Delete 0.379

Additional edit 2015-02-11:

Same test with NTFS using same slow USB2 stick with Windows 8.1. This is a good apples to apples comparison, because same program was run using Windows and Linux using exactly the same file system and storage medium:

Create 133.55

Delete 51.88

This really should blow your socks off. No, it's not 70% slower, it's over 70 times slower. then the Linux run. Deleting files is over 130x slower.