More OpenBazaar networking thoughts

Post date: Jun 28, 2015 5:02:23 PM

Pretty random stuff about OpenBazaar networking...

  • Some protocol considerations OpenBazaar UDP, RUDP vs SCTP.
  • Had a chat with friends about if anyone has really used SCTP RFC3286 in production.
  • My comments to it: I guess answer is nope. I guess everybody knows it, but nobody has used it ever. Because there's no native support for it. Of course it's possible to implement SCTP over UDP but that's one of the complex things I mentioned earlier. But this is just where programming, hobby and merit comes into play. From commercial point implementing SCTP would sound crazy.
  • But from point of programming hobby and merit of mentioning of writing SCTP implementation from scratch could be worth of the fun and of course the merit. I did it, have you done it?
  • What's the benefit? It would have more advanced flow control. Message based communication is nice, and also optional ordering is really nice. TCPs blocking head of queue is known common problem and HTTP/2 (h2 / h2c) protocols won't fix that. Because those currently run over TCP.
  • Someone suggested using a timestamp for packet ordering. I think it's a horrible idea, because high speed net and bad clock resolution on many platforms. (Off-topic, I know, but that's what came first into my mind). 4 way handshake requires lot of round trips -> high handshake latency -> really bad for DHT operations related operations. I assume rUDP is lighter on this side. I don't know exact specific implementation details of our rUDP implementation. What's the latency after it requires a ping + some simple authentication / handshake after some inactivity making it 2 to 4 way handshake before actual data gets transmitted.
  • HTTP/2 implements very similar stuff inside TCP connection compared to SCTP. It also has steams and congestion windows per connection and per stream inside a TCP stream. This is just a random thought while reading the SCTP packet format stuff in detail.
  • See 'Data chunk' section. Ordering can be implemented using flags if and when required. There's a stream sequence number. Which contains transmission sequence number (TSN) which can be used for fragmentation reassembly. There's also stream sequence number (SSN). These are multi-layered protocols, which just adds tons of complexity if we can't name specific benefits for this use case.
  • Sometimes adding tons of complexity can improve performance, but in general I just love KISS. (Keep It Simple Stupid). Because complex stuff is, well just complex, and you'll end up hunting bugs. In that sense rUDP is really nice, because it's naively simple and light, yet not technically optimal (throughput in case of packet loss and lack of sliding window and lack of SACK).
  • Would I use SCTP instead of rUDP? Especially now when we have already working rUDP implementation which has required considerable effort? Well, only if there's really well working pure Python implementation available. It would then run on all platforms and would be hopefully really easy to use and maintain, compared to having own rUDP implementation. And if situation is really such, why time was wasted implementing rUDP in the very first place?
  • Are there some serious problems with rUDP which I'm not aware about? If not, I think we're focusing on wrong things, at least from the point of view where OpenBazaar should be made mainstream and usable for normal users. Usability and reliable functionality is much more important than some minor protocol details. I know engineers, and I like perfection too, but in most of cases, it's not actually worth of it.
  • What about support for dual stack, aka using IPv4 and IPv6 in parallel? I know, this is yet another nerdy view. IPv6 isn't that common (yet). But just because we're talking these topics I'll bring it up. This isn't important, but more important than the SCTP question. So peers should be able to advertise and use two addresses which one can be behind NAT and another without. Like the case with my 4G LTE data. IPv6 is totally open but IPv4 uses CG-NAT.
  • Related reading for networking nerds? HTTP/2, h2 / h2c RFC7540
  • Is there a need to shutdown connection? Couldn't it just 'timeout' so after a certain time some kind of new hand shake is required. This kind of 'dynamic' closing of connection allows you to keep many connections "open" if that's the situation as well as close those dynamically if there's load which requires getting rid of connections.
  • I've deeply hated way how Apache handles keep-alives, because it handles those as static. I would personally set connection limit like 64 connections. If there's room to keep connections alive, connections could remain alive for hours, and if there's load spike and keep alive can't be afforded then connections should be closed after a very short wait for next request. Or rotated after N requests anyway. Of course the exact use case affects things, but this is how HTTP/2 RFC recommends doing.
  • Final note: These notes were written before version 0.5 was released. OpenBazaar Project is currently making some changes to networking stack.