Archive

Archive for the ‘Projects’ Category

Purgatory or Hell: Escape from eternal Alpha status

January 17, 2021 Leave a comment

Many of us will have laughed and scoffed at Google’s liberal use of the tag ‘Beta software’ these past years. Is the label ‘Beta’ nothing more than an excuse for any bugs and issues that may still exist in code, even when it has been running in what is essentially a production environment for years? Similarly, the label ‘Alpha’ when given to software would also seem to seek a kind of indemnity for any issues or lacking features: to dismiss any issue or complaint raised with the excuse that the software is still ‘in alpha’.

Obviously, any software project needs time to develop. Ideally it would have a clear course through the design and requirements phase, smooth sailing through Alpha phase as all the features are bolted onto the well-designed architecture, and finally the polishing of the software during the Beta and Release Candidate (RC) phases. Yet it’s all too easy to mess things up here, which usually ends up with a prolonged stay in the Alpha phase.

A common issue that leads to this is too little time spent in the initial design and requirements phase. Without a clear idea of what the application’s architecture should look like, the result is that during the Alpha phase both the features and architecture end up being designed on the spot. This is akin to building a house before the architectural plans are drawn up, but one wants to starts building anyway, because one has a rough idea of what a house looks like.

When I began work on the NymphCast project [1] a few years back, all I had was a vague idea of ‘streaming audio’, which slowly grew over time. With the demise of Google’s ChromeCast Audio product, it gave me a hint to look at what that product did, and what people looked at it. By that time NymphCast was little more than a concept and an idea in my head, and I’m somewhat ashamed to say that it took me far too long to work out solid requirements and a workable design and architecture.

Looking back, what NymphCast was at the beginning of 2020 – when it got a sudden surge of attention after an overly enthusiastic post from me on the topic – was essentially a prototype. A prototype is somewhat like an Alpha-level construction, but never meant to be turned into a product: it’s a way to gather information for the design and requirements phase, so that a better architecture and product can be developed. Realising this was essential for me take the appropriate steps with the NymphCast project.

With only a vague idea of one’s direction and goals while in the Alpha phase, one can be doomed to stay there for a long time, or even forever. After all, when is the Alpha phase ‘done’, when one doesn’t even have a clear definition of what ‘done’ actually means in that context? Clearly one needs to have a clear feature set, clear requirements, a clear schedule and definition of ‘done’ for all of those. Even for a hobby project like NymphCast, there is no fun in being stuck in Alpha Limbo for months or even years.

After my recent post [2] on the continuation of the NymphCast project after a brief burn-out spell, I have not yet gotten the project into a Beta stage. What I have done is frozen the feature set, and together with a friend I’m gradually going through the remaining list of Things That Do Not Work Properly Yet. Most of this is small stuff, though the small stuff is usually the kind of thing that will have big consequences on user friendliness and overall system stability. This is also the point where there are big rewards for getting issues fixed.

The refactored ring buffer class has had some issues fixed, and an issue with a Stop condition was recently resolved. The user experience on the player side has seen some bug fixes as well. This is what Alpha-level testing should be like: the hunting down of issues that impede a smooth use of the software, until everything seems in order.

The moral of this story then is that before one even writes a line of code, it’s imperative that one has a clear map of where to go and what to do, lest one becomes lost. The second moral is that it’s equally imperative to set limits. Be realistic about the features one can implement this time around. Sort the essential from the ‘nice to have’. If one does it right now, there is always a new development cycle after release into production where one gets to tear everything apart again and add new things.

Ultimately, the Alpha phase ends when it’s ‘good enough’. The Beta phase ends when the issue tracker begins to run dry. Release Candidates exist because life is full of unexpected surprises, especially when it concerns new software. Yet starting the Alpha phase before putting together a plan makes as much sense as walking into the living room at night without turning a light on because ‘you know where to walk’.

Fortunately, even after you have repeatedly bumped your shins against furniture and fallen over a chair, it’s still not too late to turn on a light and do the limping walk of shame 🙂

Maya

[1] https://github.com/MayaPosch/NymphCast
[2] https://mayaposch.wordpress.com/2020/12/27/nymphcast-on-getting-a-chromecast-killer-to-a-beta-release/

NymphCast: on getting a ‘ChromeCast killer’ to a Beta release

December 27, 2020 1 comment

It’s been a solid nine months since I first wrote about the NymphCast project [1] on my personal blog [2]. That particular blog post ended up igniting a lot of media attention [3], as it also began to dawn on me how much work would still be required to truly get it to a ‘release’ state. Amidst the stress from this, the 2020 pandemic and other factors, the project ended up slumbering for a few months as I tried to stave off burn-out on the project as a whole.

Sometimes such a break from a project is essential, to be able to step back instead of bashing one’s head against the same seemingly insurmountable problems over and over as they threaten to drown you into an ocean of despair, frustration and helplessness. You know, the usual reason why ‘grinding’, let alone a full-blown death march, is such a terrible thing in software development.

One thing I did do during that time off was to solve one particular issue that had made me rather sad during initial NymphCast development: that of auto-discovery of NymphCast servers on the local network. I had attempted to use DNS Service Discovery (DNS-SD, mDNS) for this, but ran into issue that there is no cross-platform solution for mDNS that Just Works ™. Before reading up on mDNS I had in my mind a setup where the application itself would announce its presence to the network, or to a central mDNS server on the system, as that made sense to me.

Instead I found myself dealing with a half-working solution that basically required Avahi on Linux, Bonjour on MacOS and something custom installed and configured on Windows, not to mention other desktop operating systems. On the client side things were even more miserable, with me finding only a single library for mDNS that was somewhat easy to integrate. Yet even then I had no luck making it work across different OSes, with the running server instances regularly not found, or requiring specific changes to the service name string to get a match.

The troubleshooting there was one factor that nearly made me burn out on the NymphCast project. Then, during that break I figured that I might as well write something myself to replace mDNS. After all, I just needed something that spit out a UDP Broadcast message, and something that listened for it and responded to it. This idea turned into NyanSD [4], which I wrote about before [5].

I have since integrated NyanSD into NymphCast on the server & client side, with as result that I have had no problems any more with service discovery, regardless of the platform.

Other aspects of NymphCast were less troublesome, but mostly just annoying, such as getting a mobile client for NymphCast. Originally I had planned to use a single codebase for the graphical NymphCast Player application, using Qt’s Android & iOS cross-platform functionality to target desktop and mobile platforms. Unfortunately this ran into the harsh reality of Qt’s limited Android support and spotty documentation [6]. This led me to work on a standard, native Android application written in Java for the GUI and using the JNI to use the same C++ client codebase. This way I only have to port the Qt-specific code on the Android side to the Java-Android equivalent.

Status at this point is that all features for the targeted v0.1 release have been implemented, with testing ongoing. An additional feature that got integrated at the last moment was the synchronisation of music and video playback between different NymphCast devices, for multi-room playback and similar. The project also saw the addition of a MediaServer [7], which allows clients to browse the media files shared by the server, and start playback of these files on any of the NymphCast servers (receivers) on the network. I also refactored the in-memory buffer to use a simple ringbuffer instead of the previous, more complicated buffer.

In order to get the v0.1 development branch out of Alpha and into Beta, a few more usage scenarios have to be tested, specifically the playback of large media files (100+ MB), both with a single NymphCast receiver and a group, and directly from a client as well as using a MediaServer instance. The synchronisation feature has seen some fixes recently already while testing it, but needs more testing to make it half-way usable.

A major issue I found with this synchronisation feature was the difficulty of determining local time on all the distinct devices. With the lack of a real-time clock (RTC) on Raspberry Pi SBCs in particular, I had to refactor the latency algorithm to only rely on the clock of the receiver that was used as the master receiver. Likely this issue may require more tweaking over the coming time to get synchronisation with better than 100 ms de-synchronisation.

I think that in the run-up to a v0.1 release, the Beta phase will be highly useful in figuring out the optimal end-user scenarios, both in terms of easy setup and configuration, as well as the day to day usage. This is the point where I pretty much have to rely on the community to get a solid idea of what are good ideas, and what patterns should be avoided.

That said, it’s somewhat exciting to see the project now finally progressing to a first-ever Beta release. Shouldn’t be more than a year or two before the first Release Candidate now, perhaps 🙂

Maya

[1] https://github.com/MayaPosch/NymphCast
[2] https://mayaposch.blogspot.com/2020/03/nymphcast-casual-attempt-at-open.html
[3] https://mayaposch.blogspot.com/2020/03/the-fickle-world-of-software-development.html
[4] https://github.com/MayaPosch/NyanSD
[5] https://mayaposch.wordpress.com/2020/07/26/easy-network-service-discovery-with-nyansd/
[6] https://bugreports.qt.io/browse/QTBUG-83372
[7] https://github.com/MayaPosch/NymphCast-MediaServer

Categories: nymphcast

Parsing command line arguments in C++

March 17, 2019 4 comments

One of the things which have frustrated me since I first started programming has been the difficulty in using command line arguments provided to one’s application. Everyone of us is aware of the standard formulation of the main function:

int main(int argc, char** argv);

Here argc is the number of arguments (separated by spaces), including the name of the application binary itself. Then argv is an array of C-style strings, each containing an argument. This leads to the most commonly used style of categorising the arguments:

app.exe -h --long argument_text

What annoyed me all these years is not having a built-in way to parse command line arguments in C++. Sure, there’s the getopt [2] way if one uses Linux or a similar OS. There are a range of argument parser libs or APIs in frameworks, such as gflags [3], Boost Program Options [4], POCO [5], Qt [6] and many others.

What these do not provide is a simple, zero-dependency way to add argument parsing to C++, while also being as uncomplicated as possible. This led me to put together a simple command line argument parsing class, which does exactly what I desire of such an API, without any complications.

Meet Sarge [1] and its integration test application:

#include "../src/sarge.h"

#include 


int main(int argc, char** argv) {
	Sarge sarge;
	
	sarge.setArgument("h", "help", "Get help.", false);
	sarge.setArgument("k", "kittens", "K is for kittens. Everyone needs kittens in their life.", true);
	sarge.setDescription("Sarge command line argument parsing testing app. For demonstration purposes and testing.");
	sarge.setUsage("sarge_test ");
	
	if (!sarge.parseArguments(argc, argv)) {
		std::cerr << "Couldn't parse arguments..." << std::endl;
		return 1;
	}
	
	std::cout << "Number of flags found: " << sarge.flagCount() << std::endl;
	
	if (sarge.exists("help")) {
		sarge.printHelp();
	}
	else {
		std::cout << "No help requested..." << std::endl;
	}
	
	std::string kittens;
	if (sarge.getFlag("kittens", kittens)) {
		std::cout << "Got kittens: " << kittens << std::endl;
	}
	
	return 0;
}

Here one can see most of the Sarge API, with the setting of arguments we are looking for, followed by the application description and usage, as it'll be printed if the user requests the help view or our code decides to print it in the case of missing options or similar.

The Sarge class implementation itself is very basic, using nothing but the STL features, specifically the vector, map, memory, iostream and string headers, in as of writing 136 lines of code.

When asked to parse the command line arguments, it will scan the argument list (argv) for known flags, flags which require a value, and unknown flags. It'll detect unknown flags and missing values, while allowing for short options (single-character) to be chained together.

I'll be using Sarge for my own projects from now on, making additions and tweaks as I see fit. Feel free to have a look and poke at the project as well, and let me know your thoughts.

Maya

[1] https://github.com/MayaPosch/Sarge
[2] https://en.wikipedia.org/wiki/Getopt
[3] https://github.com/gflags/gflags
[4] http://www.boost.org/doc/libs/1_64_0/doc/html/program_options.html
[5] https://pocoproject.org/docs/Poco.Util.OptionProcessor.html
[6] http://qt.io

Categories: C++, Projects Tags: , ,

Reviewing dual-layer PCBWay PCBs

March 11, 2019 Leave a comment

This review is an addendum to the first part in the Greentropia Base Board article series [1]. Here we have a look at the PCB ordering options, process and product delivered by PCBWay and conclude with impressions of the Greentropia Base realized with these PCBs.

Much to the delight of professional hardware developers and hobbyists alike, prices for dual layer FR4 PCBs have come down to a point where shipping from Asia has become the major cost factor. An online price comparison [2] brings up the usual suspects, with new and lesser-known PCB manufacturers added to the mix.

In this competitive environment, reputation is just as important as consistently high quality and great service. Thus PCBWay [3] reached out to us to talk about their PCB manufacturing process and products by providing free PCBs, which we accepted as an opportunity to fast-lane the Greentropia Base board [4], a primary building block of the ongoing Greentropia indoor farming project [5].

Ordering the PCBs

PCB specifications guide the design process and show up again when ordering the actual PCBs. They are at the beginning and the end of the board design process – hopefully without escalation to smaller drill sizes, trace widths and layer count.

The manufacturing capabilities [6] are obviously just bounds for the values selected in a definitive set of design rules, leaving room for a trade-off between design challenges and manufacturing cost. Sometimes relaxing the minimum trace width and spacing from 5/5mil (0.125 mm) to 6/6mil (0.15 mm) can make a noticeable difference in PCB cost. And then again, switching from 0.3 mm to 0.25 mm minimum drill size can make fan-out and routing in tight spaces happen, albeit at a certain price.

Logically we will need to look at the price tag of standard and extended manufacturing capabilities. The following picture displays pricing as of the writing of this article:pcbway_order_spec

For some options the pricing is very attractive. Most notably an array of attractive colours is available at no additional charge. With RoHS and REACH directives in place however it remains to be seen whether lead-free hot air surface levelling (HASL) will become the new standard at no added cost.

Luckily for our project we do not need to stray far from the well-trodden path and just opt for the lead-free finish on a blue 1.6mm PCB.

The ordering process is hassle-free and provides frequent status updates:

pcbway_order_progress

A month after our order, an online gerber viewer [7] was introduced to help designers quickly verify their gerber output before uploading them for the order. It must be noted however that this online feature is at an early stage and is expected to provide layer de-duplication, automatic and consistent color assignment and appropriate z-order and better rendering speed in the future.

pcbway_gerber_viewer

Gerbv [8] is a viable alternative which also provides last-minute editing capabilities (e.g. deleting a stray silkscreen element).

Visual inspection

PCBs were received within one week after ordering, packaged in a vacuum sealed bag and delivered in a cardboard box with foam sheets for shock protection. One extra PCB was also included in the shipment, which is nice to have.

The boards present with cleanly machined edges, well-aligned drill pattern and stop masks on both sides and without scratches or defects. The silkscreen has good coverage and high resolution. Adhesion of stop mask and silkscreen printing are excellent. The lead-free HASL finish is glossy and flat, and while we couldn’t put it to the test with this layout, the TSSOP footprint results suggest no issues with TSSOP, TQFP and BGA components down to 0.5mm pitch.

The board identifier is thankfully hidden underneath an SOIC component in the final product. Pads show the expected probe marks from e-test. without affecting the final reflow result. No probe damage to the pads is evident.

Realising the project

We conclude with some impressions of the assembled PCBs, which we will use in the following articles to build an automated watering system.

Here we see our signature paws with the 2 mm wide capacitor C15 next to them for scale. The pitch of the vertical header is 2.54 mm. Tenting of the vias is also consistent and smooth.

greentropia_base_mask_quality

Good mask alignment and print quality.

The next picture shows successful reflow of components of different size and thermal mass after a lead-free reflow cycle in a convection oven. As the PCBs were properly sealed and fresh, no issues with delamination occured.

greentropia_base_different_size_reflow_result

DC-DC section reflow result.

The reflow result with the lead-free HASL PCB and the stencil ordered along with it is also quite promising. No solder bridges were observed despite lack of mask webbing, which is likely due to our mask relief settings and minimum webbing width. Very thin webbing can be destroyed during HASL, so if the additional safety in the 0.15 to 0.2 mm between the pads is needed it’s worth checking back with the manufacturer.

greentropia_base_tssop_hasl_result

TSSOP reflow result.

While testing the 5V to 12V boost converter, it was found that it worked without issues. Initial testing of the ADC was also promising. As we continue to test the boards over the coming time we’ll find out whether there are zero issues, but so far it appears that everything is working as it should.

Maya

[1] https://mayaposch.wordpress.com/2019/03/06/keeping-plants-happy-with-the-greentropia-base-board-part-1/
[2] https://pcbshopper.com/
[3] https://www.pcbway.com/
[4] https://github.com/MayaPosch/Greentropia_Base
[5] http://www.nyantronics.com/greentropia.php
[6] https://www.pcbway.com/capabilities.html
[7] https://www.pcbway.com/project/OnlineGerberViewer.html
[8] http://gerbv.sourceforge.net/

Keeping plants happy with the Greentropia Base board – Part 1

March 6, 2019 Leave a comment

Last year I got started on an automatic plant watering project, with as goal a completely stand-alone, self-sufficient solution. It should be capable of not only monitoring the level of moisture in the soil, but also control a pump that would add water to the soil when needed.

Later iterations of this basic design added a scale to measure the level in the water reservoir, as well as a multi-colour LED to be used as a system indicator as well as for more decorative purposes. This design was initially developed further for my third book that got released [1][2][3] in February of this year. In chapter 5 of that book it is featured as an example project, using the BMaC [4] firmware for the ESP8266 microcontroller.

That’s where the project remained for a while, as even though a PCB design (the Greentropia [5] base board) had been created that would accommodate the project’s complete functionality on a single board, converting that into a physical product along with the associated effort and costs prevented me from just pushing the button on ordering the PCBs and components.

Thus the board remained just a digital render:

iop_plant_base_002

When I got suddenly contacted by a representative from PCBWay [6] with an offer to have free PCBs made in exchange for a review of the finished board, it made it all too easy to finally take the step to have the board produced for real.

After some last-minute, frantic validation of the design and board layout by yours truly and a good friend, the Gerber files were submitted to PCBWay. We used the Gerber viewer in KiCad to check the files prior to submitting them. Later I learned that PCBWay also offers an online Gerber viewer [7]. We did not use that one, but it’s important to use a Gerber viewer before one submits a design, to be sure that the resulting PCB will look like and function the way it should.

After a couple of days of PCB production and shipping from China to Germany, the boards arrived:

IMG_20190102_161941

Top side:

IMG_20190102_162412

Bottom side:

IMG_20190102_162432

All boards looked pretty good, with pretty sharp silkscreen features and the soldermask being aligned with the pads. We compared them with another Nyantronics PCB that we have been working on for a while now, that one being from JLCPCB. It is a good way to compare the blue soldermask that they use:

IMG_20190102_165819

Which colour you prefer is a personal choice, of course. Personally I like the more deep-blue colour of the JLCPCB board, but the PCBWay blue isn’t half bad either. The real concern is of course whether or not the PCB does what it’s supposed to, which is what we’d find out once we assembled the boards.

For this we used a professional reflow oven, courtesy of the local university:

IMG_0947IMG_0968

This resulted in the following boards, after a few through-hole components being added by hand:

IMG_20190203_041937IMG_20190203_041718IMG_20190203_042545
img_20190305_211945

Each of these boards has sockets for a NodeMCU board, which contains an ESP-12E or 12F module with the ESP8266 microcontroller. This provides the ability to control the pump output and SPI bus, as well as read out the HX711-based scale interface and soil sensor.

Microscope images of the finished boards were also made and can be found in this addendum article: https://mayaposch.wordpress.com/2019/03/11/reviewing-dual-layer-pcbway-pcbs/

In the next parts we will wrap up the remaining development of the hardware, and conclude with the development of the firmware for this board.

Maya

[1] https://www.amazon.com/Hands-Embedded-Programming-versatile-solutions-dp-1788629302/dp/1788629302/
[2] https://www.packtpub.com/application-development/hands-embedded-programming-c17
[3] https://www.amazon.de/Hands-Embedded-Programming-versatile-solutions/dp/1788629302/
[4] https://github.com/MayaPosch/BMaC
[5] http://nyantronics.com/greentropia.php
[6] http://www.pcbway.com/
[7] https://www.pcbway.com/project/OnlineGerberViewer

NymphRPC: my take on remote procedure call libraries

January 29, 2018 1 comment

Recently I open-sourced NymphRPC [1], which is a Remote Procedure Call (RPC) library I have been working on for the past months. In this article I would like to explain exactly why I felt it to be necessary to unleash yet another RPC library onto the world.

The past years I have had to deal quite a lot with a particular RPC library (Etch [2][3]) due to a commercial project. This is now a defunct project, but it spent some time languishing as an Apache project after Cisco open-sourced it in 2011 and got picked up by BMW as an internal protocol for their infotainment systems [4].

During the course of this aforementioned commercial project it quickly became clear to me that the Etch C library which I was using had lots of issues, including stability and general usability issues (like the calling of abort() without recovery option when any internal assert failed). As the project progressed, I found myself faced with the choice to either debug this existing library, or reimplement it.

At this point the C-based library was around 45,000 lines of code (LoC), with countless concurrency-related and other issues which made Valgrind spit out very long log files, and which proved to be virtually impossible to diagnose and fix. Many attempts resulted in the runtime becoming more unstable in other places.

Thus it was that I made the decision to reimplement the Etch protocol from scratch in C++. Even though there was already an official C++ runtime, it was still in Beta and it too suffered from stability issues. After dismissing it as an option, this led me to the next problem: the undocumented Etch protocol. Beyond a high-level overview of runtime concepts, there was virtually no documentation for Etch or its protocol.

Reimplementation

Fast-forward a few months and I had reverse-engineered the Etch protocol using countless Wireshark traces and implemented the protocol in a light-weight C++-based runtime of around 2,000 LoC. Foregoing the ‘official’ runtime architecture, I had elected to model a basic message serialisation/deserialisation flow architecture instead. Another big change was the foregoing of any domain specific language (DSL) as with Etch to define the available methods.

The latter choice was primarily to avoid the complexity that comes with having a DSL and compiler architecture which has to generate functioning code that then has to be compiled into the project in question. In the case of a medium-sized Etch-based project, this auto-generated code ended up adding another 15,000 LoC to the project. With my runtime functions were defined in code and added to the runtime on start-up.

In the end this new runtime I wrote performed much better (faster, lower RAM usage) than the original runtime, but it left me wondering whether there was a better RPC library out there. Projects I looked at included Apache Thrift [5] and Google Protocol Buffers [6].

Both sadly also are quite similar to Etch in that they follow the same DSL (ISL) and auto-generated code for clients/servers path. Using them is still fairly involved and cumbersome. Probably rpclib [7] comes closest, but it’s still very new and has made a lot of design choices which do not appeal to me, including the lack of any type of parameter validation for methods being called.

NymphRPC

Design choices I made in NymphRPC include such things as an extremely compact binary protocol (about 2x more compact than the Etch protocol) while allowing for a wide range of types. I also added dynamic callbacks (settable by the client). To save one the trouble of defining each RPC method in both the client and server, instead the client downloads the server’s API upon connecting to it.

At this point NymphRPC is being used for my file revision control project, FLR [8], as the communication fabric between clients and server.

Performance and future

Even though the network code in NymphRPC is pretty robust and essentially the same as what currently runs on thousands of customer systems around the world – as a result of this project that originally inspired the development of NymphRPC – it is still very much under development.

The primary focus during the development of NymphRPC has been on features and stability. The next steps will be to expand upon those features, especially more robust validation and ease of exception handling, and to optimise the existing code.

The coming time I’ll be benchmarking [9] NymphRPC to see how it compares other libraries and optimise any bottlenecks that show up. Until then I welcome anyone who wishes it to play around with NymphRPC (and FLR) and provide feedback 🙂

Maya

[1] https://github.com/MayaPosch/NymphRPC
[2] https://etch.apache.org/
[3] https://en.wikipedia.org/wiki/Etch_%28protocol%29
[4] http://www.bmw-carit.de/open-source/etch.php
[5] https://en.wikipedia.org/wiki/Apache_Thrift
[6] https://en.wikipedia.org/wiki/Protocol_Buffers
[7] https://github.com/rpclib/rpclib
[8] https://github.com/MayaPosch/FLR
[9] http://szelei.me/rpc-benchmark-part1/

Categories: C++, Networking, NymphRPC, Protocols, RPC Tags: , ,

Cerflet: like Servlets, but with more C++

May 19, 2016 2 comments

A few months ago I wrote about the research I had been doing on multiple ways to write server-based web applications, using Java Servlets, FastCGI/C++ and Qt/C++. While this showed that C++-based applications tend to be faster than Java-based ones, it only looked at single-threaded, sequential requests.

While looking at ways to get proper concurrent performance out of a Servlet-like C++ implementation I decided to look again at the POCO C++ Libraries [1] and found that its HTTP server implementation implements proper thread-pool-based working threads for excellent scaling across many concurrent requests.

After spending a few hours putting a basic wrapper library together, I wrote the following ‘Hello World’ example code to demonstrate a basic HTTP Cerflet:

#include <httpcerflet.h>

#include <iostream>
#include <string>

using namespace std;


class HelloRequestHandler :	public HTTPRequestHandler {
public:
	void handleRequest(HTTPServerRequest& request, HTTPServerResponse& response) {
		Application& app = Application::instance();
        app.logger().information("Request from " + request.clientAddress().toString());
		
		response.setChunkedTransferEncoding(false);
        response.setContentType("text/html");
		
		std::ostream& ostr = response.send();
		ostr << "<!DOCTYPE html><html><head><title>Hello World</title></head>";
		ostr << "<body>

Hello World!

</body></html>";
	}	
};


int main(int argc, char** argv) {
	// 0. Initialise: create Cerflet instance and set routing.
	HttpCerflet cf;
	RoutingMap map;
	map["/"] = &createInstance<HelloRequestHandler>;
	cf.routingMap(map);
	
	// 1. Start the server	
	return cf.run(argc, argv);
}

In the main() function we create a new HttpCerflet instance and a new RoutingMap. The latter contains the routes we wish to map to a handler, which in this case is the HelloRequestHandler. For the handler instance we create a reference to the template method createInstance<>(), with the name of our custom handler as the template argument.

What this mapping does is that when a new request is mapped against one of the keys of the RoutingMap, it instantiates a copy of the specified handler, pushing it onto a waiting worker thread.

The handler class itself derives from the HTTPRequestHandler class, which is a standard POCO Net class, reimplementing its handleRequest() method. This shows that Cerflet is more of a complement to POCO instead of abstracting it away. The main goal of Cerflet is to hide some of the complexities and boilerplate of POCO’s HTTP server, allowing one to focus on writing the actual business logic.

Benchmarks:

As for performance, an ApacheBench benchmark was run with a concurrency of 5, for a total of 100,000 requests.

1. Java Servlet

Server Software:        Apache-Coyote/1.1
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /examples/servlets/servlet/HelloWorldExample
Document Length:        400 bytes

Concurrency Level:      5
Time taken for tests:   7.697 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      56200000 bytes
HTML transferred:       40000000 bytes
Requests per second:    12992.07 [#/sec] (mean)
Time per request:       0.385 [ms] (mean)
Time per request:       0.077 [ms] (mean, across all concurrent requests)
Transfer rate:          7130.42 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:     0    0   0.5      0      14
Waiting:        0    0   0.4      0      14
Total:          0    0   0.5      0      14

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%     14 (longest request)

2. Cerflet

Server Software:
Server Hostname:        127.0.0.1
Server Port:            9980

Document Path:          /
Document Length:        99 bytes

Concurrency Level:      5
Time taken for tests:   7.220 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      19900000 bytes
HTML transferred:       9900000 bytes
Requests per second:    13850.42 [#/sec] (mean)
Time per request:       0.361 [ms] (mean)
Time per request:       0.072 [ms] (mean, across all concurrent requests)
Transfer rate:          2691.63 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:     0    0   0.5      0      10
Waiting:        0    0   0.4      0      10
Total:          0    0   0.5      0      10

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%     10 (longest request)

Notes:

In this benchmark, Cerflet is about 7% faster than the equivalent Tomcat-based Hello World example. Cerflet hereby also logs requests to console, slowing it down somewhat, while Tomcat does not. Cerflet’s Hello World example was compiled using -Og optimisation setting in 32-bit GCC 5.3 (on Windows, MSYS2). The POCO libraries version 1.6 were used, as obtained via MSYS2’s Pacman package manager.

For Tomcat the binary distribution for 8.0.30 as obtained via the official Apache site was used, with the server manually started using the provided startup.bat script. Both servers were run on a Windows 7 Ultimate x64 platform (Intel i7 6700K, 32 GB DDR4) with ApacheBench using the loopback device.

Discussion:

Without compensating for all differences between the two examples used and other potential differences, it is fair to say at this point that both Servlets and Cerflets are roughly equivalent in terms of performance for a simple Hello World example. Likely Cerflets are slightly faster (5-10%) with more gains to be obtained via compiler optimisations (-O3).

The type of operations which would be performed further in the business logic likely will have the most influence on the overall performance between these two platforms. Cerflets do however show that C++-based server-side web applications are more than just a viable option, backed by a mature programming language (C++) and cross-platform libraries (POCO).

Cerflets as they exist today are reminiscent of Spring Boot Java applications, which also feature a built-in HTTP server, thus not relying on a Servlet container (e.g. Tomcat). The advantage of Cerflets is however that they only depend on the POCO libraries (if not linked fully statically), and are not dependent on a central runtime (JVM). This significantly eases deployment.

The Cerflet project’s Github page [2] can be accessed to try the here used example oneself, or to use the HTTP Cerflet implementation in a new project. Both feedback and contributions are welcome.

Maya

[1] http://pocoproject.org/
[2] https://github.com/MayaPosch/Cerflet

Categories: C++, Cerflet, HTTP Tags: , , , ,

New Project: NGS CPU Architecture

August 30, 2014 Leave a comment

For those looking at the scarcity of posts on this blog and wondering what in the world happened to me, I can offer the following explanation: personal (health) issues, as well as the embarking on writing this one book for Packt Publishing on AndEngine game development have taken up most of my time recently. Unfortunately I haven’t had much opportunity to write on this blog for that reason. Fortunately, however, I have not been sitting completely idle and have begun a new project which at least some may find interesting.

The project is a custom CPU architecture I have been wanting to develop for a while now. ‘Great’, I can hear some of you think, ‘Another CPU architecture, why would we need another one?!’ The short version is that this is a pretty experimental architecture, exploring features and designs not commonly used in any mainstream CPU architectures. Consider it a bit of a research project, one aimed at developing a CPU architecture which may be useful for HPC (high-performance computing) as well as general-purpose computing.

The project’s name is ‘Nyanko Grid-scaling System’, or NGS for short. Currently I’m working on the first prototype – a simplified 16-bit version of NGS – featuring only a single ALU. This prototype is referred to as ‘NGS-16’. Even then it has many of the essential features which I think make this into such an interesting project, including:

– unclocked design: all components work without a central clock or pipeline directing them.
– task scheduler: integrating the functionality of the software-based scheduler of an OS.
– virtual memory management: virtual memory management done in hardware.
– driver management: drivers for hardware devices are either in hardware, or directly communicate with the CPU.

Essentially this means that there’s no software-based operating system (OS) as such. A shell will be required to do the actual interfacing with human beings and to instruct the NGS task scheduler to launch new processes, but no OS in the traditional sense. While this also means that existing operating systems cannot be ported to the NGS architecture in any realistic fashion, it does not mean that applications can not be compiled for it. After porting a C/C++ toolchain (GCC or LLVM) to NGS, the average C/C++-based application would only be some library-wrangling and recompile away from functioning.

Moving back to the present, I’m writing NGS-16 in VHDL, with the Lattice MachX02-7000 [1] as the target FPGA. The basic structure has been laid out (components, top entities, signals), with just the architecture implementations and debugging/simulation left to finish. While this prototype is taking the usual short-cuts (leaving out unneeded components, etc.) to ease development, it should nevertheless be a useful representation of what the NGS architecture can do.

The FPGA board I’ll be using is actually produced by a friend, who called it the FleaFPGA following the name of his company: Fleasystems [2]. As you can see on the FleaFPGA page [3], it offers quite a reasonable amount of I/O, including VGA, USB (host), PS/2, audio and an I/O header. The idea is to use as much of this hardware as possible with the initial range of prototypes. I also have another FPGA board (Digilent Nexys 2, Spartan 3E-based), which offers similar specifications (LEs and I/O). Depending on how things work out I may also run NGS-16 on that board. Ultimately I may want to build my own FPGA board aimed specifically at running NGS.

Over the coming months I’ll be blogging about my progress with this NGS-16 prototype and beyond, so stay tuned 🙂

Maya

[1] http://www.latticesemi.com/en/Products/FPGAandCPLD/MachXO2.aspx
[2] http://fleasystems.com/
[2] http://www.fleasystems.com/fleaFPGA.html

Categories: NGS, VHDL Tags: , , , , ,

Introducing Nt With the Qt-based NNetworkSocket Class

August 17, 2013 Leave a comment

While recently working on a project involving C++, Qt and networking using TCP sockets I came across a nasty surprise in the form of blocking socket functions not working on Windows with QTcpSocket [1]. Not working as in fundamentally broken and a bug report [1] dating back to early 2012, when Qt 4.8 was new. After a quick look through the relevant Qt source code (native socket engine and kin) in an attempt to determine whether it was something I could reasonably fix myself, I determined that this was not a realistic option due to the amount of work involved.

Instead I found myself drawn to the option of implementing a QObject-based class wrapping around the native sockets on Windows (Winsock2) and others (POSIX/Berkeley). Having used these native sockets before, I could only think of how easy it’d be to write such a wrapper class for my needs. These needs involved TCP sockets, blocking functions and client-side functionality, all of which take only a little bit of effort to implement. Any further features could be implemented on an as-needed basis. The result of this is the NNetworkSocket class, which I put on Github today [2] in a slightly expanded version.

As I suspect that it won’t be the first add-on/drop-in/something else class I’ll be writing to complement the Qt framework, I decided to come up with the so very creative name of ‘Nt’ for the project. Much like ‘Qt’, its pronunciation is obvious and silly: ‘Qt’ as ‘cute’ and ‘Nt’ as ‘neat’. Feel free to have a look at the code I put online including the sample application (samples/NNetworkSocket_sample). Documentation will follow at some point once the class has matured some more.

As usual, feel free to provide feedback, patches and donations 🙂

Maya

[1] https://bugreports.qt-project.org/browse/QTBUG-24451
[2] https://github.com/MayaPosch/Nt

Implementing A Cookiejar for QtWebKit; QNetworkCookieJar Analysis

February 24, 2012 2 comments

As some of you may know already, I am working on the WildFox browser project which while it initially was going to fork the Mozilla code is now building a browser on top of QtWebKit. See the previous blog post for details on this decision. The WildFox project page and source can be found at www.mayaposch.com/wildfox.php.

One of the features I recently implemented was an advanced cookiejar for storing HTTP cookies. Why was this necessary, you may ask? QtWebKit does implement a cookiejar in QNetworkCookieJar, but even aside from the inability to save any of the cookies to disk, a quick look at its source code shows the following issues: a limit of 50 cookies, which is less than the 300 required by the current standard for HTTP cookies (RFC 2617) [1]. It also uses a basic QList to store the cookies, which requires it to search in linear time through every cookie to find ones for a specific URL and duplicate cookies when storing them.

In other words, the default implementation is unsuitable for any web browser. One thing it does do right, however, is the way it verifies domains. Due to the design of internet Top Level Domains (TLDs) it is impossible to algorithmically determine whether an internet URL is valid, or specifies a proper TLD.

The need to verify the domain is made clear when one imagines someone setting a cookie for the domain .com, which would then be a cookie valid for every website ending with the TLD .com. Obviously this can’t be allowed and the obvious approach would be to disallow single dot domains (.com, .org, .net). This doesn’t work for domains like .co.uk, however. Disallowing two dot domains would cause issues with the former, single dot type. Further there are more variations on this, such as URLs in the US where the public suffix can entail city.state.us style domains. Clearly the only way to do this verification is to use a look-up table. This can be found in Mozilla’s public suffix list [2].

What we need for a better QtWebKit cookiejar thus entails the following:

  • the ability to store cookies to disk.
  • storing at least 300 cookies.
  • quick look-ups of cookies based on their domain.

For this we recycle the existing functionality in Qt required to do the public suffix verification. The relevant files in Qt 4.8.0 are:

  • src/corelib/io/qtldurl.cpp
  • src/qurltlds_p.h

The former contains some basic routines to obtain the public suffix which we will expand upon and the latter contains the Public Suffix list processed in a more accessible format. The latter we’ll use almost as-is, with just the Qt namespace sections removed. The former has a major omission we’ll add. The functions we’ll keep from qtldurl.cpp are in renamed form:

  • containsTLDEntry(const QString &entry)
  • isEffectiveTLD(const QString &domain)
  • topLevelDomain(const QString &domain)

We add the following function:

QString getPublicDomain(const QString &domain) {
    QStringList sections = domain.toLower().split(QLatin1Char('.'), QString::SkipEmptyParts);
    if (sections.isEmpty())
        return QString();

    QString tld = "";
    for (int i = sections.count() - 1; i >= 0; --i) {
        tld.prepend(QLatin1Char('.') + sections.at(i));
        if (!isEffectiveTLD(tld.right(tld.size() - 1))) {
             return tld;
        }
    }

    return tld;
}

This allows us to obtain the public suffix plus the initial non-public (TLD) domain. For example, “http://www.slashdot.org&#8221; would be reduced to “.slashdot.org”. It is different from topLevelDomain() in that the latter would return just the public suffix, e.g. “.org” in the previous example, which is not desirable for our use.

With the domain verification taken care of, we move on to the next stage, which involves the data structure and storage method. To store cookies on disk we elect to use an SQLite database, as this is both efficient in terms of storage density, but also prevents disk fragmentation and allows for SQL-based look-ups instead of filesystem-based ones, as used to be common with older browsers. QtSQL comes with an SQLite driver. Do be sure to use the current version of Qt (4.8) as recently SQLite introduced journaling and the Qt 4.7.x libraries still use the old SQLite client.

For the in-memory data structure we use a QMultiMap. The rationale behind this is the key-based look-up based on the cookie domain. By taking the URL we’re seeking matching cookies for and obtaining its top domain (“.slashdot.org”) we can find any cookie in our data structure using this top domain as the key. This means we can search a large number of cookies in logarithmic (O(log N)) time for a match on the domain, a major improvement on the linear (O(N)) search of the default QList.

The link between the in-memory and on-disk storage is accomplished by the following rules:

  • All new cookies and cookie updates are stored in both in-memory and on-disk, except for session cookies, which are stored only in-memory.
  • Stored cookies are read into memory per-domain and on-demand.

In addition to this I have implemented a cookie manager dialogue which allows one to look through and manage (delete) stored cookies. Expired cookies are automatically deleted the first time they are fetched from the database or before they’re stored. Blocking 3rd-party cookies is also very easy, with a comparison between the top domain and the cookie’s intended domain:

QString baseDomain = getPublicDomain(url.host());
if (skip3rd && (baseDomain != getPublicDomain(cookie.domain()))) {
     continue;
}

With this we got a relatively efficient cookie storage and retrieval mechanism with the ability to manage the set cookies. It can store an unlimited number of cookies and should remain efficient with look-ups even with over 10,000 cookies thanks to the logarithmic search of the QMultiMap.

Essential features still missing in the WildFox browser at this point are bookmarks and sessions. The next article on WildFox should be about the Chrome extension support I’m currently implementing, with as direct result XMarks bookmark synchronization support as well as the bookmarks feature. Stay tuned.

Maya

[1] http://www.ietf.org/rfc/rfc2617.txt
[2] http://publicsuffix.org/

Categories: HTTP, programming, Projects, Qt, WildFox Tags: , , ,