Archive

Archive for November, 2011

Surviving The Mozilla Build System, A Brief Guide

November 26, 2011 Leave a comment

Last year I did a couple of interviews for The Register and other sites regarding my WildFox project which in essence had the goal to add h.264 video support to Firefox using the GStreamer backend or FFmpeg. Due to circumstances I didn’t manage to do significant work on this project until a few months ago when I finally began the real modifications to the Firefox source.

As this article isn’t about the quality of the Mozilla source code, or the lack thereof, I won’t dwell on it too long. Suffice it to say that I found a lot of instances of NIH (‘Not Invented Here’) syndrome including the networking, smart pointer and threading sections. As my goal was to add Libav (recent fork of FFmpeg) support to the media backend of Firefox I became intimately familiar with these APIs as I discovered just how much of the code would be ripped out without causing adverse effects, and that the smart pointers do not work with anything but NSISupports-derived types.

Anyway, the build system… at first glance the Mozilla build system seems to use Makefiles, that is until you notice the .in extension indicating that they’re autoconfig templates. Or autobreak as lovingly called by a large section of the internet. After much trial and error I discovered that after putting my new Libav decoder & reader into /content/media/libav and the Libav includes into /media/libav of the source tree, creating a single Makefile with EXPORTS and an individual Makefile for each library of Libav (libavformat, libavcodec, etc.), I still had to edit a host of files to make it all work:

/layout/build
    Makefile.in

/toolkit
toolkit-makefiles.sh
    toolkit-tiers.mk

/content/media
    Makefile.in

After following the hints in the Mozilla documentation [1] I first discovered the files in /toolkit, and things finally began to work. Until I hit a few snags, like having to add a compiler flag to CXXFLAGS but there being no way to specify this in the Makefiles which didn’t get ignored for some reason. Libav is a C99 project and requires -D__STDC_CONSTANT_MACROS to be added to CXXFLAGS [2] to make it play nice with a C++ project. In the end I put this flag directly into the CXXFLAGS definition in the top of configure.in in the root folder. Ugly but it works.

At this point everything builds, the only thing I’m still stuck on is how to add the Libav’s LIB files to the linker flags. As usual the methods I have found do not work and even adding it to configure.in didn’t seem to do the trick.

To be quite honest I’m ready to give up on improving the Mozilla source. The changes required to bring it up to speed with proper project standards are just too daunting and severe to be handled just by me. As a fun comparison I started the WildFox-Mimic project a few days ago to investigate what it’d take to create a browser which looks and feels like Firefox, but uses Qt and the QtWebKit engine. The result is a modern, up to date browser with an HTML 5 video/audio backend which uses Qt’s Phonon which wraps around the OS’s media framework, be it DirectShow, GStreamer or something else. In other words it’s perfect. QtWebKit can also use the same NPAPI plugins Firefox uses, so Flash support is available out of the box. The Persona themes and JetPack add-ons can also be supported.

The result with WildFox-Mimic would be a browser with a codebase a fraction the size of the Firefox one, with most of the development and maintenance performed by the Qt and QtWebKit maintainers. This is the direction I think I’ll be heading towards.

Maya

[1] https://developer.mozilla.org/en/Adding_Files_to_the_Build
[2] http://libav.org/faq.html#I_0027m-using-libavutil-from-within-my-C_002b_002b-application-but-the-compiler-complains-about-_0027UINT64_005fC_0027-was-not-declared-in-this-scope

Advertisements
Categories: Projects, Qt, WildFox

Sharing Data Even More Universally; UDS Open Sourced

November 22, 2011 Leave a comment

A while ago (September 27th this year, to be precise) I announced[1] the Universal Data Share (UDS) project: a small utility using which people can share files with others without the need for a server or complicated software installs.

Since that time I have been testing the application with some friends and added a lot of bug fixes as well as some features, including:

  • Chunked downloading, to allow larger files to be transferred without the application dying on itself.
  • Downloads tab, with progress indicators and download folder.
  • Filesizes of the remote files reported in the client.
  • Much improved protocol.

The project page has been online for a while now [2]. You can find the most up to date binaries, GitHub link and protocol details there. Yes, that’s a GitHub link, as in the source being freely available. While I haven’t attached a formal license to it, I’d much appreciate it if people didn’t use significant parts of the source without attribution and send me any bug fixes or feature additions they may have produced.

Also important to note that the software is now essentially Beta-level quality, as in that it is feature-complete and relatively stable. Large-scale testing still has to be performed, naturally, and that’s where you people come into play 🙂

Maya

[1] https://mayaposch.wordpress.com/2011/09/27/qt-and-custom-networking-protocols-or-the-holy-grail-of-the-internet/
[2] http://www.mayaposch.com/uds.php

Categories: Projects, UDS Tags:

How To Really, Truly Use QThreads; The Full Explanation

November 1, 2011 92 comments

Threads in an operating system are a very simple thing. Write a function, maybe bundle it with some data and push it onto a newly created thread. Use a mutex or other method to safely communicate with the thread if necessary. Whether it are Win32, POSIX or other threads, they all basically work the same and are quite fool-proof. I’d venture to say that they’re at least a lot easier to use and handle than sockets 🙂

Those who have discovered the joys of the Qt framework may assume that threads in Qt (QThread) are just like this, but they would be both wrong and right. Wrong because years of wrong documentation from Trolltech/Nokia on QThread has caused countless people to use QThreads in a convoluted and highly inappropriate manner. Right because QThreads are in fact quite easy to use, as long as you ignore the incorrect official Qt documentation on QThread [1] and the myriad of wrongful methods being employed.

The main thing to keep in mind when using a QThread is that it’s not a thread. It’s a wrapper around a thread object. This wrapper provides the signals, slots and methods to easily use the thread object within a Qt project. This should immediately show why the recommended way of using QThreads in the documentation, namely to sub-class it and implement your own run() function, is very wrong. A QThread should be used much like a regular thread instance: prepare an object (QObject) class with all your desired functionality in it. Then create a new QThread instance, push the QObject onto it using moveToThread(QThread*) of the QObject instance and call start() on the QThread instance. That’s all. You set up the proper signal/slot connections to make it quit properly and such, and that’s all.

For a basic example, check this class declaration for the Worker class:

class Worker : public QObject {
    Q_OBJECT

public:
    Worker();
    ~Worker();

public slots:
    void process();

signals:
    void finished();
    void error(QString err);

private:
    // add your variables here
};

We add at least one public slot which will be used to trigger the instance and make it start processing data once the thread has started. Now, let’s see what the implementation for this basic class looks like.

// --- CONSTRUCTOR ---
Worker::Worker() {
    // you could copy data from constructor arguments to internal variables here.
}

// --- DECONSTRUCTOR ---
Worker::~Worker() {
    // free resources
}

// --- PROCESS ---
// Start processing data.
void Worker::process() {
    // allocate resources using new here
    qDebug("Hello World!");
    emit finished();
}

While this Worker class doesn’t do anything special, it nevertheless contains all the required elements. It starts processing when its main function, in this case process(), is called and when it is done it emits the signal finished() which will then be used to trigger the shutdown of the QThread instance it is contained in.

By the way, one extremely important thing to note here is that you should NEVER allocate heap objects (using new) in the constructor of the QObject class as this allocation is then performed on the main thread and not on the new QThread instance, meaning that the newly created object is then owned by the main thread and not the QThread instance. This will make your code fail to work. Instead, allocate such resources in the main function slot such as process() in this case as when that is called the object will be on the new thread instance and thus it will own the resource.

Now, let’s see how to use this new construction by creating a new Worker instance and putting it on a QThread instance:

QThread* thread = new QThread;
Worker* worker = new Worker();
worker->moveToThread(thread);
connect(worker, SIGNAL(error(QString)), this, SLOT(errorString(QString)));
connect(thread, SIGNAL(started()), worker, SLOT(process()));
connect(worker, SIGNAL(finished()), thread, SLOT(quit()));
connect(worker, SIGNAL(finished()), worker, SLOT(deleteLater()));
connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
thread->start();

The connect() series here is the most crucial part. The first connect() line hooks up the error message signal from the worker to an error processing function in the main thread. The second connects the thread’s started() signal to the processing() slot in the worker, causing it to start.

Then the clean-up: when the worker instance emits finished(), as we did in the example, it will signal the thread to quit, i.e. shut down. We then mark the worker instance using the same finished() signal for deletion. Finally, to prevent nasty crashes because the thread hasn’t fully shut down yet when it is deleted, we connect the finished() of the thread (not the worker!) to its own deleteLater() slot. This will cause the thread to be deleted only after it has fully shut down.

I hope this small tutorial is of some use to people out there. It has taken me months to put together this method of doing things involving countless frustrating hours of debugging and Google searches which kept repeating the same wrong mantra of ‘sub-class QThread’. Constructive feedback on this tutorial is much appreciated. Emails to Nokia on the wrong QThreads documentation might be a good idea too as I would love for the myths surrounding QThread to finally be erased from this world 🙂

Maya

[1] http://doc.trolltech.com/4.7/qthread.html