There’s no such thing as an optional dependency

On several occasions I have tried to explain my opinion about “optional dependencies” (also known as “suggested dependencies” or “dev requirements”) and I’m doing it again:

There’s no such thing as an optional dependency.

I’m talking about PHP packages here and specifically those defined by a composer.json file.

What is a dependency?

First let’s make sure we all agree about what a dependency is. Take a look at the following piece of code:

namespace Gaufrette\Adapter;

use Gaufrette\Adapter;
use \MongoGridFS;

class GridFS implements Adapter
{
    private $gridFS;

    public function __construct(MongoGridFS $gridFS)
    {
        $this->gridFS = $gridFS;
    }

    public function read($key)
    {
        $file = $this->find($key);

        return ($file) ? $file->getBytes() : false;
    }
}

This GridFS class is part of the Gaufrette filesystem abstraction library, though I heavily modified it.

To determine all the dependencies of this code we can ask the following question:

What is needed to run this code?

You need to think of several things:

  1. Which PHP version is needed to run the code without getting a syntax error? Maybe you even need a specific patch version (like 5.3.6) because of a bug in older 5.3 versions that could interfere with your code.
  2. Which PHP extensions should be installed?
  3. Which PEAR libraries should be installed?
  4. Which other packages should be installed?

In the case of the GridFS class the PHP version should be at least PHP 5.3, because of the use of namespace. Also the \MongoGridFS class should be available. This class is part of the mongo PECL extension for PHP. The \MongoGridFS class is only available since version 0.9.0 of that PHP extension, so we have to make sure that we explicitly mention this version constraint. Finally, it appears there are no other packages needed to be able to use the GridFS class. So when we would create a composer.json file for a package that contains the GridFS file, it would look like this:

{
    ...,
    "require": {
        "php": ">=5.3",
        "ext-mongo": ">=0.9.0"
    }
    ..
}

Now this is an exhaustive list of the dependencies of package that contains the GridFS class: when these dependencies are installed, nothing stands in the way of using this class in your application.

The actual list of dependencies of knplabs/gaufrette

As I already mentioned the GridFS class is part of the Gaufrette library which provides a filesystem abstraction layer so you can store files on different types of filesystems without worrying about the details of those filesystems. Let’s take a look at the composer.json file of this library:

{
    "name": "knplabs/gaufrette",
    "require": {
        "php": ">=5.3.2"
    },
    "require-dev": {
        ...
    },
    "suggest": {
        ...
        "amazonwebservices/aws-sdk-for-php": "to use the legacy Amazon S3 adapters",
        "phpseclib/phpseclib": "to use the SFTP",
        "doctrine/dbal": "to use the Doctrine DBAL adapter",
        "microsoft/windowsazure": "to use Microsoft Azure Blob Storage adapter",
        "ext-zip": "to use the Zip adapter",
        "ext-apc": "to use the APC adapter",
        "ext-curl": "*",
        "ext-mbstring": "*",
        "ext-mongo": "*",
        "ext-fileinfo": "*"
    },
    ...
}

After what we’ve discussed above, this is quite a surprise: the library says it has only one actual dependency: a PHP version that is at least 5.3.2. Everything else is either a “dev” requirement or a “suggested” requirement.

Of course people who use Composer and Packagist for some time now (like myself) have become quite used to this way of advertising the dependencies of a package. But it is just wrong. As we concluded earlier, ext-mongo is a true dependency of the GridFS class, yet looking at the composer.json file it is only a suggested dependency.

This means that if I want to use the class in my project, it is not sufficient to require just the knplabs/gaufrette package. I also have to add ext-mongo as a requirement to my own project. Which is semantically wrong: it is not my project that needs the mongo extension, it is the knplabs/gaufrette package that actually needs it. Besides, how do I know which version of ext-mongo I have to choose? Dependencies listed under the suggest key in composer.json don’t come with version constraints, so I have to figure them out myself.

Not just this package

knplabs/gaufrette is not the only package out there that advertises actual, required dependencies as “suggested” dependencies. It is a convenient way for package maintainers to put a lot of different classes in a package that may or may not be needed by users. Since using those classes is optional, their dependencies are made optional too. But package maintainers forget that dependencies never are optional. They are always required, since the code would not be executable without them.

The solution

What package maintainers should do is split their packages. In the case of knplabs/gaufrette this means there should be a knplabs/gaufrette package containing all the generic code for filesystem abstraction. Then each specific adapter, like the GridFS class, should live in its own package, e.g. knplabs/gaufrette-mongo-gridfs. This package itself has the following dependencies:

{
    ...,
    "require": {
        "php": ">=5.3",
        "knplabs/gaufrette": "~0.1"
        "ext-mongo": ">=0.9.0"
    }
}

No hidden dependencies there: everything is truly required for using the code in this package.

On the other hand the knplabs/gaufrette package has almost no dependencies anymore, and the “adapter” packages are listed under the suggested key:

{
    "require": {
        "php": ">=5.3.2"
    },
    "suggested": {
        "knplabs/gaufrette-mongo-gridfs": "For storing files using Mongo GridFS",
        ...
    }
}

This approach has many advantages:

  1. The main package will be very stable. There are almost no reasons for it to change anymore, since all the moving parts are inside the “adapter” packages.
  2. The adapter packages can have different specialists as maintainers, for instance the knplabs/gaufrette-mongo-gridfs can be maintained by someone who knows all about MongoDB.
  3. Users don’t have to keep track of available updates for parts of the library they don’t use.
  4. Users don’t have to manually add extra dependencies to their projects (which means they don’t have to worry about version constraints for them).

So keep in mind, next time you are tempted to add a suggested dependency to your package: is it an actual dependency of (part of) the code in this package? Then split the package and reinstate that dependency as a true requirement. If all the code in the package works perfectly well without that suggested dependency, then you are indeed allowed to advertise it as a suggested dependency.

Want to know more?

bookpage-principles-of-php-package-design

I’m working on a book about package design principles, based on the work of Robert Martin. You may register yourself as an “interested reader” and receive a considerable discount when the first part of the book becomes available next week.

You may also want to read some of the articles about package coupling by Paul Jones (Frameworks are good, components are awesome!, Symfony components: sometimes decoupled, sometimes not). He maintains the Aura framework and components and does a great job when it comes to package coupling.

Posted in Book, Principles of PHP Package Design | 13 Comments

Test Symfony2 commands using the Process component and asynchronous assertions

A couple of months ago I wrote about console commands that create a PID file containing the process id (PID) of the PHP process that runs the script. It is very usual to create such a PID file when the process forks itself and thereby creates a daemon which could in theory run forever in the background, with no access to terminal input or output. The process that started the command can take the PID from the PID file and use it to determine whether or not the daemon is still running.

Below you find some sample code of what a daemon console command would look like. It forks itself using pcntl_fork(), a fascinating function which has different return values at the same time (but in different processes!).

use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class DaemonCommand extends Command
{
    protected function configure()
    {
        $this->setName('daemon');
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $pid = pcntl_fork();

        if ($pid === -1) {
            throw new \RuntimeException('Could not fork the process');
        } elseif ($pid > 0) {
            // we are the parent process
            $output->writeln('Daemon created with process ID ' . $pid);
        } else {
            file_put_contents(getcwd() . '/daemon.pid', posix_getpid());
            // do something in the background
            sleep(100);
        }
    }
}

Usually you would test a console command using the ConsoleTester class as described in the official Symfony documentation. But we can not use it in this case. We need to isolate the process in order for pcntl_fork() to work correctly, otherwise the PHPUnit test runner itself will be forked too, which will have the effect of running all the following unit tests twice (which can have very strange effects on the test results).

So first we need to start the daemon command in its own process. We can use the Symfony Process Component for this.


use Symfony\Component\Process\Process;

class DaemonCommandTest extends \PHPUnit_Framework_TestCase
{
    /**
     * @test
     */
    public function it_creates_a_pid_file()
    {
        $process = new Process('php app/console daemon');
        $process->start();
    }
}

However, since our console commands crosses the boundaries of one process, this will not suffice. As soon as the daemon child process has been created, it has a life on its own and you can not be sure if and when the daemon has executed its tasks, like creating a PID file. Simply calling assertTrue() like this

public function it_creates_a_pid_file()
{
    $process = new Process('php app/console daemon');
    $process->start();

    $this->assertTrue(file_exists('daemon.pid'));
}

would be nice, but turns out to be unreliable. Tests like these are called “flickering tests”, since they will sometimes fail, sometimes succeed, in a very unpredictable manner. So before we start making assertions about the expected behavior of the daemon we need to wait a reasonable amount of time for the child process to get up and running:

public function it_creates_a_pid_file()
{
    $process = new Process('php app/console daemon');
    $process->start();

    // 5 seconds should be enough to start the daemon, right?
    sleep(5);

    $this->assertTrue(file_exists('daemon.pid'));
}

Better safe than sorry: we wait 5 seconds and make the assertion. This is of course really bad for your test suite. Where the other 1000 unit tests all run in one second, this one test takes at least 5 seconds! Even worse, 5 seconds may not even be enough when, for instance, the daemon somehow triggers the Symfony cache to be rebuilt, which takes much longer than that on most machines. Then again, if the cache does not need to be rebuilt, 5 seconds may be way too much, and 500 milliseconds would be more appropriate.

Polling

Instead of putting the process into a long sleep we need to check regularly if the daemon.pid file has been created, make an assertion and leave the test method as soon as possible. In other words, we need a polling mechanism and a probe. The probe inspects the current system’s condition and lets the polling mechanism know if it is happy about that condition. In case the desired condition is never reached, the polling mechanism keeps track of time and gives up after a number of seconds (I read about this first in Growing Object-Oriented Software, Guided by Tests, an excellent book that taught me many new and interesting things about TDD).

In the case described above, we could implement such a polling/probing/timeout mechanism with a simple loop:

$keepTrying = true;
$startTime = time();
while ($keepTrying) {
    if (time() - $startTime > 5) {
        $this->fail('We waited for 5 seconds but the PID file was not created');
    }

    if (file_exists('daemon.pid')) {
        $keepTrying = false;
    }
}

Well, I don’t like this type of error-prone code in my test methods so I decided to abstract some things into a fully tested library for PHPUnit, which is called PHPUnit Asynchronicity. You can install it in your project using Composer (the package is called matthiasnoback/phpunit-asynchronicity). It allows you to use one simple assertion to accomplish the same thing as described above:

use Matthias\PhpUnitAsynchronicity\Eventually;

class DaemonCommandTest extends \PHPUnit_Framework_TestCase
{
    /**
     * @test
     */
    public function it_creates_a_pid_file()
    {
        $process = new Process('php app/console daemon');
        $process->start();

        $this->assertThat(
            function () {
                return file_exists('daemon.pid');
            },
            new Eventually()
        );
    }
}

You could abstract this a bit further, which would make it better readable:

class DaemonCommandTest extends \PHPUnit_Framework_TestCase
{
    public function it_creates_a_pid_file()
    {
        ...

        $this->assertEventually(
            function () {
                return file_exists('daemon.pid');
            }
        );
    }

    private function assertEventually($probe)
    {
        $this->assertThat($probe, new Eventually());
    }
}

The Eventually class is a PHPUnit constraint that accepts two arguments: a timeout in milliseconds (so 5000 milliseconds would mean a timeout of 5 seconds) and a wait time in milliseconds. After each wait time the probe will again be asked to examine the current state of the system.

By the way, it’s also possible to create a probe that is not a closure, but a stand-alone class:

use Matthias\Polling\ProbeInterface;

class PidFileExists implements ProbeInterface
{
    public function isSatisfied()
    {
        return file_exists('daemon.pid');
    }
}

Then it reads even better:

class DaemonCommandTest extends \PHPUnit_Framework_TestCase
{
    /**
     * @test
     */
    public function it_creates_a_pid_file()
    {
        ...

        $this->assertEventually(new PidFileExists());
    }
}

Posted in Console, PHPUnit, Symfony2, Testing | 2 Comments

About coding dojos, the Symfony meetup and my new book

Last week was a particularly interesting week. I organised three coding dojos, on three different locations. The first one was at the IPPZ office (one of the companies I’ve worked for), the second one was at the SweetlakePHP meetup in Zoetermeer. The last one was for a group of developers from NOS, a large Dutch news organization. Although that would have been enough, I finished the week with my revised talk “Principles of PHP Package Design” at a meeting of the Dutch Symfony Usergroup in Utrecht…

About this coding dojo thing

First, let me explain what I am up to, organizing all these coding dojos!

A coding dojo is a meeting of developers (who are in this case to a certain minimum level acquainted with PHP). Two of them take a seat behind a computer in front of the others, who are the “audience”. They get a programming assignment and start writing code to satisfy the requirements. They use a strict test-first approach, taking small steps, slowly approaching the finish. After about seven minutes, the roles change, so that in te end everybody gets their turn in writing some code (and tests!). And of course, the assignment is done.

A coding dojo programming assignment is called a “kata”. There are many of them, some of which you may know already (FizzBuzz, Minesweeper, Bowling Game, Roman Numerals, etc.). They all are a way to practice your TDD skills and while doing so, you can sharpen your eye (or nose!) for code smells and learn to apply OOP design principles.

The dojo itself is a pretty safe area, where you can talk about anything that comes to mind while you are programming (you would usually keep all those things to yourself) or watching somebody else programming. It is also a place where you can learn about other approaches, which can be more (or less) fruitful than your own habits. Thus far my conclusion is that a coding dojo is a wonderful way to get a team’s thoughts about best practices and quality standards better aligned. It appears that regularly being in a coding dojo helps to achieve this even better.

I intend to organize coding dojos more often, on many different locations and occasions. If you, your organization or your local user group is interested, please contact me.

Dutch Symfony Usergroup meetup

As you may know, I’m working on a book called Principles of PHP Package Design. A couple of months ago I thought up a talk with the same title for the AmsterdamPHP usergroup. And even though it was a bit “rough” back then, it was recorded, so you can even watch it on YouTube. On this specific occasion it seemed to me that the way I approached package design in my talk did not resonate very much with the audience. In fact, it brought up quite a lot of counter-reactions. This meant for me that 1) I needed to make myself much more clear and 2) maybe I was saying too many things that totally did not align with the way people thought about package design at the time.

I feel that much has changed. Last week I modified big parts of the contents of the presentation. I added many images to it. Then I presented it at the Dutch Symfony Usergroup meetup in Utrecht. These are the slides:

I was very happy to see that the package design concepts and principles were well received by the audience. This is why I think it’s almost time to publish the book even though it is not finished yet. The first half of it is almost done, and after that it should be possible to regularly release another chapter.

Lean publishing

So during the next couple of months I will start releasing chapters of my second book. Until the book is completely done, you will be able to buy it for a reduced price.

bookpage-principles-of-php-package-design
The regular price will be 29 dollars, but:

If you have bought or will buy A Year With Symfony within the next few months, you will get a 25% discount on Principles of PHP Package Design (you will receive a discount coupon per mail).

If you didn’t, but you did subscribe using your email address on the book’s page, you will automatically receive a coupon code for a 20% discount.

So stay put and you will soon learn more about all of this!

Posted in Book, Coding dojo, Principles of PHP Package Design, Symfony2 | Leave a comment

A Year With Symfony: Bonus chapter is now available!

My first book, A Year With Symfony, has been available since September 4th last year. That’s not even six months ago, but right now it has almost 800 readers already (digital and printed edition combined).

During the past few days I’ve taken the time to write an extra chapter for the book. Consider it as a big thank you to everybody who bought the book! I feel very much supported by the Symfony/PHP community and this really keeps me going.

Annotations

annotations-chapter

The bonus chapter is about annotations and how you can use them to control the flow of a Symfony application. It is a subject I’ve written about some time ago, but this time I discuss many more details and different approaches.

Digital edition

If you have bought the digital edition from the Leanpub website you can just download an updated version of the book.

Paper edition

Within a couple of days, newly printed books will also have the bonus chapter.

If you have already bought the paper edition on the Lulu website I can send you a PDF/MOBI/EPUB file containing just the new chapter. Please send a mail with your request to matthiasnoback@gmail.com. In your mail, please mention your order number and the order date.

Un Año Con Symfony

The Spanish translation of the new chapter is not yet finished. If you’ve bought that book, you will be notified automatically when it is available.

Thanks for helping

The new release of the book contains many small fixes of typos and syntax problems that were reported by John Nickell, Rodrigo Rigotti, Andras Ratz, Rémy Gazelot, Piotr Kędra, Dorian Villet, Greg Szczotka, Pera Jovic, Sebastian Blum and Rolf Babijn. Thank you for that!

Also, the new chapter has been proof-read by a former co-worker – Matthijs van Rietschoten – and by the translator of the Spanish edition of “A Year With Symfony” – Luis Cordova. Thanks to you too!

Posted in A Year With Symfony, Book | 2 Comments