Running the WordPress Tests

A while ago I set out to run the Core WordPress test suite – the unit tests – in both PHP and in JavaScript. I’m not one to enjoy leaving around possibly-messy state on my computer so I like to package things up nice and cleanly inside of Docker. Since I hadn’t run the tests before it turned out to be quite a journey of learning and experimenting to get things running – far more work than I anticipated. In the end I found something that seems to work for me.

Trying to find an existing solution

My first attempts involved scouring the internet for posts or documentation on how to run the tests but I had a hard time finding relevant and recent information. The official guide left me full of questions and assumes you have a working test environment already – wasn’t too helpful.

Next I asked a few WordPress contributors what they use and the feedback was similar: “I use a working test environment.”

So I tried to create my own setup from within Docker. I spent some time trying to get the phpunit base image to work but it only worked for PHP and left me without necessary modules. Then I tried starting from the node base image but had trouble getting the PHP packages installed. Although I really wanted to keep a very minimal setup I figured out that I had to create my own Docker image.

Making my own solution

My solution uses a custom Dockerfile paired with a simple docker-compose.yml configuration. I use docker-compose so that I can keep the database separate from the code; this came in handy when I realized that I had to use mysql instead of mariadb because it ended up being a one-line change.

Getting some tests running was easier than getting all tests running. I believe that at this point I have figured out all the implied dependencies and included them into my Dockerfile. The Core documentation notes that php-mbstring is needed but it skips over a number of other required dependencies. I still have ten tests that get skipped and I don’t know why, but I can’t put more time into it now.

You can inspect my Dockerfile to see what I did. There’s plenty of room for improvement and I’d like to refine it as I understand everything better. I wanted to be able to mix the phpunit and node base images and thought I might have been able to with multi-stage builds but that turned out to be more confusing for me, so I stuck with the imperative install of everything inside of ubuntu.

The strangest thing that I had to work through while putting this together was that every time I ran phpunit it would spit out a chunk of binary-looking data into the console, even if all I was running was phpunit --list-suites.

Installing...
Running as single site... To run multisite, use -c tests/phpunit/multisite.xml
Not running ajax tests. To execute these, use --group ajax.
Not running ms-files tests. To execute these, use --group ms-files.
Not running external-http tests. To execute these, use --group external-http.
�}k�G��g�W�Ȗg�
��s���ʪ�|3�}s����A3�ϓY�����R7��8_�y�^7������EY�{s�N�Dzn��s�.��篽�<���׌�Nw���t��E��(�I�A��fh�)�^���2�Ro�7��� $A:�Y/��"��|�s����VI�$-\    i�&u�*-�    �൱�����"�0�ʃ�`i.��&�I9�âh�z����9ȲA���/Ţ��<{~��_�y���YZ��`';F?膝,o-    �^sç?}�5ާ��(�V�<��˳���Y�Q\xM�K ���y�L��x#�U�~��n<b6�w��ʰ`�q�e�]��i/<�d��u¼N�}D�J�R��[�{��~(��`�Rp
�X]�q��ü����Fawt����[�Y��õ ��l4f�҉���S��<��P��Q�lR�xe�$�j����M��M�&�������B��y�`߀� .<�>��=����2�����!����n�;���W��p���DS�Q|�=OPhW����^��#Y;q��:��&��9�����������5��A7~�    �S�`�H|�[��ꖡ�ꌊw-X�L����a���+n���u��gX#�$ɚ�ψl�fG�FG�3SMi/x��uӔ�d�Dٚ?
Һ�.։�E�Wc����{��iս�MCUu�QW�}8Ơ=7�?n�i����
��7d��^RI��V済(_�Kw׵���D����I,�-{A>E2�t��o����Z=a�!�륥�~�(�V�풵%���Q�:��ᚿ�����#
[and thousands of lines more before the normal text output starts]

This turned out to be a consequence of not having php-curl installed. In its absence, a test file which downloads GlotPress data was using the fallback fsockopen implementation of WordPress’ HTTP library and for a reason unknown to me was printing the gzip-compressed copy of the file it downloaded. How do I know this? I saved the output to a file, cut out the recognizable text, did a hexdump, saw the gzip magic number, then ran the file through gunzip. Funny.

Note that one thing not shown in the Docker files is that I clone the wordpress-develop directory on my own computer then mount it into /wp in the Docker image.

docker-compose up mysql &
docker-compose run wordpress
~# grunt phpunit
~# grunt qunit

In conclusion

I’d like to know how many people are running these tests and I’d like to understand any automated runs of them better. Running against the latest code in wordpress-develop gives 61 errors and 28 test failures. Something seems wrong here and I think it means we are ready for some updates to the Core testing documentation and process. I know that my setup is still wrong in several places, but I would love to catalog all of the required steps to get it 100%.

Feel free to use my setup. If you have questions just leave a comment here. Thanks for reading!