Shortcode shortcomings

In my last post I discussed the implications of higher-level abstractions over HTML in the content that we author on the web. Love them or hate them, shortcodes are a big part of how WordPress has traditionally enabled content authors to engage with such higher-level abstractions. In this post I want to share some of the challenges shortcodes present and why Gutenberg has chosen a different path, what that path is, and what it means for us all.

Continue reading “Shortcode shortcomings”

Gutenberg posts aren’t HTML…

…and my eraser won’t work with my ink pen!

Lately I’ve been hearing plenty of conversations about the way that Gutenberg posts are saved inside of post_content. If this is all mumbo-jumbo to you, Gutenberg is the project building an entirely new editing experience for WordPress, the software which powers this blog as well as 28% of the internet. Gutenberg is an experimental, risky, and visionary rethinking of how people should be able to create content and structure a website. It’s a break from traditional editors as well as over a decade of tooling and plugins to support that.

Gutenberg also stores necessary information in HTML comments.

<!-- wp:dmsnell/demo { "format": "serializedHtml" } -->
What the heck is going on?
<!-- /wp:dmsnell/demo -->

Continue reading “Gutenberg posts aren’t HTML…”

React in Five Minutes

Three days ago I gave a presentation on React.js at the TucsonJS meetup. The goal was to provide a high-level overview of what React is talk about why someone ought to be interested in applying it. At the end of the talk I gave a live coding demonstration where we iterated a design for a blogging website (it used WordPress.com for the content generation and management).

The actual presentation was built with reveal-md and the source can be found in my GitHub gists.

If you would like to get started and see what a basic React application looks like, you can check out my react-in-five-minutes repository and dive in (consult the README.md for instructions). The app was designed to be just complicated enough to show how to accomplish real-world problems and yet simple enough not to hide the essence of React in a bunch of logic and noise.

Today I created my first monad, and it’s in PHP!

A while ago I caught the functional bug. This is an irresistible urge to write code that conforms to functional principles: mathematical concepts ported into programming that open up new patterns because of the assumptions that are upheld. Essentially it’s a way of doing things that makes it cleaner to program in, makes debugging easier, and generally produces higher quality code.

Monads one small piece of this big picture and a very common one people use is called the Maybe monad. With Maybe you can repeatedly apply functions to a value if and only if it isn’t already nothing. Basically it allows the programmer to shift or remove a bunch of needless error handling.

[sourcecode language=”JavaScript”]
// No Maybe, no happy
const getTotalCost = products => products.reduce( ( total, product ) => {
let sum = total;

if ( ! product.cost ) {
return sum;
}

sum += calculateCost( product.cost );
if ( ! ( product.labor && product.labor.cost ) ) {
return sum;
}

sum += calculateCost( product.labor.cost );
return sum;
}, 0 )

// Maybe a little bit happier
const getTotalCost = products => products.reduce( ( total, product ) => {
let p = Maybe( product ),
add = ( sum, m ) => ( m instanceof Nothing ) ? sum : sum + m.value;

let pCost = p.bind( p => p.cost ).bind( calculateCost );
let lCost = p.bind( p => p.labor ).bind( l => l.cost ).bind( calculateCost );

return [ pCost, lCost ].reduce( add, total );
}, 0 );
[/sourcecode]

While maybe not the greatest example in the world, you can see how it linearized the flow of the function and we were able to remove all of those conditional statements. There are way better examples of this, but the gist is that we can safely compute the labor cost for each item because calculateCost won’t be called for it if there are no associated labor costs in the object.

This isn’t the monad I created.

It’s not even the language I created it in.

I used PHP because I like pain that’s the language of WordPress. How can we bring functional programming patterns to PHP? That’s been the question I’ve been trying to answer.

My first monad handles a familiar PHP pattern to anyone who needs to spit out some text to the user with optional fallbacks in case that string is blank.

[sourcecode language=”PHP”]
// Not too bad, but a little verbose – this could easily get out of hand
function get_title( $blog_id, $fallback ) {
$title = get_blog_option( $blog_id, ‘blogname’ );

if ( empty( trim( $title ) ) ) {
$title = $fallback;
}

if ( empty( trim( $title ) ) ) {
$title = get_blog_option( $blog_id, ‘siteurl’ );
}

if ( empty( trim( $title ) ) ) {
return null;
}

return sanitize( $title );
}
[/sourcecode]

I realized that much of the code I write simply checks that a variable matches a certain condition and then sets it to a new value if it doesn’t, so I decided to create what I am calling the Until monad. It behaves similarly to Maybe, but takes a test function and will continue to process until that test function resolves true.

Monads normally bind functions, but this is PHP where function calls are expensive and the syntax for creating anonymous functions and closures is exhausting. Therefore I have cheated a little bit and allow plain old variables to be passed into the bind method. Let’s explain by illustration:

[sourcecode language=”PHP”]
// Pretty straightforward and concise!
function get_title( $blog_id, $fallback = null ) {
return (new UntilText())
->surely( get_blog_option( $blog_id, ‘blogname’ ) )
->surely( $fallback )
->surely( get_blog_option( $blog_id, ‘siteurl’ ) )
->bind( ‘sanitize’ )
->extract()
}
[/sourcecode]

There are several things going on here. First of all, I have created a subclass of the Until maybe specifically for texts. It makes sure that we have something that isn’t just a blank. We could continue to develop this subclass to make sure that we don’t get strings that are just blank Unicode characters as well.

[sourcecode language=”PHP”]
class UntilText extends Until {
public function __construct( $value = null ) {
parent::__construct( function( $t ) {
return is_string( $t ) && ! empty( trim( $t ) );
}, $value );
}
}
[/sourcecode]

Here you can see the test function – it’s just a plain old function that returns a boolean value. The Until itself looks like this (in pseudocode to spare you the hassle):

[sourcecode language=”python”]
class Until extends Maybe:
construct( test, value )

surely( updater, args ):
if test( value ) if false:
newValue = updater or updater( value, args )
if test( newValue ) works:
return unit( newValue )
return unit( value )

extract():
if not test( value ):
return Nothing
return unit( value )
[/sourcecode]

Why surely? Well, I couldn’t find a good antonym for maybe, so I though, “surely if it hasn’t met the test yet, it will after I update it with this!” This is special because it holds the original passed value until the test succeeds. Note that we can still pass around our good friend, the Maybe bind, offering us some pretty complicated chains. One more example back in JavaScript:

[sourcecode language=”JavaScript”]
const getPreview = post => {
let test = preview => preview.size && preview.size < 200 * 1024 && qf( preview ) > 20,
thumbnailer = source => p => buildThumbnail( source( p ) );

return Until( test, post )
.surely( getFeaturedImage )
.surely( thumbnailer( getFeaturedImage ) )
.surely( getFirstImage )
.surely( thumbnailer( getFirstImage ) )
.surely( getDefaultThumbnail() )
.bind( this.convertToBase64 )
.value;
);
[/sourcecode]

Wow – that was a bit heftier. That’s a function that attempts to build a suitable image preview for a blog post and it tries in sequence the featured image, then a scaled down version of the featured image, then the first image in the post, then a thumbnail of the first image in the post, then a default thumbnail image, finally converting that value to base64 if and only if we found a qualifying preview image. Whew! Note that we store the post object in the monad and it stays there until we get a successful value. That means we don’t have to pass it in at each stage, because it will be applied to those functions that get passed into surely(), equivalent to surely( getFirstImage( post ) ).

Remember that these are lazy evaluations and the chain stops on the first valid update, so it’s a pretty efficient way of getting to your final value.

Honestly I don’t know if someone else has already done something like this. It definitely fills a need I have when writing functions that don’t do much more than build a data structure with varying levels of fallback values. This is also made possible by Anthony Ferrara’s Monad-PHP library – a very big thanks to him!

Shake Reduction

Blur reduction is the next step in improving my astrophography. In fact, I’ve spent lots of time reading and researching blue reduction techniques. With most of what I do, the camera is actually in focus, but the distortions in the lens distort the picture in ways related to the shape of the glass.

Although not directly applicable, I tried out Photoshop‘s shake reduction tool which implements a similar method of deblurring – iterative deconvolution based on inferring a blur kernel from the image. A friend sent me the following test image, taken while on the road and providing plenty of motion blur to eliminate.

You can click on the photos to enlarge them. I’ve also been wanting to get more familiar with ImageMagick‘s tools for similar problems. Specifically we can use Fred’s deconvolution script to perform a non-iterative deconvolution of a source image and a given filter, or in this case the blur kernel. Because this method takes a much simpler approach to deconvolution, it’s very sensitive to noise and small values in the kernel. The noise parameter controls a sort of “fudge-factor” to prevent those divisions from blowing up.

[sourcecode language=”bash”]
./fftdeconvol -n 0.01 hudson.jpg blur_kernel.png deblurred.jpg
[/sourcecode]

Although this approach takes more work than Photoshop’s (as if ImageMagick ever seemed easier at first glance), the result is stunningly better than I was able to get out of Photoshop. The extra work involved manually figuring out the motion path for the blur and creating a kernel image. It wasn’t that hard: the top left corner of the wooden sign gave away a motion of around forty pixels at a positive thirty-degree angle.

[sourcecode language=”bash”]
convert -size 2736×2736 xc:black -fill white -draw "line 1350,1378 1385,1358" -alpha off blur_kernel.png
[/sourcecode]

Finding the right noise value did take some trial-and-effort, though I imagine it wouldn’t be too hard to make some form of automation for this step.

Finally, here is the comparison between the original, Photoshop’d, and ImageMagick’d images.

Megapixel Munching

A friend recently sent me some test pictures from his new Canon PowerShot SX710. He was comparing the quality settings and sending me the results. Curiosity got the best of me and I decided to dig in myself.

These are the originals scaled down for viewing on the web. We will use some 100% crops of these for comparison. Since we’re only dealing with two images, I manually found the crop coordinates by visually matching features in Photoshop.

These were all shot at the same aperture, ISO setting, and exposure length and there’s not too much difference between the shots. The higher-quality and higher-resolution image does appear to have more grainy noise in it. That graininess compared to the smoothness of the smaller image is probably what accounts for the 80% larger file size – high frequency data takes a lot to store.

More megapixels lets us do neat math on the image. For example, by binning the pixels in a 2×2 manner, we cuts the noise in the image by half.

At this point I couldn’t tell a difference between the two images, but the binned JPEG version is about 13% larger than the medium-quality and 13% smaller than the large quality. I assume that if we ran some quality test we would find that indeed the binned version lies somewhere between the two originals, or in other words, for the same resolution, the binned version of the higher quality original will contain more useful information than the medium quality version.

We could also apply a median filter to limit noise from the high quality source. At work here in comparison to binning is that we are using a median-type operation instead of an averaging one.

The 5×5 median clearly has less noise than the original medium-quality source, but it has lost detail and we have already lost spatial resolution in comparison – it was mainly thrown in here just for fun. The 3×3 is a more practical comparison and we can tell that it did a good job of removing noise. Even compared to the medium-quality source it appears to have smoothed some noise while decently preserving detail.

Since the fence is out of focus for all versions, let’s take a look at an object in the foreground. The noise is much more evident around the blades and leaves. I created the following two crops differently, by opening up the images in OSX’s preview app and taking a screenshot. You can see that the magnification is high enough to see the original pixels in the first image but more detail exists in the second.

In this more extreme example the binning seems to preserve more of the detail from the high quality source while mitigating some of the noise. It’s not great though.

Summary

Apparently it doesn’t make much of a difference in these conditions which quality setting the camera uses to store its JPEG image. Granted, according to exiftool these images were captured at around ISO 800, which if pushing it for a camera with a sensor this small. I’d be interested in redoing this test with the camera on a tripod and set to ISO 100 where the noise component would be minimized. The higher quality settings are intended to preserve high-frequency data, so if you have a noisy source, you won’t be doing much more than preserving that noise.

For most casual photography, you won’t be able to tell much of a difference between the M and L settings, even after comparing for resolution. So don’t feel guilty for getting more out of your memory card and working in the lower resolution.

Let’s get a well-lit, ISO 100, on-a-tripod comparison shot of something with lots of edges, lots of grids, lots of fine detail all in focus, something around f/7 and compare that next!

The Makefile for building these images

[sourcecode language=”bash”]
IM=$(shell which convert)

M_CROP="492×610+1783+793"
L_CROP="700×850+2500+1000"
CROP_SIZE="700×850"

default: original_crops binned_crop median

median: L.jpg
$(IM) L.jpg -median 3×3 -crop $(L_CROP) -resize $(CROP_SIZE) -filter box L_Median.jpg
$(IM) L.jpg -median 5×5 -crop $(L_CROP) -resize $(CROP_SIZE) -filter box L_Median5.jpg

binned_crop: L.jpg
$(IM) L.jpg -resize 50% -filter box -resize 200% -filter box -crop $(L_CROP) -resize $(CROP_SIZE) -filter box L_Binned.jpg

original_crops: M.jpg L.jpg
$(IM) M.jpg -crop $(M_CROP) -resize $(CROP_SIZE) -filter box M_Crop.jpg
$(IM) L.jpg -crop $(L_CROP) -resize $(CROP_SIZE) -filter box L_Crop.jpg

clean:
@rm -f *Crop*
@rm -f *Bin*
@rm -f *Median*
[/sourcecode]

Panderings in median calculations

Generating a mean or average value is computationally simple and memory efficient. Generating the median, however, is expensive and bloaty. Since it literally took a day of running on my computer to generate the median for a big astrophotography shoot, I was wondering if there weren’t a faster way, so here is the result of a small experiment I ran at the nudging of a good friend.

Continue reading “Panderings in median calculations”

Mobile Wi-Fi Power Tools

As a digital nomad I find myself trying to work among a plethora of WiFi arrangements and issues. One of the most common frustrations is the act of connecting/reconnecting to a network, and had I taken the time a year ago to develop the following scripts, I would have more than made up for the spent time. Nonetheless, the inspiration and know-how only came after a time of brushing up my BASH-fu while reading man bash.

These snippets should be saved in ~/.bash_profile or ~/.bashrc depending on your setup.

Problem 1: Randomize MAC address

Although programs exist to do this for you, none are required. I’ve been manually running it from the terminal and making up the numbers on the fly, but a more automated and more random solution is better.

[sourcecode language=”bash”]
# randomize MAC address
function macr () {
local CMD="sudo ifconfig en0 ether "
for i in `seq 1 12`; do
C=`echo "obase=16; $(((RANDOM % 15)+1))" | bc`
CMD="$CMD$C"
if [ $((i % 2)) == 0 ] && [ $i != 12 ]; then
CMD="$CMD:"
fi
done
eval $CMD
}
[/sourcecode]

Running macr from the command line will immediately prompt your for your password (if not already cached) and change the MAC, meaning it might kick you off your authenticated session.

Many thanks to Jonathan Zdziarski for his original script which I modified.

Problem 2: Swap DNS servers

Many places that offer free Wi-Fi have a login screen with a notice, maybe a button to accept the terms of service, and maybe a login screen. If you are like me and tend to use Google’s DNS servers or any custom DNS, then you run into the problem that you will never see this login screen and thus won’t ever authenticate to the wider Internet. It’s a hassle to open the network settings and manually remove the DNS servers, click login, then repeat and add the custom ones again. This command will allow you to check on the DNS settings from the command line and swap out your custom servers of choice.

[sourcecode language=”bash”]
# swap DNS servers
function ldns () {
if [ "on" = $1 ]; then
sudo networksetup -setdnsservers Wi-Fi 8.8.8.8 8.8.4.4
elif [ "off" = $1 ]; then
sudo networksetup -setdnsservers Wi-Fi Empty
elif [ "get" = $1 ]; then
local NETWORK_SETUP=$(sudo networksetup -getdnsservers Wi-Fi)
if ! [[ $NETWORK_SETUP =~ \b(?:\d{1,3}\.){3}\d{1,3}\b ]]; then
NETWORK_SETUP=”
fi
local RESOLVE_CONF=$(cat /etc/resolv.conf | grep nameserver | cut -d’ ‘ -f2,2)
local SCUTIL=$(scutil –dns | grep nameserver | cut -d’:’ -f2,2)
echo "$(echo $NETWORK_SETUP $RESOLVE_CONF $SCUTIL | tr ‘ ‘ ‘\n’ | sort -u)"
fi
}
[/sourcecode]

You can turn on “local DNS settings” with ldns on and revert back to the router-given DNS settings with ldns off. Verification comes with ldns get. When I connect to a network, I can now follow this quick and easy sequence.

[sourcecode language=”bash”]
ldns off
# connect to WiFi network, login or accept terms
ldns on
[/sourcecode]

That simple!