boutell: (Default)
How much snow is on Jill's bucket? Once again we attempt to answer the ancient question with inappropriate web technologies.
boutell: (Default)
Here comes the snow again! And we're ready to measure it... with PHP scripts and the Internets.

The forecast for Delaware is a bit inconclusive. Will there be any actual snow? Who knows? But we're ready to measure it goshdarn it.

Edit: fixed the link. There was not a lot of snow, but the SMS performed admirably
boutell: (Default)

Difficulties in the application of web technology to the measurement of snow on Jill's bucket

Or, I'm Dreaming of a White Paper

Thomas Boutell, BSS, BSL (**)

In February 2010 we had remarkable success in measuring snowfall on Jill's bucket through the misapplication of web technology. This experience led us to deploy the same technology with great confidence in December 2010 (*). However, new challenges arose.

Chief among these was camera drift. As we will see, the camera's viewpoint drifted upwards considerably during the night. Consider these three images taken by Jill's snowcam:

Figs. 1-3: camera drift.

As you can see, the top of Jill's snow gauge has drifted downwards by many pixels in each successive image (34 pixels from the top, then 56 pixels from the top, then 70 pixels from the top). This is the easiest way to observe the drift problem, although there was also horizontal drift as you can see in Figure 4.

What does this mean for our snow measurement system?

The snow measurement system (SMS) depends on the advance selection of a calibration point which is definitively (1) on the bucket, and (2) already in the snow. The snow detection algorithm then proceeds upwards until it encounters a significant discontinuity in pixel brightness. This is presumed to be the point at which the snow stops. Since Jill's snow gauge is of a contrasting color this worked well in February.

However, last night, with significant wind, the camera's view drifted upwards steadily through the night. That is, the bucket appeared to sink.

The end result was this:

Fig. 4: SMS FAIL!

What went awry? As you can see, the original calibration point (indicated by a red dot) is now well above and to the left of the actual presumed origin point of the snow (also indicated by a red dot). The snow detection algorithm obediently measured upwards from this point until it encountered a discontinuity (the bottom of the swingset), resulting in an incorrect snowfall measurement. Had the camera and the bucket remained in exactly the same position and orientation the snowfall measurement would have been accurate. Alas, under heavy wind conditions this is not a reasonable expectation.

Thus we have (inadvertently) shown that advance selection of a snow calibration point (SCP) is a problematic strategy for the measurement of snow via PHP, gd, webcams, equally obsessive friends in adjacent states and the Internets.

Future Directions

What steps can be taken to improve our results? It is apparent that the snow gauge is a difficult feature for the human eye to miss. It is reasonable to ask whether the gd-based PHP script responsible for measuring snowfall could detect the snow gauge.

We believe this is possible since the gauge is by a considerable margin the darkest feature in the image. One strategy would be to identify the snow gauge as a contiguous group of dark pixels, count them, compare this to the known result for the first frame in the series, and thus determine the amount of snowfall.

Looking at an early frame, this seems quite viable:

Fig. 5: a frame from the beginning of the series, representing no snow (to a first approximation)

However there is the problem of glare (see fig. 1). At differing times of night there is a considerable white glare on the upper part of the snow gauge.

A possible solution can be seen in the less reflective, non-monochromatic snow gauge of February 2010 (see Fig. 6).

Fig. 6: February 2010 snow gauge

The wooden snow gauge has two distinct advantages: color, and a relative lack of glare.

This suggests a way forward. It is believed that a blue, green or other distinctly non-grayscale snow gauge of a non-glossy nature may be the best option as it offers the option of distinguishing snow from snow gauge by hue as well as value.

Although it initially appeared that the original yellow snow gauge could be a viable option, analysis of images from February 2010 indicates that the top of the snow gauge was in fact registered as pure white by the webcam. There is also the possible problem of incorrect analysis due to the presence of unruly canines.

Green, however, is distinguishable even under full glare conditions and thus may be a better choice, provided that unseasonable vegetative growth is not present. Blue and red also have potential.

In an ideal world, the present conditions at Jill's house (e.g. snow on the ground) would be harnessed to test this option. Images of a green snow gauge of known height, with and without several inches of snow on the bucket, and with and without nighttime floodlamp conditions, would be captured and made available for analysis at our laboratories. It would then be possible to tune the SMS appropriately in advance of the next snowfall, testing the potential for automatic detection of the snow gauge and automatic measurement of the percentage as yet uncovered by snow, resulting in an accurate snowfall figure. It is our hope that such experiments may be carried out in the near future.

The author wishes to thank [ profile] xtingu for making her snowcam available and aiding and abetting his misapplication of the data in any number of ways over the course of these experiments, and [ profile] mrlich for his patient assistance, indulgence and tacit endorsement of these activities.

(*) Bullshitter in Snow Science, living near the Broad Street Line.
(**) While the December 2010 results appear correct on the official page, this is the result of a retroactive recalibration of the system and thus unsatisfying.
boutell: (Default)
Dear Super Badrats,

I launched a new website yesterday. What Shall We... is an insanely simple way to plan dinner, drinks, a movie party, or pretty much anything that isn't formal with people that you place at least a little trust in. I'm gunning for a level of simple that just doesn't happen with tools that require accounts (who wants another account anywhere ever?) and limit the amount of input that your peers are allowed to give in the decision. I'm also shooting for realtime here - you can watch people vote and suggest stuff on the fly. Which hopefully encourages folks to stay involved and come to consensus on the decision.

Please give it a look-see and let me know what you think!

What Shall We...

boutell: (Default)
The gd graphics library is and for many years has been the standard way of generating graphics on the fly in PHP and a lot of other web development languages. When you call imagecreatefromjpeg in PHP, you're using gd.

There are excellent alternatives that are in fact more capable in many ways, but people still dig gd for its dead-simple API and stodgy reliability.

gd is my doing— I originally built it back in '93 for a project at Cold Spring Harbor Laboratory and, in part since it was heavily based on freely available GIF compression and decompression code by Jef Poskanzer and David Rowley, I was able to make it open source software.

gd didn't catch on because of particularly brilliant code, I can say that for certain, but rather because it was easy to use and had a very consistent API. Things work like you'd expect and are called what you'd guess. I'm not much of a graphic designer, but it's fair to call me a good API designer (when I'm not being sloppy).

gd has been largely stable but did go through a few big changes. When Unisys suddenly began enforcing their patent on the compression used in GIF images, GIF compression had to disappear from gd for a few years. Folks who patched it back in without telling me as part of other contributions to gd taught me to read every line of code contributions carefully.

Then came gd 2.0, with support for truecolor images— meaning faithful reproduction of photos and support for JPEG— and alpha channels. Truecolor was widely appreciated. My somewhat sleazy implementation of alpha channels— only seven bits rather than eight to avoid big changes to the entire API— is not as widely used, partly because PNG alpha channel was supported so poorly by browsers for so long.

The last versions I was involved with saw the addition of animated GIF output support, which never quite made it into the PHP documentation. Since PHP has become by far the most public face of gd, that meant that it never really entered the consciousness of the average developer.

A few years ago I passed on maintenance of the gd library to Pierre-Alain Joye of the PHP project. I wasn't dealing with contributions in a timely manner and my work rarely involved graphics at the time, so it was time to pass the baton of this particular "cathedral-style," benevolently-dictated open source project to a more motivated party. Now gd seems to have gone a bit stale in its new home as well, but this is not entirely a bad thing since the code is quite mature and doesn't need radical transformation.

However... there is no such thing as bug-free code.

Yesterday I needed code to generate animated GIFs and, ironically, didn't remember gd's support for animated GIF output right away. It took me a bit of banging around to realize a fine solution for turning perfectly good JPEG photos into silly animated GIFs was right under my nose.

Unfortunately, it didn't work. With 256-color GIFs, I kept producing invalid GIF files. Which led me, for the first time in nearly a decade, to the gd source code that generates the GIF header:

         * Indicate that there is a global colour map
        B = GlobalCM ? 0x80 : 0;

         * OR in the resolution
        B |= (Resolution - 1) << 5;

With an 8-bit palette, Resolution - 1 = 7. Shift left five bits and you have just crushed the global colormap flag.

Augh! No wonder my animated GIFs with local colormaps weren't working.

I consulted the GIF specification and found the resolution should have been shifted over only four bits.

This bug is old. I mean, really old. It goes back to the code I borrowed from Jef Poskanzer who borrowed from it David Rowley who wrote it back in the year dot. Even the tool I used to track down the bug is old. It was never reported before because the resolution field is universally ignored and single-image GIFs made by gd always have a "global" colormap anyway. So the bug never manifested itself before animated GIF support was added.

I've submitted a bug fix to the new gd team, and given them a nudge elsewhere just in case they are, like me, sleeping comfortably knowing that gd can't possibly have any bugs left after seventeen years of happy production use... right?

It may take a while for that fix, along with the long-delayed documentation of and wrapper functions for animated GIF support, to find its way into PHP. But I plan to be a more vocal advocate for maintenance of gd now that it's back in my professional life as a web developer, among other things too sneaky and sekrit to talk about just now.
boutell: (Default)
Thanks to all, especially [ profile] iamo, for the thoughtful responses to my question about fastcgi.

I eventually figured out that the fastcgi "dynamic server" could live in a shared folder, resulting in only one set of PHP processes, if I used a simple Alias directive to map that path into all of the virtual sites.

With that change I was able to do this:
<IfModule mod_fastcgi.c>
  # One shared PHP-managed fastcgi for all sites
  Alias /fcgi /var/local/fcgi
  # IMPORTANT: without this we get more than one instance
  # of our wrapper, which itself spawns 20 PHP processes, so
  # that would be Bad (tm)
  FastCgiConfig -idle-timeout 20 -maxClassProcesses 1
  <Directory /var/local/fcgi>
    Options ExecCGI
  AddType application/x-httpd-php5 .php
  AddHandler fastcgi-script .fcgi
  Action application/x-httpd-php5 /fcgi/php-cgi-wrapper.fcgi

Coupled with the wrapper script I showed before, that works like a big working thing.

Then I switched Apache to the worker MPM rather than the prefork MPM (again, thanks for the suggestions), which really blew performance for static files through the roof.

The only remaining catch: no support for php_value settings in .htaccess files. Doh. Fortunately the settings I had in place there were easily shared among the virtual sites in question, so I swept them to php.ini.

(Yes, I know I really shouldn't have .htaccess files on sites where I'm concerned about performance anyway.)

Thanks again for the tips!
boutell: (Default)
Dear extremely amazing Apache geeks,

I don't ask much. I just want PHP to run in one process pool WITH a shared APC cache, and Apache to be in its own process pool as a front end so it can serve static stuff really, really fast, and not pin down a huge Apache process with mod_php in it when it's just serving something statically. Better yet I'd like to use the worker MPM so it's threaded and even more ridiculously fast.
fastcgi is supposed to be the way... clicky if you think you might be able to help. )
Thanks for your advice!
boutell: (Default)
The first official stable release of our content management system is out!

Sure, this is still not as straightforward as installing WordPress. Right now our primary goal is still to enlist other developers to help us make it more awesome by providing them with something rock-solid they can sell their clients on... something a whole lot more client-friendly than Drupal.

But oh, does it feel good to slap that "stable" stamp on a solid product.

(T- ten seconds to the first of many inevitable observations to the contrary (: )
boutell: (Default)
"The PHP development team is proud to announce the availability of the second release candidate of PHP 5.3.0 (PHP 5.3.0RC2). This second release candidate focused on bug fixes and stability improvements and we hope to only require minimal changes ahead of the next release. Many, but not all, of the new features are already integrated in the official documentation on not that we are aware of minor issues including that we still want to fix (though most of them have also affected PHP 5.2) and a crash bug in affecting source files with a size of exact 8k (See Bug #48034). Expect an RC3 in 2-3 weeks time, though for most users there will not be a noticeable change meaning that now is the time to start doing the final testing of PHP 5.3.0 before it gets released with any unnecessary incompatibilities with your project. Please download and test the sources for *nix and Windows binaries and report any issues found."

Oh yeah, I'm in a HUGE rush to test this thing against my project. Unless any of my files happen to be 8K in size.


I am, actually, in a hurry to have PHP 5.3 because it offers a cycle-breaking garbage collector, which means it can be used for long-running processes. But come on.
boutell: (Default)
My jsonwrapper code is currently part of Facebook's official standard library for building Facebook apps in PHP.

I only noticed because I was poking around to make sure I'd unpacked their latest code in the right place and started to worry when I saw what seemed to be my own code in what should have been a brand new folder from Facebook! Hey, what's going on, that's my code... that's... oh yeah, I released it. So they used it. Cool.

Sadly, I put it in the public domain and left out any credit notice in the source out of a sense of modesty: jsonwrapper itself is maybe 30 lines of code, if that, and all it does is wrap somebody else's library that actually implements JSON for poor bastards who still have PHP versions earlier than 5.2 so that you can write code exactly as if you had 5.2. I didn't feel it was appropriate to take ostentatious credit for:


# In PHP 5.2 or higher we don't need to bring this in
if (!function_exists('json_encode')) {
        require_once 'jsonwrapper_inner.php';

require_once 'JSON/JSON.php';

function json_encode($arg)
        global $services_json;
        if (!isset($services_json)) {
                $services_json = new Services_JSON();
        return $services_json->encode($arg);

function json_decode($arg)
        global $services_json;
        if (!isset($services_json)) {
                $services_json = new Services_JSON();
        return $services_json->decode($arg);

jsonwrapper has its own home page on my site, so I get some credit for it that way. And besides, taking credit for helping old versions of PHP limp along isn't exactly rock star stuff.

But was it a wise choice on my part not to say "w00t first, copyright me, MIT license, have fun" in a piece of code that might, I don't know, wind up in Facebook's standard client library? Nope. Next time I'll be smarter.

boutell: (Default)
The Symfony plugins site has implemented a clever scheme of "I Use It" buttons. These let you share the fact that you're actually using a plugin, as long as you have a developer account, which anybody who's created or contributed to a plugin does.

If you're using any of my Symfony plugins, kindly vote for them. Thanks!
boutell: (Default)
We're doing a lot of groovy stuff lately.

Two big things and one little thing:

1. pkContextCMSPlugin is a content management system written as a Symfony plugin. It emphasizes editor-friendly context-sensitive features. Double-click on content to edit, create and rename pages via controls in the breadcrumb trail, reorder pages by dragging them in the side navigation... as a general rule, we don't believe in dumb, ugly, modal admin pages.

2. pkMediaPlugin is a repository for images, and a meta-repository for video. (The video is actually hosted by YouTube. We plan to add support for more back ends, since they pretty much all have sexy APIs now.) pkMediaPlugin isn't just for Symfony developers; you can set it up in a Symfony project and then utilize the media repository you've created from other sites. You can easily create an interface to allow users to select media, put them in a particular order, and return to your application with that information. Then talk to our JSON-based API to find out everything there is to know about those images and videos. Then request them in any size and cropping style... and if you request that same version again, automatically get it delivered instantly by Apache as a static file.

3. We use a Campfire chatroom to keep up with what's going on during the day, and we use lots of Subversion version control repositories. So I wrote svncampfire, which provides updates in the chatroom when commits are made to folders we are about. Across several repositories, not all of which we control. If you have similar needs, check it out.

Is all this stuff in PHP? Yes. Does that mean you can't use it if you're not a PHP person? Well, you certainly don't need to be a PHP expert to benefit from svncampfire. pkMediaPlugin has a documented, RESTful API that responds with simple JSON objects, so you can use that too provided you're willing to set up a symfony PHP project to host it. pkContextCMSPlugin is for PHP developers (and their lucky clients) right now, but we're in the process of working out some exciting strategies to bring it directly to front end developers, and even directly to folks who don't know HTML and CSS.
boutell: (Default)
Hey geeky monkeys,

I need a new programmer's editor. Or a band-aid for vim. I can't decide which.

I love vim so this is kind of a bummer.

Reasons for this change:

1. I need to copy and paste with the mouse naturally without the need to manually switch in and out of autoindent mode, etc.

2. I need tabs that work intelligently: if there is already a tab open for a file, it gets reused. I don't want to use a command to look for an existing tab followed by yet another command to open the file if it is not open. That's dum. But it seems to be the best you can do with vim 7's buffers and tabs. I also need to be able to browse for that file without pain.

3. My coding style has always been designed around an 80-column limit, so I break lines a lot, using indentation to carry meaning even within a single complex statement. Old-school programmers are of course used to this. But my coworkers have served notice that modern computers have wide screens and I need to knock it off with the multiline statements already. I'm not convinced they're right in all situations, but the 80-character rule is starting to look pretty arbitrary. I could address this with a fullscreen vim window of course, but for that to work well, tabs have to work well. See #2.

Things I have tried:

1. Eclipse. I tried Eclipse with the PDI PHP context sensitive stuff. It was awesome for a few days and then it refused to open a particular file ever again until I nuked the Eclipse project (not the actual code of course). This does NOT give me warm fuzzies.

2. jEdit. Fired this up today. Icons that make Mapedit look like iTunes. And no native support for tabs!

Things I'm aware of:

TextMate. My coworkers swear by this. Fabien Potencier of the Symfony project swears by this. It's neither open source nor free, but it's $50-ish, which is reasonable. Starting to look really good.

Emacs. I was going to roll my eyes at this but it looks like there are serious native-GUI emacs versions worth trying, with PHP modes, and I will check them out. When there's a GUI I want it to follow GUI conventions, so I'll get pretty annoyed pretty fast if it reinvents too many wheels there.

Thoughts from the peanut gallery?
boutell: (Default)
Band name: "symfony sync production go"
boutell: (Default)

In seriously geeky news, a new release of sfGuardPlugin appeared today. It includes my group filter stuff. Looks like Fabien ported it to Symfony 1.1 as well. Neat.

Those who have already overridden the heck out of generator.yml will want to add _groups to the filters: list in that file. Those who are using the stock generator.yml, or just adding additional stuff in the usual way, won't need to do a goldarn thing.

Speaking of seriously geeky, is anyone else out there in reading-Tom's-LJ-land actually using Symfony?

While we're at it, for those who know what such things are, what's your preferred web development framework?

I won't try to come up with a formal poll here because I'd just leave out stuff. Examples that spring immediately to mind: Symfony, Zend, Drupal, Joomla, Zope, Rails, Struts, Cake. Tell me all about the awesome stuff I forgot to mention.
boutell: (Default)
Image_JpegMarkerReader and Image_JpegXmpReader are out. These are PHP packages relating to JPEG images. Image_JpegXmpReader, which you may find useful, extracts the metadata (title, description, photographer credit, etc.) found in JPEGs produced by Photoshop and compatible programs (*). This is accomplished quickly and painlessly, without the need to load extensions written in C.

At P'unk Ave, we needed these capabilities to quickly and painlessly import photos into GPTMC Pressroom, a site we built for the Greater Philadelphia Tourism Marketing Corporation. The GPTMC has zillions of photos; their staff knows Photoshop. We could make them reenter all that metadata. But we don't, because we rock.

The Image_JpegMarkerReader package, which you are less likely to use directly, provides support for fetching individual "markers" from a JPEG file. A JPEG file is made up of two things: "markers," which contain descriptive information like XMP (among other things), and the actual image data. Most people won't need to deal with this package much, but perhaps you want to fetch something else from a JPEG marker, something Image_JpegXmpReader doesn't cover. In that case, Image_JpegMarkerReader is for you. You can subclass it just as I did or use it directly.

Soon you'll be able to install these packages with a single pear command, which will follow the dependencies and install both:

pear install Image_JpegXmpReader

For now, though, you'll get warnings that the package is in beta, an error message, and an explanation that you can manually install them like this:

pear install channel://
pear install channel://

This is all it takes to do what 99% of programmers will ever want from Image_JpegXmpReader. If the image has no XMP marker or just doesn't contain these particular fields, these methods return false. Yes, there are fancier methods provided for fancier cases.

require 'Image/JpegXmpReader.php';
try {
  $xmp = new Image_JpegXmpReader($file);
  // Photographer credit
  $creator = $xmp->getCreator();
  // Long description
  $description = $xmp->getDescription();
  // Caption
  $title = $xmp->getTitle();
/ // Returns an array of tags (like "shopping" or "education") that
  // have been assigned to this image
  $tags = $xmp->getSubjects();
} catch (Exception $e) {
  echo("Exception thrown while parsing $file, it's probably not a JPEG\n");
And that's pretty much all there is to it. Hope you find it useful!

You can find a somewhat fancier test program here.

(Note: both packages require PHP 5. If you are still running PHP 4, I'll start feeling sorry for you whenever I get through laughing at you. Unless it's at your employer's insistence, in which case I'll skip straight to feeling sorry for you.)

(*) Photoshop used to save this stuff in EXIF format, which PHP can handle "out of the box," but Adobe decided to move to XMP instead. For flexibility and XML-ness, I suppose.

Hey, cool

Jun. 28th, 2008 08:18 pm
boutell: (Default)
Fabien Potencier accepted my Symfony bugfix for the next bugfix release in the 1.0.x series after all.

(It was previously reclassified by someone other than Fabien, this was his first weigh-in on the subject.)

Knowing that the man most responsible for Symfony viewed the issue similarly gives me a good feeling about using it. Which is important, because we use it a lot.
boutell: (Default)
Kevin Van Zonneveld has created a library of popular PHP functions, translated to JavaScript.

I'm not sure it's a great idea to make one language look too much like another if, at the end of the day, they are not the same, because it might make it harder to avoid dumb mistakes. But regardless of how you feel about that, it's a neat library with a lot of useful stuff that's not standard equipment in JS.

I found it because I was googling around to see how one implements something like PHP's function_exists in PHP, and lo and behold, it's included in php.js.

One downside that I don't imagine Mr. Zonneveld is too worried about: it's likely to increase the number of people who hold a deep conviction that if they were somehow just a little better at JavaScript, they wouldn't need to know PHP (or any other server side language) at all. And yes, I do hear from those people regularly.

Fortunately they all figure it out sooner or later and cough up the $10/month for real hosting with, y'know, a database to put stuff in.

September 2014

2122232425 2627


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 23rd, 2017 11:26 pm
Powered by Dreamwidth Studios