Hackathon Project – FriendMozaic

I recently participated in the Facebook Hackathon at the University of Waterloo. The project I chose to work on was suggested by one of my teammates, who showed an app for a Facebook clone in China that uses thumbnails of your friends’ profile pictures to rebuild your profile picture. We called our app FriendMozaic.

You can find the code here: https://github.com/DouglasSherk/FriendMozaic

This project got an honorable mention, which I was happy with considering most of my team had to leave just a few hours in. The project that won was Ninjaquote, which to me was basically a clinic on how to do UI design and polish things that are relatively simple concepts at the core.

What it Does

Back to FriendMozaic, if you don’t understand what I mean by my explanation of what it does, here’s what the result of our app’s churning looks like:

This is a joke picture of me, which I used for most of my testing because of the decent contrast between different parts of the image. The user is given an option to select up to 500 of their friends to build a mosaic profile picture out of.

I don’t have any information on the China Facebook clone app anymore, but when I saw it, I was instantly sold on building it for Facebook. We decided that, to reduce load on the server, we would do as much of it as possible in canvas/JS as opposed to PHP/GD, which is what I automatically decided would be the best way to do it. Also, while I generally don’t like doing things just because they’re technically more difficult (instead preferring building something useful and ideally simpler), we decided that it was worth going out of our comfort zone and using canvas for the purposes of the hackathon.

I was the only one on the team who had any experience with canvas (mostly from writing conformance tests while I worked at Mozilla), so I ended up writing all of the canvas code, including pulling pictures from profiles and rendering them. This is what I spent most of my time on; stitching this code into the UI that my team built took a relatively short time afterwards.

Image Generation Algorithm

What I really want to focus on here, though, is the technical problems I ran into. Canvas is a relatively simple API, but there are some “gotcha”’s that are important to see and be able to respond to.

For starters, the algorithm used to actually figure out how to populate the squares of the picture was relatively simple. Basically, my algorithm consisted of the following:

  • Load the user’s profile picture.
  • Load the profile pictures of the user’s friends that they selected.
  • Redraw the canvas every time a batch of 5 friend profile pictures are successfully loaded. As part of this redrawing:
    • Divide the user’s profile picture into 10×10 segments.
      • For each 10×10 segment, get the average RGB value (i.e. add up the RGB values of each pixel, then divide by 10*10).
        • For each friend profile picture currently loaded (or all if loading is complete), compare average RGB value to the RGB value calculated
          in the parent step using a simple distance formula. Store this as the closest if it is.
      • Draw a friend’s profile picture which is the closest in RGB value to the 10×10 block.

That’s it! We considered adding stuff like tinting so that each 10×10 block looked closer to the original, but didn’t have time to do that.

Cross-Origin Resource Sharing

Despite this simplicity, there were some other problems. The main one, which took me almost 4 hours to solve, was caused by CORS (or rather the lack of support for it). CORS stands for “Cross-Origin Resource Sharing”, which is a technology that allows you to bypass a security feature in the DOM which prevents you from copying pixel data from domains other than yours. This is done because otherwise, scripts could grab arbitrary data from other sites while being authenticated as the browser user (since this is not run server-side). As far as I know, this has nothing to do with the WebGL/canvas
uninitialized memory exploit.

I ran into this stuff at Mozilla but never fully understood it. The nature of conformance tests is that they’re run entirely locally, so the errors I ran into there were the same, but were solved in a completely different way. This time I had to fully dig into the issue and figure out what was happening and how to deal with it. I was surprised by how little documentation or support there is on this issue. Maybe there’s just not a lot of people doing cross-origin canvas work.

The basic problem was that I was loading Facebook profile pictures from facebook.com, using instructions provided by friendmozaic.com, then reading the pixel data of these images, but the DOM security features prevented me from doing this. Fortunately, CORS provides a way around this. The basic setup of CORS is the following:

    • Load your image in the following way:Note the crossOrigin parameter.

 

  • Set up the server pointed to by http://crossdomain.com to give the following header: Access-Control-Allow-Origin: * You can also change the wildcard to a specific domain or set of domains. This can be set in a .htaccess or, for example, as part of PHP’s header() function.

 

 

More information can be found on this Mozilla blog post by my previous colleague and mentor, Benoit Jacob. So this looks fairly trivial, but unfortunately the second step was impossible in my case. Facebook doesn’t pass these headers as part of their content (despite the stuff I was grabbing being available without authentication), so I couldn’t use this method. The only other way I could think of was to just proxy the images.

Proxying Image Requests

What does that mean? I wrote a simple PHP script that takes in a URL parameter being the image requested, and the server would make that request then give it to the client as if it came from the server’s domain. Here’s what that looks like:

Please do not use this! It obviously requires much more authentication and validation before it can actually be put into production code.

With some polish, this is the method we currently use. It actually works fairly well, although it adds some heavy latency to the actual loading of the profile pictures. In general, it takes ~10 seconds for the mosaic to even begin compositing.

One last sort of technical problem! Facebook is terrible at exposing some basic functionality. For example, here’s a code fragment I found on Stack Overflow (I think) which scrapes the path to the user’s profile picture, i.e. the big version:

This could be a lot better, but I suppose it works.

Conclusion

Something I realized after I had built it in canvas was that the whole “re-render every 5 picture loads” mechanism would have been impossible or infeasible if I had done this in GD. Also, any interactivity, which I initially wanted to do, would have been much more difficult. It made me realize that I should have thought this through more at the beginning instead of just diving in. Fortunately, I made the right decision, despite initial reservations.

That’s it for now. I’m still tossing up whether or not I should continue this project. If I do, I’ll probably rewrite the whole UI and most of the canvas code. I noticed a non-mispelled version of our project already exists at http://friendmosaic.com, which I actually didn’t find until I registered friendmozaic.com (I was tired and actually thought that was the correct spelling). Fortunately, these guys seem to only work with Twitter.

Leave a Reply

Your email address will not be published. Required fields are marked *