Sandstorm – personal cloud, self-organzing cluster

I’ve heard a lot of Meteor news lately, but somehow I missed Sandstorm. Your own personal cloud. Install services easier than installing apps on your phone. Add machines and they self-organize into a cluster. This sounds just way too awesome. Looks like they use Meteor heavily. Jade Wang (formerly of the Meteor Development Group) is a co-founder.

Apps must be packaged for Sandstorm (made into “grains”). The list of ported apps is pretty inspiring. Included are: draw.io, LibreBoard, HackerSlides, Let’s Chat, Paperwork… All were new to me, several are written in Meteor, and I was able to check out all of these in seconds. I’m hooked.

how to upgrade MongoDB 2.6 to 3.x on Ubuntu

sudo mv /etc/apt/sources.list.d/mongodb* /tmp/
echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update && sudo apt-get install -y mongodb-org

And I also had to do fix my replica set in the MongoDB shell (necessary for Meteor oplog tailing):

var a = {"_id" : "rs0", "version" : 1,"members" : [{"_id" : 1, "host" : "localhost:27017"}]};
rs.reconfig(a, {force:true});

Oplog: a tail of wonder and woe

First, your TL;DR:

  1. Stress test your Meteor app.
  2. Oplog tailing may be less efficient for particular workloads.

Background

My work involves using crowdsourcing to assess and improve technical skill. We’re focusing on improving basic technical skills of surgeons first because—no surprise here—it matters. A more skilled surgeon means patients with less complications. Being healthy, not dying. Good stuff.

One way we gather data is a survey app where crowdworkers watch a short video of real human surgery and answer simple questions about what they saw. For example:

  • Do both hands work well together?
  • Is the surgeon efficient?
  • Are they rough or gentle?

Turns out the crowd nails this! Think of it this way: most anyone can recognize standout performers on the basketball court or a playing a piano, even if they’re not an expert at either. Minimal training and this “gut feel” are all we need to objectively measure basic technical skill.

Meteor

So, a survey app. Watch a video, answer a few questions. Pretty straightforward. We built one in-house. Meteor was a great choice here. Rapid development, easy deployment, JavaScript everywhere, decent Node.js stack out of the box, all that.

And of course we used oplog tailing right from the start because much of what read about oplog tailing made it sound like it was the only way to go. Sure, you’ll want oplog tailing for realtime (<10sec delayed) data when you have multiple apps connecting to the same MongoDB database. But if you don’t need that, you may not need it at all, and you may not want it.

Traffic pattern

Our traffic is very bursty. We publish a HIT on Amazon Mechanical Turk. Within minutes, the crowd is upon our survey app. Our app generally does fine, but folks complained of very slow survey completion times when we started hitting somewhere around 80 DDP(?) sessions in Kadira. Each DDP session in our survey app should equate to one simultaneous active user (hereafter “user”).

Here’s what we want to know:

  1. Why does our app slow down when it does?
  2. Can it scale [linearly]?
  3. Are there any small code or configuration changes we could do to get a lot more performance out of the existing hardware?

Spoilers:

  1. Meteor pegs the CPU when oplog tailing is enabled.
  2. Yes, if we disable oplog tailing.
  3. Yes, disabling oplog tailing and clustering our app.

Stress test

We created a stress test to get a better feel for the performance characteristics of our app.

The test uses nightwatch to emulate a turker completing a survey. Load the survey app, click radio buttons, enter comments, and throw in a few random waits. Many threads of the nightwatch test are spawned and charge on in parallel. The machine running nightwatch needs to be pretty beefy. I preferred a browser-based stress test because I noticed client-server interactions amplified the amount and frequency of DDP traffic (hello Mr. Reactivity). It was also easier to write and run nightwatch then pick the exact DDP traffic to send.

Notes on our app:

  • We use mup to deploy to Ubuntu EC2 servers on AWS.
  • Tested configuration uses one mup-deployed Meteor app.
  • The app connects to a local MongoDB server running a standalone one-member replica set (just to get the oplog).
    • I also tested with Modulus, scaled to one 512mb servo in us-east-1a. Non-enterprise Modulus runs with oplog tailing disabled, and the app connects to MongoDB on a host other than localhost.
  • Our app uses iron:router.
  • Our app doesn’t need to be reactive. Surveyees work in isolation. But this is how we wrote the app, so that’s what I tested.

Results

I ran a series of stress tests. Ramp up traffic, capture metrics, change code and/or server configuration, repeat. Here are the results.

Takeaways:

  • Each row in the spreadsheet represents one test.
  • Every test ran for 5 minutes.
  • When one “user” completes a survey, another one begins (so the number of users is kept more or less constant during evey 5-minute test).
  • There are lots of notes and Kadira screenshots in the results spreadsheet. For the Kadira screenshots, the relevant data is on the rightmost side of the graphs.
  • I think Kadira session counts are high. Maybe it isn’t counting disconnects, maybe DDP timeouts take a while, or maybe the nightwatch test disconnects slowly.
  • Row 3. At 40 users, the CPU is pegged. Add any more users and it takes too long for them to complete a survey.
  • Row 5. Notice how doubling the cores does not double the number of test passes (less than linear scaling along this dimension).
  • Row 6. Ouch, we’re really not scaling! Might need to investigate the efficiency of meteorhacks:cluster.
  • Row 7. Oplog tailing is disabled for this and all future tests. MongoDB CPU load is roughly doubled from the 40-user, 1-core, oplog-tailing-enabled test.
  • Row 9. Too much for one core: 6.5% of the tests failed.
  • Row 11. This is what we want to see! 2x cores, 2x users, 2x passes. In other words, with oplog tailing disabled and double the number of cores, we supported double the number of users and doubled test passes.
  • I should have also tested 160 users, 4 cores, oplog disabled. I didn’t. Live with it.
  • Disabling oplog tailing seemed to allow the processing load to shift more to MongoDB. MongoDB appeared to be able to handle same more… gracefully.
  • I didn’t get very far with Modulus. I’m very interested in their offering, but I just couldn’t get users (test runs) through our app fast enough to make further testing worthwhile.
  • A DNS issue prevented capturing Kadira status while running on Modulus.
  • cluster lives up to its promise—adding cores and spreading load.
  • I don’t think we’re moving much data, but any reactivity comes at a price at scale (even our so far little bitty scale).
  • Our survey app could and should be modified to use much less reactivity since, as I mentioned earlier, it is unnecessary.

Server-side profiles

This is somewhat of an addendum, but I figured it might be useful.

Here’s what the Meteor Node.js process does when 10 users hitting our survey app running on one core.

Oplog tailing enabled:

Pie chart server profile with oplog<br /><br /><br />
tailing

Oplog tailing disabled:

Pie chart server profile without oplog<br /><br /><br />
tailing

Takeaways:

  • Note that these pie charts only show %CPU usage. CPU and network are the primary resources our app uses, so this is fine.
  • The profile data for each slice (when you drill down) are very low-level. It’s hard to make any quick conclusions (or I just need more practice reading these).
  • When oplog tailing is enabled, the Cursor.fetch slice is about twice as big, and none of the methods causing that CPU load are ours. Perhaps this is the oplog “tailing” in action?
  • When oplog taling is disabled, drilling into Cursor.fetch shows us exactly what specific methods of ours are causing CPU load. Even if oplog tailing is more efficient, this level of introspection was priceless. We need this until we learn to better debug patterns in our code that lead to more CPU when oplog tailing is enabled.
  • The giant ~30% slice of “Other” is a bit of a bummer. What’s going on in there? Low-level/native Node.js operations like the MongoDB driver doing its thing? Sending/receiving network traffic?
  • Kadira monitoring isn’t free CPU-wise, but it is worth it.
  • What should these pie charts look like in a well-optimized application under load? Perhaps the biggest slice should belong to “Other”?

Further reading:

Feedback/questions/comments/corrections welcome! I’d espeically love to hear about your experiences scaling Meteor.

Looking for Online Video Cropping, Blurring, Hosting

Does anyone know of an online video editing service where I can upload my videos and edit them? This is for a work project. Here are the basic features I need:

  • crop – remove first x minutes, last y minutes
  • blur – faces, logos, etc
  • video storage
  • video streaming
  • encrypted, private

YouTube video editing comes really close, but I need more fine-grained control over blurring. I also need better licensing. Currently the only license and rights ownership choices are “Standard YouTube License” (they own it) or “Creative Commons – Attribution”. I want “Adam’s Private/Personal Video License”.

Adobe Premiere Express might have been perfect, were it not mothballed. Sounds like an online version of Adobe Premier.

Yes, You Should Swap

If you’ve ever set up a machine by hand, you’ve probably had to decide how much of your disk to set aside as swap.

I’ve often wondered “why swap at all”? This quote by Nick Piggin from 2004 finally helped me answer the question.

no matter how much ram you have, swap can increase performance by allowing unused anonymous memory to be paged out, thereby increasing your maximum effective RAM

Found via this post on Hacker News, where the poster raises the point that some filesystem buffers might be extremely “hot” (frequently used), but might only fit in physical RAM (where they should be) if some swap space is available to page out other “cold” information.

The Most Awesomest Software Project Ever

  • Virtue abounds. Everyone on the team really really cares about doing a great job.
  • Awesome documentation, and clear/sensible standards around same.
  • Easy to test. Easier to test than not test.
  • Easy to code. New developers can get code, run apps/services, change/commit code their first day on the job.
  • Extant coding standards.
  • Extant tool to lint code (ensure conformance to coding standards). Integrated with source control so you can spot lint errors before you commit.
  • All tools work with any dev setup–they are not tied to any operating system nor IDE.
  • Distributed, modern source control.
  • Continuous integration. Every commit on every branch is automatically tested.
  • Issue tracking.
  • Integrated source control, continuous integration, and issue tracking.
  • Well-crafted, useful automated tests.

Top Five Rules of Email Etiquette

These are my email etiquette rules. I’d love to hear your own email etiquette rules, too.

Please call me if I fail to follow any of these simple rules. Literally. Like, pick up the phone and call me. Or tell me in person.

  1. Always reply to a personal email written to you.
  2. If several questions are asked in an email, address them all.
  3. Do not allow the body of your email to be smaller than your signature.
  4. Care. For instance, use proper grammar, spelling, sentence structure and capitalization.
  5. (reserved)

There are of course exceptions to the first rule, such as when the email is ending a conversation. It may help to consider not replying akin to ignoring someone trying to talk with you.

Three steps to invest in Bitcoin now

Disclaimer: I know approximately jack squat about speculative investing, and only slightly more about how Bitcoin works. Be prepared to lose.

If you decide to buy one (or some) Bitcoin as an investment, here’s my (USA-specific) advice:

  1. Buy Bitcoin via Coinbase. It’s easy and works with a standard USA checking account. Write down your purchase details for tax purposes.
  2. Don’t leave your Bitcoin at Coinbase. Create a paper wallet and send your Bitcoin to that.
  3. Lock the paper wallet away in a safety deposit box and forget about it.

I’m not sure how to properly pluralize “Bitcoin” so I didn’t bother ever adding an “s” in this post.

Last of all, consider actually using Bitcoin, instead of just investing! It’s complicated. It’s scary. You will be challenged. You’ll learn about some interesting subjects, and you’ll be on the forefront of what is surely a revolution in currencies.

How to really trade a Casascius physical Bitcoin for U.S. dollars

It appears Coinbase paper wallet private keys are in a proprietary format, so my “easy” method won’t work. Here are instructions to really trade your Casascius physical Bitcoin for U.S. dollars.

  1. Obtain the private key. Carefully remove the hologram sticker from the back of the physical coin. A bunch of letters and numbers are printed on the back (example). Those letters and numbers comprise the “private key” for a Bitcoin wallet containing some amount of Bitcoin(s). Whomever possesses this private key may send any fraction (as little as 0.00000001) of the wallet value to another wallet.
  2. Create a wallet on Blockchainhttps://blockchain.info/wallet/new
  3. Import the Bitcoin to Blockchain. In your Blockchain wallet, click “Import / Export”, then paste the mini private key from your Casascius physical Bitcoin under “Import Private Key” and click “Add Private Key”.
  4. Sign up on Coinbase. Use this referral link and I’ll get $5 after you exchange 1 Bitcoin.
  5. Send the Bitcoin to Coinbase. On Coinbase click “Send/Request”, then “Request Money”. Leave the form blank and just copy the address. In your Blockchain wallet click “Send Money” and use the address you just copied.
  6. Sell your Bitcoin for USD. This part is pretty straightforward. Click on Buy/Sell to sell your Bitcoin and transfer the USD to your bank account.