Monday, July 25, 2016

Music Theory, Sound Science and Digital Audio

This isn't the demo.
This is just a screenshot.
I've been working on some JavaScript music code for a possible future project.  So far I've learned a few interesting things about music theory, sound, and digital audio, and I've reached a milestone so here's Keyboard Demo, a little electronic keyboard demo.

Here are some things I needed to learn to get this far:

Generating audio with FFmpeg


I wanted to be able to play a range of pitches; the standard 88-key piano keyboard runs from C0 to C8, so at least those.  I also wanted to support IE.  There don't seem to be a lot of good baked-in options for that sort of thing, so I knew the lowest common denominator would be to have a bunch of tiny audio files, one for each pitch.

I considered generating all the pitches with MIDI in Anvil Studio, recording with Audacity, and then cutting that audio up into individual files, but that would have taken forever.  After poking around, I discovered that you can generate audio in batch using FFmpeg.

FFmpeg is a very useful tool for audio and video manipulation.  It can transcode audio and video files, change sample- and frame- rates, apply filters, crossfade, and probably hundreds of other tricks I don't even know about.  I didn't know it could generate audio (or video) until I ran into this superuser.com question.  The idea is to use ffmpeg's filtergraph feature as an input, with a command line like this:

ffmpeg -f lavfi -i "<filtergraph string>" <output>

FFmpeg's filtergraph strings are pretty complicated, but making a sine wave is pretty easy:

sine=frequency=440:duration=1

Yup.  That's a sine wave.

A quick python script to generate 88 calls to FFmpeg and now I've got all the pitches I need as individual m4a files (IE won't play ogg, sadly).  There's some interesting music theory in just the choice of frequencies, of which I'll give a taste here:

I generated pitches using equal temperatment, meaning that each pair of adjacent pitches has the same frequency ratio.  As temperaments go, this is the one that all the cool kids are using.  It beats out well and meantone temperament, and generally makes sense if you want to play in multiple keys using the same set of pitches.  Apparently it produces some slightly impure intervals, though I haven't yet compared it to just intionation with my own ear.

Then I based the frequencies around the notion that A4 (the A above middle C) is 440 hertz.  That's called concert pitch, or at least that's what we call it lately, and it's not the only pitch reference out there.  It's nice that it's a round number like that - makes it easier to remember. I've heard it said that music played in different keys can "feel" different, as though simply transposing a piece upward a semitone can turn it from melancholy to hopeful, or something.  But the fact that the exact frequency of a pitch has changes through the ages, and that different ensembles might choose something other than the standard, makes me a little skeptical.  Some people have perfect pitch, of course, but I don't know how that works with respect to different pitch references.  I guess both the feeling-of-the-key and perfect-pitch concepts are relative to contemporary practice.  Anyway, 440.

My ears


The range of frequencies that humans can hear is (very roughly) 20 Hz to 20 kHz, which are nice round numbers that should make you suspicious about the error bars around them.  I don't know what the standard deviation is, but I was surprised when I went to listen to those audio files I made.  I couldn't hear most of the low octave (C1-A1, ~32-55Hz)!  I was sure that I had made a mistake in my script.  I've played an 88-key piano before and while my memory, like my pitch, isn't perfect, I seem to recall being able to hear those low notes.

I ran a highly scientific test using this YouTube video, and sure enough, my ears don't kick in until around 55 Hz.  I didn't get to check on the high end of my range because the dogs started barking and I can't blame them because those are some annoying frequencies up there.

So what's going on?  Well, as you know, a piano doesn't sound like a sine wave.  It produces a crazy mess of noise that goes well beyond the fundamental frequency.  Some of the biggest frequencies in the spectrum are multiples of the fundamental frequency (called "harmonics"), but also the whole audible range is spattered with low-amplitude impurities, and all that stuff together makes the timbre of the note.  That's why those frequency spectrum visualizers are never very satisfying to me - it's all a mess, all the time.  I guess it turns out that when I play C1 I'm not usually hearing the fundamental frequency at all (although more volume can help), but my brain is putting the harmonics and other noice together and I end up with a good approximation.

Adding harmonics


So I went back to those sound files and added some harmonics.  This stackoverflow answer has some numbers in it for relative frequencies of harmonics for a piano, supposedly.  I don't know where they got those numbers from, but I figured it was better than just guessing.  I plugged those into my script and generated some more complicated FFmpeg filtergraphs.  For example, heres A4 with just the first three harmonics:

sine=frequency=440.00000:duration=1[s1];
[s1]volume=volume=3[i1];
sine=frequency=880.00000:duration=1[s2];
[s2]volume=volume=1.197[i2];
sine=frequency=1320.00000:duration=1[s3];
[s3]volume=volume=0.897[i3];
[i1][i2][i3]amix=inputs=3

The actual filtergraph I used was a little more complex: more harmonics, plus I added a delay to each harmonic so that I didn't end up with a sawtooth-like shape.

Now I can hear the low notes.  Plus, the notes are a little less annoying.  It's interesting that adding frequencies other than the one I want to play - impure, dirty, frequencies - ones that don't belong - actually make the sound more real.

Aliasing


Hey wait a minute.  I can hear the low octave now, just fine, but the high octave sounds just terrible.  The notes I hear don't even seem related to the ones I'm asking for.  Luckily, I remembered something from college about the Nyquist frequency (so I guess it wasn't a total waste).  I had picked a sample rate of 16 kHz, because I wanted to save bandwidth and it made the audio files nice and small.  Sure enough, if your sample rate is less than twice some frequency you'd like to hear then sorry, that's just not gonna happen.  Instead you're going to hear a bizzaro mirror-world frequency, and you probably won't like it.

To me, the best analogy is to that old experiment where you look at a fan in a strobe light: at a certain strobe frequency the fan appears to stop spinning, and then if you strobe faster the fan seems to rotate backwards.  It's not a perfect analogy, but basically this is the kind of crazy stuff that happens when analog meets digital.  I probably could have fixed it by filtering out the high frequencies or increasing my sample rate, but these audio files are just for IE users.  I can just drop those pitches from the range.  The IE users probably won't miss them.  What are the non-IE users going to get?

The Web Audio API


Modern browsers have the Web Audio API built-in, and with that I don't need any silly audio files laying about.  I can generate my noises in real time!  It's as simple as this:

    var context = new AudioContext();
    var oscillator = context.createOscillator();
    oscillator.frequency.value = 440;
    oscillator.connect(context.destination);
    oscillator.start();

That's produces a pure A4 sine wave.  Add more oscillators for the harmonics, watch out for aliasing, put a "gain" node in there because the default oscillator is super loud, and we're in business.  There are some cool tricks like amplitude envelopes that I could use to make better or crazier sounds (see this web synthesizer site for instance).  Really, the engineering of synthesizers is an enormously deep rabbit-hole, and I'm tempted to jump in, but this will do for now.

Volume


Another thing about sounds:  Loudness.  I used to think it was mostly about how far up and down the waveform moved.  Nope.  It's also not about the rate-of-change of the waveform at any given point.  If you're thinking about amplitude of sine waves, that's closer, but still not right.

Uh-uh.
Nope.
From what I now understand, it's a combination of the amount of energy in the wave over a certain span of time, and the sensitivity of the human ear to the frequencies involved.  A big spike in the wave means the speaker (for example) has to push a bunch of air molecules really hard, but it has to keep working on that air for a while before it will affect how loud we think the sound is.  Plus, even a very energetic sound may be very quiet if it's near the edges of the range of our hearing, like those low-octave pure-sine-wave notes I couldn't hear.  Loudness is a messy concept, and, when it comes to commercial music, thank goodness for ReplayGain.

In my demo you'll notice that some of the notes are louder than others, due partly to the fact that I didn't vary the oscillator amplitudes as the pitch went up, but also due to a phenomenon represented well by the Fletcher-Munson curves, which show that some frequencies just sound louder than others even when the "sound pressure level" is the same.

The interface


The interface for this demo is pretty simple:  Click and hold a key to play it.  If you want to get fancy, you can hold down shift and click different notes to make chords.  I took a tape ruler to the little electric keyboard I have for some measurements and then used a little dynamic SVG to generate the keys.  There's a little dot on middle C just to get you oriented.  That's about it.  I have some bigger plans for this code, so stay... tuned.  See what I did there?  Tuned?  It's.. oh, ok.  Here's the link again:

Monday, July 18, 2016

It's EELectric!

Do you like eel-based puzzle games?  Ready for some mind-bending diagonal-moving action?  No?  Nevertheless, let me introduce you to It's EELectric!, an original puzzle game I've been working on.

In this game you star as an apex predator, the deadly Electrophorus electricus, better known as the electric eel. This is not a realistic eel simulation. In fact, I looked it up: electric eels don't look like this thing at all.

Plus, I don't think there starfish in the same... you know what, just swim with it.  Your job is to eat all the fish on the level, but before you chow down on the unsuspecting prey you have to kill it with your electric shock attack.  And you're hungry and getting hungrier, so you'll have to be careful not to waste any moves.  Otherwise you'll perish and have to press R to go back in time to try again.  I already mentioned it's not realistic.  SHOCKING.

Enjoy 37 fiendfishly difficult pools of funderwater puzzling!  Sound sea-ffects?  It's um, reel.. something.  Hmm.

Anyway here it is:

Wednesday, May 25, 2016

Callooh Callay, World!

I wrote some puzzles for the 2014 MIT Mystery Hunt.  For one of them, Callooh Callay, World!, I invented a new esoteric programming language called Wonderlang.


It's frabjous.

I imagine this is how one would program computers in Wonderland, if there were any.  It's not meant to be particularly difficult to use, but it has a different perspective.  It requires a few characters not found in the ASCII character set: I suspect keyboards in Wonderland would have a great deal more keys than ours.

Wonderlang hasn't made it onto esolangs.org (yet).  To be fair, I haven't written a proper specification for it, which seems likely to be a requirement.  But then that's probably another natural characteristic of programming in Wonderland - why should they need to write this stuff down, or if they did, despite needing to, why should they go and actually do it?  After all, they're all mad there.


Wednesday, May 11, 2016

Hexfold

Here is HEXFOLD, a web-based puzzle game inspired by the puzzle toy Cool Circuits.  For some background, see my post analyzing Cool Circuits, and the one describing how to extend Cool Circuits to a hexagonal board.

Level 7: "Lucid Sitar"
TL;DR: The object of the game is to place the pieces on the board to create a loop, using every piece and obeying certain constraints that are different for each level.  I made 36 levels (the 37th and final one is randomly generated) which more or less increase in difficulty from hand-holding tutorial to absurdly hard.

I should warn you, this isn't an easy, kid game like Robot Quest or a simple, casual game like Logoplex.  Oh, sure, it starts out easy enough.  But that doesn't last.

If you give it a try, please provide feedback in the comments below.

Before you ask: the level names are just arbitrary adjective-noun combinations.  I generated a bunch using The Sillifier, and picked out the ones that amused me.

HEXFOLD (Web)

It's also available as a Chome App:

HEXFOLD (Chrome Store)

Thursday, May 5, 2016

Cool Circuits: Hexagons

If you haven't read my previous post about Cool Circuits, better go read that first.

What if Cool Circuits was made with a hexagonal grid instead of a square one?  More precisely, what if the turns in each piece were 60 degrees instead of 90?

We'll use the same pattern for generating the pieces: Each piece is a path composed of five turns either clockwise or counterclockwise, and each piece is uniquely described by a sequence of four bits, where each bit indicates whether the next turn is in a different direction than the previous turn.  Here's what the pieces would look like:

"JUVENILSQ"
Compared to the 90-degree pieces, these new ones are a little more stretched out.  If you've been paying attention you'll notice there's a new one.  The last piece in that image corresponds to the bit sequence 0001, which wasn't possible in the original Cool Circuits because it would have intersected itself.  I'm going to call it "Q".

Can we make a circuits with all of these pieces?  Nope.  In 90-degree-land, each piece contributed a single 90-degree turn (either way) to the overall path, and since there were an even number of pieces that meant we could end up with the complete 360-degree turn necessary for a circuit.  But with these new pieces, 6 of them contribute a 60-degree turn, and 3 of them contribute a 180-degree turn.  That means if we use all the pieces we'll always end up 180 degrees off. Here's a tabular summary:

LetterDescriptionFlippedContribution
J00111100180
U1001(same)±60
V10111101±60
E0110(same)180
N01011010±60
I1111(same)±60
L01111110±60
S00100100±60
Q00011000180

So we'll have to drop at least one piece.  For now, just to be true to the original puzzle, we're going to have to let Q go.  Sorry Q.  Maybe we'll find a job for you later.

Behold!  Hexagons!
Because the turns are 60 degrees, the pegs on the board will be arranged in a hexagonal grid.  Clearly, the overall shape of the board should also be a hexagon, but what size?  Five seems like a nice diameter - it would have almost the same number of points as the original puzzle.  I modified my solver from the original puzzle for these new parameters, kicked it off, and sat back to watch the solutions roll in.  Sadly, there are none.

Cool Circuits has a neat property that every solution leaves behind 6 untouched pegs.  In "Cool Circuits, Jr" they add 6 rings that you can use to wrap those pegs.  Once you've done that, there is no position left untouched.  This hexagonal version doesn't have that feature.  For each turn you make, you make it impossible for the path to later hit the point you didn't turn towards.  Near the edges, that can cascade and make other points untouchable as well.  I suspect this is a clue to why there aren't any solutions in the 5-board.

Before we give up on this board though, what if we had even fewer pieces?  I tried dropping J and E (leaving us with only pieces that contribute 60 degree turns), and I found that there are solutions to that variant.  Exactly four of them (ignoring solutions that are equivalent via rotation and reflection).  Here they are:

Four unique solutions for the 5-board, using the UVNILS pieces.
Note how the V and N pieces can be swapped (and flipped and rotated) to convert one solution into another.  That's also true in some of the solutions to Cool Circuits, though in both cases it depends on how tightly-packed the circuit is.

To me, this doesn't seem like enough solutions for an interesting puzzle game.  Sure, with rotation and reflection we could make more circuits, but they would all end up feeling the same.

Time to raise the diameter to 7.  You can probably guess going into this one that there will be too much space on the board.  Going back to 8 pieces, I let the solver run overnight, and it says there are 1503 circuits.

Alternative boards.
So with lots of space (A) we have lots of solutions.  I also considered removing the six outer-most pegs (B: 474 solutions) and removing four pegs from each of three edges, forming a somewhat triangular board (C: 130 solutions), but the bigger hexagonal board seems more thematic to me.  The large number of solutions does mean that we'll need to have plenty of constraints in each puzzle to make it uniquely solvable.

Overall, while not as elegant, I think this configuration works pretty well as a variant of Cool Circuits.  I'm working on a web-based playable version called HEXFOLD, which I'll post here once I've finished the levels and had some people play-test it.

Monday, April 25, 2016

Cool Circuits

I came across a particular kind of puzzle, the design of which is really fascinating to me.

I wanted to get my son some electronics toys for his last birthday.  I had one of those 100-in-1 electronics kits when I was a kid but he's still a bit young for the complex wiring, or at least that's what I thought.  I ended up picking Snap Circuits, which he loves.  I had to pick one more thing because you never know with kids, it's safer to get a couple of smaller things than one big one, so I tossed in Cool Circuits, because it had a similar name, and I think Amazon suggested it or something.  He also loved that one.  It's not really about electronics, it's just a puzzle game, but I've come to realize it's a very cleverly-made one.
A cool circuit in progress.

Here's how it works: the board has 25 pegs, arranged in a kind of diagonal grid.  There are 8 serpentine pieces which can fit in between the pegs, and each has a unique shape.  For each puzzle you are given some constraints (e.g. part of the circuit must travel along this path) and you must put the pieces on the board in a way that meets the constraints.  The one universal constraint is that the pieces must form a single loop.  In the toy, that triggers some mechanism which lights up the board and plays some music, but I've found that feature to be pretty finicky.

It's pretty tough to make a circuit from scratch, so the constraints tend to help more than hurt, and the more of them there are the easier the puzzle.

At first I thought the shapes of the pieces were arbitrary, but there's a mathematical beauty and completeness to them that took some pondering to discover.

As a font, it would not be easy to read.
Each piece is made of a sequence of five quarter-circles.  The manual describes them as "J", "U", "V", "E", "N", "I", "L", and "S".  So if you're looking at a given piece, starting from one end, you might describe it by listing the chirality of each segment.  For instance the "V" piece would be: turn right, left, left, right, left.  Or maybe clockwise, counter-clockwise, etc..  But that's a terrible description because the same piece would be described differently if you started from the other end, or flipped it over.

Math!
We can normalize the description for flipping by listing not the turns themselves, but the changes.  So if it starts out in one direction, we just say whether the next turn is the same as the previous one or different.  The "V" piece would then be different-same-different-different.  Now imagine same as 0 and different as 1 (e.g. 1011).  Each piece is described by four bits.  But some pieces have a different description if we start from the other end.  To correct for that, for each piece we'll pick the description that has the smallest number when treated as binary.

Here are the eight pieces:

LetterDescriptionFlipped
J00111100
U1001(same)
V10111101
E0110(same)
N01011010
I1111(same)
L01111110
S00100100

In this scheme, you can't have three zeros in a row, because that would make the piece intersect itself.  So 0000, 0000, and 1000 are out.  That means every workable bit sequence is represented.  These aren't arbitrary shapes at all, it's all possible unique shapes of length 5!  At length 4, there are 6 possible pieces.  At length 6, there would be 15.

All possible locations for the first piece.
Testing just the blue ones is enough.
So how many circuits are there? Let's start with an upper bound. For now let's say the first piece we put down can go in any point, in any direction, flipped or not. There are 224 possible first points (red and blue arrows in the diagram to the right).  However, some of those are equivalent to one another, because we could just rotate or "flip" the board itself.  So we'll cut that down to 28 (blue arrows only).

Suppose each piece can fit two ways at one of the blue arrows.  Actually, some pieces fit fewer ways than that, and sometimes they won't fit at all, but let's not work out those details yet.  After that first piece, we choose a next piece and fit it on in one of four ways.  This overestimated upper bound says there are fewer than 28×8!×2×47 paths (36,993,761,280). That's pretty high.  Luckily, many of the paths terminate quickly, either running off the board or crossing and earlier section of the path, so we're not going to need to search them all just to find the complete circuits.

I wrote a program to brute force all the paths.  It found 17,473 paths, starting from those blue-arrow positions, that used all 8 pieces without self-intersection or running off the board.  Of those, 1,520 ended up where they started, forming (possibly cool) circuits.

We're overcounting with that number though, because for any given unique solution, we're seeing it 16 different ways (counting reflection and cycling through the 8 pieces as starting pieces).  Now I very well may have made a mistake (or several) here, but I think the final answer, the number of unique solutions to this puzzle, the number of Cool Circuits, is 95 (no, see UPDATE below).


Solver Source Code

UPDATE 4/27: I found a mistake.  I didn't account for some solutions that fit so tightly that they leave four empty pegs all along one side of the board.  Those solutions are double counted in my initial number because they can be translated into another solution.  I was accounting for rotation and reflection, but not translation.  There are apparently 11 solutions like that, so my final answer (today) is 84.

Monday, April 18, 2016

Name Chains

Some people's last names sound like first names, such as Larry David.  Some have first names that sound like last names, like Taylor Swift.  And some have both, like Campbell Scott and Wallace Shawn.

FWIW
I downloaded a list of names of actresses and actors from IMDB and did some messing around with the data.  At first, I wanted to create chains of names, where the first name of each person matched the last name of the one that came before.  I think I'd get even better results if I could just extend my list to celebrities in general (not just IMDB actors and actresses), but so far I haven't found any better lists.

On second thought, I'll take the check.
There are a 3.6 million people in these lists, and most of them are quite obscure.  I'm not great with knowing celebrity names, but I think even the most die-hard movie memorizer would have trouble recognizing the names of all the "Documentary Couple" people in When Harry Met Sally, one of whom was apparently Peter Pan, and all of whom are in this database.  Plus - and I feel bad about this - the likelihood that I'll recognize an actor or actress by name decreases the longer it has been since that person graced the screen.  I figure recognizability is important in the kind of name chains I want to make, because that's what makes the chain interesting when compared, say, to a randomly generated list of names.

But how can I consider only the recognizable thespians?  I don't know a good way to do it, so I picked a bad way.  I created a "credit score" for each person, which does not at all reflect their quality as a performer, and honestly it's a pretty bad measure of fame, too.  But it was something.  It goes like this: for each credit they have in a TV episode or movie released in or after 1970, they get a number of points (1 point for a TV episode, 5 points for a direct-to-video movie, 10 points for a made-for-TV movie, and 20 points for a... standard(?)... movie).  Anyone with 30 or fewer points was discarded.  Maybe I could have linked their performances with ratings, orcounted how far down the cast list they were, or discounted multiple appearances on the same TV show, but I didn't. I did drop everyone for whom I couldn't parse out a first and last name, and people with the 'Jr.' suffix, because they won't fit well in a chain.

"I'm gonna play it safe. I'll wager $0."
Most of the top credit scores are porn stars, simply because they make a lot of movies.  I'll let you guess who tops that list.  Yes, that's right.  Skipping those, the top score goes to Shakti Kapoor (12423), a prolific Bollywood actor.  Skipping Bollywood as well, we get to Frank Welker (10544), a prolific voice actor.  Skipping voice actors too... Alex Trebek (8144), whose score was bumped by thousands of episodes of "Jeopardy!".  It goes on like this for a while.  Like I said, it's not a great algorithm.

Now, with only around 428,000 names, and a slightly better (but still terrible) chance of each name being known by an average reader, let's look at some numbers.  Here are the basic high-fives:

Top Five First Names, Actors
Michael4,420
David4,223
John4,030
Robert2,480
Paul2,217
Top Five First Names, Actresses
Anna1,110
Sarah1,105
Jennifer1,075
Maria981
Laura916
Top Five Last Names
Smith1,357
Lee1,249
Jones1,016
Johnson1,000
Williams979
For chaining, those would be great names to see switched.  Somebody named Smith Michael would be very handy.  Let's call his name "inverted".  Sadly, there's nobody with that name, but there are plenty of other inverted names.  I gave everyone an "inversion score": (F_L * L_F) / (F_F * L_L), where for instance F_L is the number of people whose last name matches your first name.  Low inversion scores include Jennifer Lopez (0.000004) and Chris Williams (0.000005).  Top score goes to King Jeff (4,884), but others include Rooney Mara (455) and Pruitt Vince (235), whose names definitely sound backward to me.

In this data set, there are around 285,000 "dead ends" - people whose last name matches nobody's first name.  Reedus, Gurira, or Cudlitz, for instance.  That means if I picked matching names randomly, there's a 2 in 3 chance at each link that the chain will end there.  If I picked each name optimizing for how many choices I'll have after that, it's going to be a very boring chain, full of Michael John, John Michael, Michael Paul, Paul John, John Paul, Paul Michael, and so on.  All real people, apparently.  It turns out this is a hint about how many branches there are in this system, and why my next idea won't work.

I opted to avoid a centipede metaphor.
I tried to automate chain-generation.  I wrote a program (hereafter abbreviated as IWAP) that searches for the top-scoring chains/cycles of a given length.  2-cycles are interesting because that's basically two people whose names are switched.  My favorite among those is Keith David/David Keith.  The problem is that the program takes too long for lengths greater than 2.  There are a ton of possibilities, and even if I just pick a few random ones, that's not really what I'm looking for - I want good ones.

Perhaps there are still some things at which humans can beat computers.  I'll just try doing it by hand.

Ok, maybe not entirely by hand.  IWAP that takes a name and lists everyone with that as their first name.  IWAP for last names, too.  In each case, it sorts the list by yet another score, this one the product of the person's score and the number of choices I would have for the next link.  With these programs I can just explore the tree of possibilities manually.  Here are some chains I've produced so far:

Michael Shannon Elizabeth Ashley Judd Nelson Franklin
Mia Sara Gilbert Gottfried John Oliver Platt
James Taylor Elizabeth Jordan Elizabeth Taylor James
Clark Gregg Henry Thomas (Ian) Nicholas Brendon

One more idea before I wrap this up.  A simple 2-person chain can be used as part of a puzzle.  If I give you the first name of one person, and the last name of another, it could clue the name I didn't give you, which they share.  For instance, Meg Reynolds would clue Ryan (at least, given the particular set of people I narrowed it down to).  In theory, another pair of names could clue another "middle" name, and the middles could be put together, and so on.  In practice I think it would be very difficult to construct such a tree of names and have it be solvable by hand.  For the simplest 2-person case, IWAP that takes a name and finds unique clue pairs, but once again lots of human intervention is necessary to prune the possibilities.  Here are some examples for you to try:

Kurt Crowe
Clive Teale
Aaron Sorvino
Samuel Rathbone
Brion Woods
Gene Osbourne
Currie Norton
Clint Hesseman
Diane Stapleton
Peter Mewes