software

It's a horrendously long blog title, I know, but I wanted to mash in all the keywords I was looking for when trying to find a solution to this very sticky problem so that other folks running into the same issue can spend the next six hours doing something more productive.

jQuery makes grabbing one group of elements from a page, duplicating them and adding them somewhere else really easy. You just use a combination of clone() and something like append(), as in the following:

$('#surrounding_div').clone().appendTo('#some_other_div);

The problem with dynamically adding form elements in IE

Woot! So easy! Right, until you start goofing with forms and IE (Internet Explorer) (appropriate sound). The problem is that IE doesn't allow you to manipulate the 'name' attribute in the DOM directly. So, something like this won't work:

$('#form_element').clone().attr('name', 'new-name').appendTo('#my-form);

If you do that with a radio button, the interface will tell you that there's a problem, because since the new radio button has the same name as the original one, only one can be selected. Bugger!

Luckily, the solution is clear, but it took forever to figure it out. Instead of adjusting the name attribute directly with something like attr(), you need to edit it manually in a block of html and then add that html to the page. So, something like this would work:

$('#form_element').parent().html().
  replace(/orignal_name/, 'new_name').appendTo('#my-form');

There's a couple of key points here:

1. Because we're getting the html using $.html(), we need to move up the DOM to grab the element surrounding the element we wish to add using $.parent(), then we can grab the html using $.html().
2. I'm using the javascript replace() function to pass a regex pattern and replacement text to the html.

What about dynamically cloning groups of inputs with jQuery?

In my particular use case, I was grabbing a set of form elements and adding them this way. There were a couple of challenges, which was that I couldn't use $.parent() to get the outer element, and the other was that the names of the form elements were being set dynamically. I learned a couple of tricks to help with this:

Use the regex modifier /g to replace all insteances of a pattern. So, when doing replace(), if I needed to replace ALL instances instead of just one, it would look like this:

replace(/original_name/g, 'new_name')

Because I couldn't use $.parent(), in the end I had to wrap an element around the html using simple contatination, like so:

var newHTML = '<div>' + $('#element').html() + '</div>';

I hope that helps you, it was sure a bugger to figure out.

It recently came to my attention that there are some gaps in my conceptualization of Drupal security. I was fortunate enough to have this pointed out to me by the Drupal Security Team, and not by a DOS, CSFR, SQL injection or XSS attack. After publicly bemoaning the mild lashing I received, four members of the Drupal community suggested I read Cracking Drupal. One of them even sent me a copy. No other book was even mentioned, which says to me that - considering how recently it was released - the book fills a void of knowledge that was seriously aching for coverage, and fills it well.

Over years of developing, I've become familiar with the various vulnerabilities that make their way into code, but I've never felt like I could build a complete defense. My knowledge has been piecemeal, drawing from documentation, books, interesting conversations and other people's code. In my case, Cracking Drupal did a fantastic job of gluing these pieces together into a comprehensive frame of mind.

What becomes clear very quickly in Cracking Drupal is that Drupal is quite a wily beast that gives developers real incentive to learn security. There are few functions in Drupal whose exclusive purpose is security, and Greg makes it clear that learning how to secure your site has definite side benefits: "When developers learn and use the API, they are not only safer but more effective and more efficient." When you learn how to use different aspects of the Drupal API (forms, translations, helper functions, theming) you gain bits of security as a bonus. If you set out to learn Drupal security, you'll come out the other end with a pretty solid grasp of Drupal APIs. Either way, it's a win.

Cracking Drupal is surprisingly brief. In 134 pages, Greg covers a lot of ground including:

• An overview of the different types of attacks one is likely to encounter, from physical to social
• Most (if not all) aspects of the Drupal API that have security implications
• Coverage of security-related contributed modules
• An introduction to the Drupal Security Team
• Demonstrations of exploiting common weaknesses in Drupal modules and how to fix them

An interesting choice is made in Cracking Drupal to keep a somber atmosphere around the subject matter. In almost any other context, this would be an immediate turn-off. I appreciate humor and optimism to drive my enthusiasm when reading. In contrast with other instructional books which end a chapter with a "go for it, get things done!" message, this book ends chapters with lines like "This paranoid perspective is a good one to maintain as you write, review, and implement features on your site." and "Remember that it is nearly impossible to fully protect yourself from a dedicated and persistent attack." and "If nothing else, I hope this chapter has scared you a bit about the realities of just how easy it is to exploit insecure code and sites".

In a book covering attacks that can result in a very serious loss of time and money, this lack of optimism is probably a good thing. And the final chapter, "Un-cracking Drupal" does leave the reader with the sense that something can be done. It's difficult work, but it's doable. Ultimately, I think the book drives home the fact that the most effective way to make a module or theme secure is to do it right from the start.

The title chapter of Cracking Drupal was probably the most lively and hands-on part of the book. I came out of it feeling like I could really enjoy exploiting vulnerabilities for the greater good. Because of this reaction, I think it would have been a good candidate for a first chapter to really whet people's appetites.

Overall, I think Cracking Drupal does a tremendous service to the community by pulling together the most important aspects of Drupal Security into one solid, compact document. While I came into the book having already been introduced to many of the concepts, it filled in a few gaps, and made the subject matter finite and approachable (albeit a little scary). I suspect this book will serve well as a guide and quick reference as I dive into identifying and patching up vulnerabilities in the modules I maintain.

A couple things I learned

While the greatest benefit to this book was the broad, sweeping overview of security, there were a few additional gems that will come in handy later on:

• There's a lot more to hook_menu() than I was aware of. Good coverage of examples on p.55
• I didn't realize that you had to exit after using drupal_access_denied(). p.59
• Ah, db_placeholders() - a useful function for passing a number of variables to db_query() p.65
• I had no idea there was such a robust node access API. Wow!

Notes in the margin

Below are a few unorganized comments that constitute my wish-list for future versions and complements to the author:

• Good quote regarding the definition of security: "For this book I’ll define site security as follows: A site is secure if private data is kept private, the site cannot be forced offline or into a degraded mode by a remote visitor, the site resources are used only for their intended purposes, and the site content can be edited only by appropriate users."
• I would have liked to see more AJAX security strategies and techniques covered.
• I liked all the Drupal 7 references, gives a good feel as to the direction of things
• I was surprised that there were not more brutal admonitions about hacking core, but suspect that's because they represent much fewer vulnerabilities than badly designed contrib.
• I was happy to see some coverage of CVS and DRUSH, namely using CVS to keep code up-to-date
• Nice coverage of security-related modules starting around p.41
• A brief mention is made that using mixed-mode SSL is pretty pointless. This is a big deal, I wish it had gotten further coverage.
• Being more of an optimist, I appreciated this particular phrase: " Every day there are more and more techniques beingdeveloped to attack sites, but every day there are also Drupal users reviewing code and providing new modules and enhancements to core to keep your site safe." Ahh, a glimmer of hope!
• Would have liked to see more coverage on the use of form tokens. If one must step outside of the forms api, this could be very important
• I liked that theme safety was covered, and thought the take on it was interesting: Make the theme secure by giving themers no reason to make stupid mistakes.
• Since the 'Vulnerable' module was patched up in the end, maybe it should actually be named to indicate that it's meant to be a useful module. That would feel more like a practical example.

Skip to the video demo

Open the Drupal.org project page for Query-Based Views

One of my first Drupal modules ever was Ajax Table, which gave some rich tools to users who would would normally circumvent Views to create a report or feed by constructing it programmatically. The problem being solved was "what does one do when they need to build a report starting with a query?" The Views answer is to expose everything you need to Views through modular hooks. This is an awesome long-term solution because, once your tables and custom output functions are exposed to Views, they can be used by any other view, and you can combine stuff in really novel, nifty ways.

However, a lot of people don't have the time or expertise to do that. They need to start with a query, and they would normally create a custom function, run the query directly and then generate the output programmatically. I think that there will always be use cases for this, and even if the same report could be created in Views, some people will do it this way simply because it's faster for what they need. Ajax Table was a utility module for these use cases. By running a query through a helper function and passing a couple parameters, users could generate ajax-reloading, searchable, sortable reports with very minimal work. It was handy, and I used it all the time. I still use it for my internal time-keeping / invoicing / proposal generating system.

The functionality was useful enough that I built out a second iteration called Query-Based Views, this one using Drupal configuration instead of straight-up code. I demo-ed the module to my local Drupal User Group, and got some positive responses regarding of the ajax functionality, rapid development and in-line editing. However, the general consensus was that the functionality would make a much greater impact if contributed as a Views plug-in, or as patches to Views itself.

So I shelved the project, thinking that until I knew enough about Views to integrate this, it would be better not to fork efforts.

Then a few days ago it struck me that maybe I should ask Earl Miles (merlinofchaos), the developer of Views, if the idea of starting with a query was actually Views-compatible. I was thinking that maybe the Views query builder was just an element of the UI that could be replaced out with something supplying a raw query. He said that it really wasn't. Too much of the power of Views is derived from abstracting out the query building process into the UI that's there.

That means there's actually room for this type of module. It's true that a lot of functionality is duplicated, but Query-Based Views runs on a different paradigm that's incompatable with Views, so it's reasonable for it to do so. Some specific bits like in-line editing could be abstracted more to be able to work with Views and Q-Views, but as a first effort, I think this module should fit the bill.

So I spent a couple of days porting the Drupal 5 version of Q-Views to Drupal 6, fixed a bunch of bugs and am releasing it as an alpha. The bulk of the code is written by the Chris Shattuck of yesteryear, who was not as well versed in Drupal ways, so if the module is useful enough a code rewrite is in order for a non-alpha / beta release. But as it stands, it's functional and ready to roll.

Video Demo

Q-Views has a lot of functionality, so this is a longer video (20 min). Feel free to skip around, though.

Skip to the video

To celebrate by recent release from employment, I spent several days busting my butt to put the sexy back in administration. Namely, I updated the Drupal Navigate module to include some new features, and to fix some long-standing bugs that had been making administrators feel less than sexy for the last several months.

For the uninitiated, Navigate is an administration module that works a little like Administration Menu. It loads a sidebar with widgets which allow users to search the menu, nodes or users, construct favorite lists, and load up expandable / collapsable menus. It works really well for clients who aren't used to Drupal. The newest release allows admins to set default widget sets, and adjust user sets. You can also theme it (a funky lemon theme is included as an example).

There's a video demo below, but here's a quick list of improvements made in this release:

• Made snappier through quicker transitions and fewer ajax calls
• Added ability to manage default widget sets for users and user roles (all ajax, btw)
• Made theming Navigate really easy
• Fixed compatibility issues with Administration Menu
• Added keyboard shortcuts
• Added XHTML compliance
• Added import / export ability for Favorites and for entire widget sets, so you can quickly deploy a set from one site to another
• Added 'customize' permission, to keep *certain* users from messing up their own sets
• Added ability to search users
• Squashed some bugs
• Re-factored lots of code to be more Drupal-esque
• Made some minor layout adjustments

I'm hoping to put in some work to help with the current administration tools in D7, but before that, I needed to grease my wheels a bit with some contrib work to to anchor some jQuery techniques I'd learned, and re-familiarize myself with D6. It feels good getting so much done so fast. You can do that with contrib work, but it's hard to be that productive in core Drupal. Things just move at a different pace there. Now that I've gotten a bit out of my system, I think I can crack down a bit and start seeing what I can do for core.

And here's the demo video to celebrate Issue Queue Zero (at least for the D6 version):

I spent some time today digressing from my regular work to see if anyone was working on and iPhone / iPod Touch looping app that works like Ableton Live. I wanted something with variable-length loops and that would play loops as you recorded them. It took me a surprisingly long time to dig Loopy up, so here's my blog post and vote. Here's how Loopy works works:

• Tap a speaker icon to start recording (no limit to length, afaik - woot!)
• Tap the icon to stop. It will start to play immediately (if you're familiar with Ableton live, it works the same way)
• Tap another speaker to start recording the next loop. You can record any length of measures, another woot!
• There's six tracks (the six speakers you see below), but you can drag one speaker onto another to merge them.
• If you start recording before the measure starts, it records the lead-in and subtracts the space from the end, if you stop recording before the end of the measure. That's pretty cool, Ableton doesn't even do that.

I love having this in my pocket. Because I have an iPod touch, I also need my headset with mic in my pocket, too, but that's a small price to pay for such uber coolness. On @loopyapp on Twitter, the developer posted a short clip of Imogen Heap using Loopy. That led me to Imogen's web site, which lead me to her YouTube channel, where I got caught up in further digressing from work. What a cool, down-to-earth person she seems to be. Hopefully I run into her at some Tweet-up. Maybe she'll record our conversation and put it in a song...

There are some things about this app that can no doubt be improved, and it seems like the developer, Michael Tyson, is pretty on top of it. It's the kind of app that makes me want to quit my day job and go into iPhone development.

Here's my wish list / gripes:

• A foot pedal attachment to start and stop recording. I tried my bare big toe to record a guitar track, but it wasn't very reliable. If there could be a full-screen button, then the foot might work. One suggestion in the forum was more practical, to allow the user to set the count-in and length.
• Every once in a while the sound stops and I have to quit the app and come back.
• Probably due to the mac headset mic I use, the sound quality is dismal. This app will result in me purchasing yet more fun hardware for my Touch.
• Setting the tempo seems a little hit and miss. Maybe it's my huge finger.
• There's some latency, and that is a little annoying. Laying down a couple of ill-concieved beats (I am no beat boxer but have always longed for the prestige that goes along with being one) made it clear that any precise recording would be a challenge.

Overall, wicked cool! Thank you, and I'm looking forward to version 2!

After getting familiar with the GTD (Getting Things Done) ethos a couple years back, I've spent a disturbing amount of time cobbling together systems for capturing, processing and taking action on my stuff. It hurts to think about all the dead end trails, the false starts and the straw men that kept me from fully embracing a system to free up my head once and for all. A huge part of my problem was committing to a system long enough to figure out if it worked. I've been working off of my current system for a couple months now, and feel more confident with it every day, so I feel like I can document it online without inadvertently sending someone else down a frustrating path.

Capturing ideas

I use a few technologies to capture ideas, depending on what I want to do with it.

Ideas the require action = OmniFocus

I've given several GTD programs test drives, and I end up coming back to OmniFocus. After reading many positive reviews on Things, I tried moved everything over to it and tried it for a few weeks. But it just didn't work with the way I think. I think it projects and sub-projects, which is how you structure OmniFocus. Things relies on tagging, which is a simple concept but doesn't map well to my mind. It was also difficult to get the perspectives I wanted on my actions.

I also use OmniFocus on my iPod touch to capture an action when I'm away from my computer.

Ideas that I can flesh out right now = Evernote

In the past, I've used hierarchical lists or individual text files for this. The problem with that is that I have to think about what to name the files and where they should go, or I get one huge file that seems a unwieldy really fast. I tried OmniOutliner for w while for this, and it worked reasonably well, but I ended up spending way to much time structuring it. Evernote lets me start fleshing something out immediately, and I don't think about where it should belong, since the search works so fast. Since it's got some rich text, I can still do some hierarchical lists when I need to, or I can just stream-of-consciousness it.

Like OmniFocus, I use the iPhone version of Evernote when I'm out and about. For example, the other night I went out on a date with my wife and I kept running into little things I wanted to remember. I kept a new note open in Evernote and jotted things down with my one hand. Call me corny, but the iPod touch is perfect because you don't have to kill the mood by withdrawing your hand from your significant other's to take a note.

Stuff that might be useful or entertaining to someone else = Ecto and Twitter

If the item I want to share is quick, I tweet it. If it's going to take some time, I create a placeholder for a blog entry in Ecto. If I have time, I might write the entry immediately, but if I want to put a pin in it for later I'll just add a descriptive title and maybe a little brain-dump for the text. This blog entry is an example of one of those.

Working out stuff on the inside = Journal.txt

While it isn't often mentioned in the genre of productivity, I think it's worth mentioning that sometimes there are things that you need to think through, things that are bugging you and getting in the way of doing what you really want to do but that are also really hard to put a finger on. I find the best way to work this stuff out is to just start writing, stream-of-consciousness style. I've got a text file so I don't even have to think about formatting, and I just start typing. Without fail, a session with journal.txt leaves me more free and ultimately more productive.

Deciding what to do next

After capturing, the next part of GTD is about deciding what to do and doing it. The idea is that you should be able to re-evaluate your actions on a regular basis so you don't end up wasting time on stuff that doesn't get you closer to where you want to be, and that this evaluation process should be painless, even pleasurable.

Evaluation = OmniFocus

The first thing I will normally do switch to OmniFocus, and review my next actions. I have a list of daily activities which trigger stuff like clearing my e-mail, checking my calendar, petting the cat, that kind of thing. As I go through the process, I might flag certain actions or arrange their due dates to accurately reflect what I want to do.

Execution = OmniFocus or Evernote

The most important part of an effective day for me is having a list of prioritized actions to work from. It allows me to stay focused on stuff instead of continually re-evaluating what I should do next. I'll create this list either by flagging items in OmniFocus, or by creating a bulleted list in Evernote. If I need to think through my priorities, I'll do a stream-of-consciousness on Evernote.

Execution of time-sensitive stuff = Cell phone

My $20 cell phone is the most reliable method for reminders because it's always in my pocket and it works on vibrate mode, so it will alert me even in loud areas. When I add something to my calendar I need to be reminded of, I'll also add an entry to my cell phone. Knowing I have a fail-safe method for timely reminders does a lot for my mental well-being, and it's worth the double-entry.

I spent an hour or so testing out some different ways of ripping songs from someone else's iPod onto my mac for the purposes of wedding entertainment. The best tool I found was installing MacFuse, then iPod Disk. After installing, the iPod will show up as a mounted disk, and you can browse files by artist, album or genre, just like on the iPod. When it worked, it worked well, but it also crashed a few times (sometimes bringing Finder along with it) and I had to restart to get it recognized again.

Recently, I conducted a number of in-depth interviews with members of our company to get a fix on the current status of our project, and to gather any research had been done in our market so that I could share it with other team members. The process was really interesting, and something I'd advocate for anyone who feels like their team has maybe become a little disconnected from one another. Here's a quick list of the technology I used for folks who might also be conducting remote in-depth interviews they will need to review later.

• The interviews were conducted over Skype, partly because I use Skype as my business line, but also so I could run the sound streams through recording software directly.
• I used WireTap Studio (recently purchased as part of the MacHeist bundle) to record two streams during the inverview, one from my USB headset mic and other from Skype itself
• After the interviews, I listened to each one and transcribed pertinent parts to text. I tested several options for variable rate playback, so I could listen to the interviews in double-time (there was a lot of material to review), and ended up using Quicktime Player. After you open an audio file, you can select Window > Show A/V controls, and use the Playback Speed option to chose a rate. For the more pronounced speakers, I could use a 2.5x speed playback. For faster talkers, I had to take it down to 2x. The Jog Shuttle also helped a lot to rewind a sentence or two back.
• I posted the resulting document in Google Pages, which we use for our in-house wiki, and which will give everyone an option to review and add comments.

The interviews went pretty smoothly, technically speaking. It helped to conduct an interview with a fellow coworker initially, before interviewing the major stakeholders, so I could get any technical difficulties out of the way, and basically do a trial run.

In preparation for impending baby-dom, we bought our first real camcorder, a high-definition Cannon IXIA HF100. It's about a hundred times smaller than camcorders when I was a kid and it's dead quiet because it uses an SD card for storage. My enthusiasm for the new gadget abounded. Imagine my disappointment when I discovered that I could not actually watch the resulting .MTS video files on my mac. After extensive searching, I've found a couple tools for decent MTS file integration with the mac, so here they are:

VLC

VLC will play MTS video without any additional codecs. And it's free. Nice.

HD Quick Look

Even though VLC will play the video, you won't get MTS file thumbnails in finder until you install this little app. It seems like it should be free, but the $7 price tag is reasonable enough.

Voltaic HD

Voltaic will convert MTS files to Quicktime format (maybe others as well, I haven't fired it up for a while). The $35 price tag on this one made me cringe, but it's one of only two product I know of that actually converts MTS files directly. Oddly enough, iMovie will import MTS from the camera, but won't import once it's a stand-alone file. I scoured the internet trying to find a way to trick iMovie into converting it anyway, and the only option I found was to clone your SD card and mount it as a drive whenever you want to import the video. The process seemed a little too crufty for my liking, so I shelled out for the Voltaic app.

Roxio Toast 10 Titanium

At $80, it probably isn't worth it to get this app just for the MTS file conversion, but from what I read, it does it quite well, and faster than Voltaic HD.

When I was considering purchasing a mac and ending (or rather, reducing) Microsoft's reign of terror in my life, I was told my multiple sources that one of the productivity boons of macs is that you never have to restart them due to memory getting eaten up by invisible gnomes in your chassis. The truth is that OSX definitely appears to do a better job, but it seems to leave memory scraps behind, just like windows.

I shelled out twenty bucks the other day for a little app called iFreeMem, which will - on demand - run through your ram and clean out any cruft, leaving you with a bigger green slice in that memory pie. I use it several times a day, and it seems to work really well even though I keep wondering if maybe the program just overrides the graphical display in activity monitor and makes you think you have more than you do.

iFreeMem also adds a nice little menubar icon that shows a monochrome pie-chart display of current memory usage. I find myself glancing up there when I'm using a bunch of programs at once to make sure I'm not going to be surprised by a spinning beach ball of death anytime soon.