I’m excited to be programming in Java again. I bought one of those fancy touch screen Android phones in October because I wanted to start writing applications for mobile phones (OK… I wanted a new toy). I skimmed the Android framework documentation and did the Hello World tutorial back in December. But late last week I dove in head first and decided to start writing an app.

I believe that the best way to learn how to do anything is to just do it. Overcoming the challenges you encounter while building something real is where you learn all the twisty little quirks that every platform has. The online Android developer guide and reference is fantastic. And I love books (any recommendations?). But, for example, in the last few days of actually building an an Android app I’ve learned:

• There’s a difference between a paused activity and one that’s just obscured.
• Trying to load Drawable resources in onLayout() of a custom component breaks the Eclipse WYSIWYG layout tool (“NotFoundException: Could not find drawable resource…”).
• SoundPool feels much more responsive than MediaPlayer for playing short sound effects.
• DatePicker uses the same internal representation as Calendar.
• Creating a custom component that extends a standard one and draws itself is actually pretty simple.

So far, developing on the Android platform (version 1.5 right now) feels solid and is very productive. The framework takes care of  a lot low-level junk for you so you can concentrate on app features. I feel like I’ve wrapped my head around the “Android way” of Activities and Intents and Listeners. Now it’s just a matter of becoming a little more comfortable with Java and learning the capabilities of the framework itself.

I recently had a hard drive go bad on me which actually turned out to be a motherboard going bad. The hard drive may be fine but I don’t want to take any chances, and anyway, you can buy 1 terabyte now for about $70 US. There was definitely something goofy going on with access to the drive. I keep daily backups but a perfect backup of a corrupted source is no good. My most important files are my digital photos, mostly JPG images. I did some visual spot checking but once I hit the 8,500 photos in 2005 I looked for a better alternative. Here’s what I came up with. (This works on Linux computers. You’ll have to alter your process on Windows and Macs but the principles are the same.)

First, find a nifty little utility called jpeginfo and install it. jpeginfo is a command line program that attempts to quickly decompress a JPG file and tell you what happens. It’s not infallible, but I’m betting that if the JPG file decompresses successfully then the files are good.

Then you just open a shell, cd to your photos directory and run jpeginfo -c on all of your files. Like so:

    cd photos
    find -iname "*.jpg" -print0 | xargs -0 jpeginfo -c | grep -e WARNING -e ERROR

Any corrupted JPG files will be listed.

How it works: the find command lists all files ending with .jpg (case insensitive). It passes it to xargs which runs jpeginfo -c on each file. The grep command shows just the files that had a WARNING. Instead of grep WARNING you could also redirect the output to a file so that you can see all of the results.

I found 4 corrupted JPGs in my collection of over 35,000 photos. I knew about some of these. At least 2 I think were corrupted during transfer off the memory card. JPG images are surprisingly resilient. Even with bad data in the file a lot of the image could be decoded in 3 of 4 of the images. Here are the best and worst of the corrupted 4.

Information on changing the default Gnome terminal size is out there but it is scattered and scarce. I have two 20 inch monitors. The default size (80×24) is too small. Here’s how you can change the default. This applies to all launchers, menu items, and keyboard shortcuts:

• Run gconf-editor
• Open desktop → gnome → applications → terminal
• Change the value of exec in the right pane to: gnome-terminal –geometry 132×40
  (Where 132×40 is whatever size you want.) (Use a real “x”, don’t copy and paste the multiplication symbol above.)

I use an automated deployment script for BigHugeLabs.com. I’ll regularly do around 10 deployments every day so it’s important that the system has as much automation as possible. The deploy script builds CSS and JS files, updates shared libraries, deploys files to several locations, and does some testing.

One test I recently added was an automated test for browser SSL warnings. SSL warnings appear in a browser when the browser requests an https page but some of the resources requested on that page are unencrypted. While a study has shown that most people will ignore the warnings, that still leaves a lot of people who take them seriously.

The test works by using wget to request SSL pages and test for non-encrypted resources. Here’s the relevant Bash:

wget --quiet -O - "https://bighugelabs.com/cart.php" | grep -P '(script|img).*?src="http://'
if [ $? -eq 0 ]
then
  echo "     *** SSL WARNING IN CART/CHECKOUT! ***"
  beep -r 3
else
  echo "     OK"
fi

wget requests the page and pipes it to stdout. grep scans the page using a Perl regular expression for tags that load http:// resources (rather than relative URLs or https://). $? -eq 0 tests that the exit status of grep is zero. The exit status is 0 if grep finds a match and 1 if it does not. For good measure the script beeps three times if there is a problem.

This isn’t fool-proof by any means. The regular expression isn’t all-inclusive, for one. It also ignores problems that can be caused by scripting. But it’s a start that should catch some problems caused by embedding resources directly into the page.

If you are using the expire_logs_days setting for automatic binary log removal in MySQL and your binary logs are not automatically deleting themselves (SHOW MASTER LOGS), it’s probably because the index is out of sync with the actual log folder. Manually remove the logs (/var/log/mysql) and then do RESET MASTER to get things back in sync again.

“Because of the recent growth due to Facebook Connect and internationalization, we are running out of our allocated 32bit UIDs.” Running out. Of 32bit UIDs. I wish I had Facebook’s problems.

I’ve been using the web-based version of Evernote in earnest ever since Google announced their intention to abandon Notebook. It’s nice overall but has some quirks that I do not like. The upload quota for one. The fact that I have to login all the time. Also the pop-up upgrade ads. And I don’t like that I can’t clip images by right-clicking them; it looks like you can, but you can’t—you have to highlight them first. Clipping a page takes too long because you have to wait for it to generate a screenshot of the whole page.

Too many interruptions in flow to consider paying money for the service. Which is a bummer because I really wanted to like it. Now I have to keep looking.

If you’re as frustrated as me by online note taking apps, you might want to check out Cloud Notes, a blog whose owner is on a quest to find the perfect online note taking solution.

This is my keyboard. There are many like it, but this one is mine.

So I thought I was losing my mind when a simple LEFT JOIN on a new table was taking forever to complete:

SELECT email FROM a LEFT JOIN b ON b.email=a.email WHERE b.email is null

That’s about the simplest LEFT JOIN you can write to get all rows in a that are not in b. This is useful, say, if you want to get a list of email addresses that have not unsubscribed from your mailing list (where you store unsubscribed addresses in another table).

Here was the EXPLAIN. This is bad:

+----+-------------+-------+------+---------------+------+---------+------+------+-------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+------+-------+
|  1 | SIMPLE      | a     | ALL  | NULL          | NULL | NULL    | NULL | 5244 |       |
|  1 | SIMPLE      | b     | ALL  | NULL          | NULL | NULL    | NULL | 4493 |       |
+----+-------------+-------+------+---------------+------+---------+------+------+-------+

Table a in this case has 5,244 rows and table b has 4,493 rows. The EXPLAIN is saying that MySQL intends to not use any indexes and do a full table scan on table b for each and every row in a.

I thought I was losing my mind because I do these types of queries all the time and they always use the right indexes and they’re very fast and all is right in the universe. And in fact I did a similar query on a different set of tables in the same database and it worked perfectly fine, just like I expected it to.

After literally hours of pulling my hair out, I found the culprit: the email column is a varchar and, crucially, table a and b were in different character sets. (My other query worked because that column was an int.) Table a was in utf8 and table b was in latin1. This prevented MySQL from being able to use the index.

The fix:

ALTER TABLE b CONVERT TO CHARACTER SET utf8

Then the query returns instantly. This is how LEFT JOINS are supposed to look:

+----+-------------+-------+--------+---------------+---------+---------+-------------+------+-------+
| id | select_type | table | type   | possible_keys | key     | key_len | ref         | rows | Extra |
+----+-------------+-------+--------+---------------+---------+---------+-------------+------+-------+
|  1 | SIMPLE      | a     | ALL    | NULL          | NULL    | NULL    | NULL        | 5244 |       |
|  1 | SIMPLE      | b     | eq_ref | PRIMARY       | PRIMARY | 602     | db1.a.email |    1 |       |
+----+-------------+-------+--------+---------------+---------+---------+-------------+------+-------+

Be careful using ALTER..CONVERT TO. It converts the actual data in the fields and can convert types (e.g. from text to mediumtext). Read the documentation and know what you are doing before going in guns blazing.

This is one of those problems that’s best to avoid. I didn’t specify utf8 when I created the table and my defaults were wrong. The old tables were created and ported over from a different database. If this mess happens to you, may God have mercy on your soul. Also, check your character set encoding defaults on your MySQL configuration, for each database, and each table in each database (yes, they can all be different).

Kind of a mish-mosh of snippets below. A lot of them are useful if you’re managing servers. The rest are useful more generally and occasionally. Many of these utilities may already be on your system, others you may have to install. (Hopefully you’re using a Linux distro that has all of these in its standard software repository so you can just search, click, install.)

Get a random number between 1 and 100

echo $(($RANDOM % 100 + 1))

Actual non-swapped physical memory in use by a process in kilobytes

ps aux | grep process_name | awk '{sum += $6}; END {print sum}'

For example, to find out the total amount of memory being used by all Apache processes, use “apache” as the process name.

Count the number of processes with a given name

ps aux | grep process_name | wc -l

Total system memory

head /proc/meminfo | sed -n /MemTotal/p | sed -r 's/MemTotal:\W*(.*).kB/\1/'

This is useful in scripts that need to compare memory against the total memory available on a system.

Print all lines in a file that contain a pattern

grep 'pattern' filename

Print all lines in a file that do not contain a pattern

sed -n '/pattern/!p' filename

Do something once, at some later time

at time
> type your commands at the prompt
CTRL-D

This one is kind of odd. For example, first you type at 09:30 at a console. A prompt will appear. Then you type commands you would like executed at that time. Press CTRL-D when you are done entering commands.

Mail output of a command to someone

command | sed -e "1iSubject: Command results\n" | sendmail -t email_address

The sed -e bit prepends the subject line to the beginning of the output for sendmail.

Relabel an ext2/ext3 (Linux) formatted drive

e2label device label

For example, sudo e2label /dev/sda1 data.

Relabel a DOS formatted drive

First, create ~/.mtoolsrc:

# For example, where /dev/sdh2 is replaced with your MSDOS device
drive c: file="/dev/sdh2"

Then:

mlabel c:

Convert password protected PDF files to unprotected PS

pdftops -upw password filename.pdf

I used this the other day when I needed to load a password protected PDF into GIMP.