log_processing_rules:
   - type: multi_line
     name: new_log_start_with_date
     pattern: \[\d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])

Here is my full YAML file, for example (in a custom config directory. filename: /etc/datadog-agent/conf.d/custom.d/conf.yaml) - based off a Forge created server. I used wildcards to ensure that it was universally available to all hosts and sites.

logs:
  - type: file
    path: /home/forge/*/storage/logs/*.log
    service: laravel
    source: laravel
    log_processing_rules:
       - type: multi_line
         name: new_log_start_with_date
         pattern: \[\d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])

I've been getting "convert to CrashPlan Pro" emails for a month or two now. I didn't have any interest. But now it is being forced, as they've announced they are exiting the consumer market. They provide an "easy" way and a discount for 12 months to migrate you over to their "small business platform" - except get this.

Some cloud backups will be restarted

Some of your devices have more than 5 TB of data backed up to CrashPlan Central. Due to technical platform constraints of the migration process, each device's cloud backup must be less than 5 TB.

If you continue, these cloud backups will be permanently removed after you migrate your account to CrashPlan for Small Business, and these devices will start new cloud backups.

So two of my NAS units are actually close to 100% backed up (it only took YEARS) - one at 14.7TB, the other at 23TB. Both of those backups, which took forever, are going to be removed. Even though the consumer CrashPlan could support it, and the CrashPlan Pro talks about unlimited backups ("it can be gigabytes or terabytes!") they give me no way to migrate that. That has to start over from scratch.

Need I remind you that CP's upload performance is abysmal, the client is bloated and "Java-ey" and resource intensive. The whole thing is garbage. Sadly, it is the only Linux capable unlimited client out there though still.

If you don't need Linux, don't touch CP. Go with Backblaze. CP has tons of complaints of lost or broken backups, horrible speeds, then this kind of shit.

To allow you time to transition to a new backup solution, we've extended your subscription (at no cost to you) by 60 days. Your new subscription expiration date is 09/21/2017.

How is that 60 days? It's 30 days from today.

I just feel bad for all the people complaining on Twitter about prepaying and CP's policy is no refunds. Their backups will sit on the "Crashplan for Home" system until it expires, I guess - but a backup system with an expiration isn't much of a backup system at all. :)

One of my clients contacted me saying his site is not performing properly and it has been getting worse. It's on a server I setup, so the server shouldn't be the issue, but I wanted to make sure there's not something weird going on with it, and see if I can find some causes.

He's using my CloudFlare full page caching method - so that's good. His frontend performance (for cached pages) is lightning fast. But the backend and uncached pages (and 404s) are abysmal.

I notice the standard stuff - UberMenu, Genesis theme - both very general purpose things, which we all know sacrifices performance for flexibility. UberMenu can be a beast. "I can do everything" themes often are too. Anyway, the site WAS just fine previously with those, so it's something else. In the PHP-FPM slowlog and New Relic I see "maybe_unserialize" keep coming up over and over. That function ideally would never exist if it wasn't for WP storing serialized and unserialized strings in the same places and having to detect if it's working with one or not (such an amazing design...) - but either way, it shouldn't be as slow as it is.

That function is used all over the place - but there are a couple key places to look. Especially when cron is disabled in wp-config.php and I am seeing calls to wp_get_schedule, wp_next_scheduled, and other cron-looking functions, on what seems nearly every page load. I wound up finding the culprit. The client had used a plugin called "Easy Social Metrics" and what it would do is add to the cron queue at least one row per post. So there were hundreds if not thousands of "easy_social_metrics_update_single_post" one-time cron items that were being autoloaded EVERY PAGE LOAD - because the "cron" option is an autoloaded option. So every page load inside of the WP frontend or in the admin area was loading tons of serialized text in this single variable, checking if it was serialized, unserializing it, possibly updating/modifying it, reserializing it, etc.

The sad thing was he didn't like the plugin so he had deleted it. But it left behind all the residue.

I've seen similar issues with other options, the "rewrite_rules" one was abused by some other plugin as well in the past on another site (I don't have the details anymore) but this is the second time I've had to forcefully kill off an option to actually rescue performance. Sadly, this is nothing any normal end user who is using only plugins to interact with their site would ever notice, until they installed some sort of cron manager and saw all these excess items hanging around. Not many people would install a cron manager though, or should need to. So that would just mean they have to hire someone now to diagnose why this magical CMS that can do anything is so abysmally slow.

It does have a nice looking admin interface, low barrier to entry for end users, but the foundation is horrible, the plugin and theme ecosystem seems rich because of the number of options and downloads, but the quality and performance on most are lacking.

Here's a chart of the performance. You'll see the first time I killed the "cron" option around 3:13pm, which was rewritten in full again and caused the second spike, and then when I finally purged it for good around 3:17pm. Site is back with sub-1 second average and remained that way. Which is about as good as you're gonna get with 20+ plugins including UberMenu and a kitchen sink theme. :)

2. a.com

3. www.a.com

4. b.com

5. www.b.com

6. c.com

7. www.c.com

8. d.com

9. www.d.com

10. e.com

11. www.e.com

This is why we use things like CloudFlare and Let's Encrypt. Depending on your needs. It's 2017, who actually needs to buy SSL certs anymore?

Before my customer purchased the certificate I had to ask, since "domain" means something different to me (you know, an actual domain) and this is what support told me - I verified this after the fact. Now as she states "the primary covers both [www and non-www]" - you can squeeze in an 11th host, but since I needed the domains fully covered, I could only use it for five domains either way.

While you can upload a file with a plus "+" in it, it won't reject you. However, you cannot directly access it. You can only access it by encoding the "+" to "%2b" - so unless you are doing that in your links, you'll wind up with "access denied" S3 XML error. Because the "+" is interpreted as a space - instead of the actual "+"

While they could fix this, it looks like they won't at this point, as it will change the behavior that everyone is used to. I found some threads from years ago complaining about it, even back then there was little expectation the behavior would be changed.

Total complaints with S3: 3
Total happiness points with S3: 1,292

# whitelisted IP ranges - will not have limits applied
geo $geo_whitelist {
  default 0;
  1.2.3.4 1;
  2.3.4.5/24 1;
}

# whitelisted user agents - will not have limits applied
map $http_user_agent $whitelist {
  default $geo_whitelist;
  ~*(google) 1;
}

# if whitelist is 0, put the binary IP address in $limit so the rate limiting has something to use
map $whitelist $limit {
  0 $binary_remote_addr;
  1 "";
}

limit_req_zone $limit zone=perip:30m rate=1r/s;

References:

• http://serverfault.com/questions/177461/how-to-rate-limit-in-nginx-but-including-excluding-certain-ip-addresses
• http://stackoverflow.com/a/30083902

Other than the one hard-coded thing at the moment, this seems to work well. Sets up the New Relic configuration as desired, on instance creation. Assuming stock PHP 5.x Amazon Linux AMI.

Just drop this in .ebextensions/newrelic.config or something of the sort, and rebuild your environment. Replace API_KEY with your API key, and change or remove the newrelic.framework line if not using Wordpress (this page has a list of supported frameworks)

packages:
  yum:
    newrelic-sysmond: []
    newrelic-php5: []
  rpm:
    newrelic: http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm
commands:
  "01_configure_sysmond":
    command: nrsysmond-config --set license_key=API_KEY
  "02_start_sysmond":
    command: /etc/init.d/newrelic-sysmond start
  "03_install_php_agent":
    command: newrelic-install install
    env:
      NR_INSTALL_SILENT: true
      NR_INSTALL_KEY: API_KEY
files:
  "/etc/php.d/newrelic.ini":
    mode: "000644"
    owner: root
    group: root
    content: |
      extension=newrelic.so

      [newrelic]
      newrelic.appname = "`{ "Ref" : "AWSEBEnvironmentName" }`"
      newrelic.browser_monitoring.auto_instrument = true
      newrelic.capture_params = true
      newrelic.enabled = true
      newrelic.error_collector.enabled = true
      newrelic.error_collector.record_database_errors = true
      newrelic.high_security = false
      newrelic.license = "API_KEY"
      newrelic.transaction_tracer.detail = 1
      newrelic.transaction_tracer.enabled = true
      newrelic.transaction_tracer.explain_enabled = true
      newrelic.transaction_tracer.explain_threshold = 2000
      newrelic.transaction_tracer.record_sql = "raw"
      newrelic.transaction_tracer.slow_sql = true
      newrelic.framework = "wordpress"

This was my first foray into the world of bash completion, so I incorrectly thought "this can't be that hard - everything has completion now."

After Googling for hours, trying various things, finding statements saying that it wasn't possible - I stumbled upon the solution. It works properly and was surprisingly easy.

The compgen command typically works with spaces for the word separators. What I had to do was make sure the source data was separated by something else, such as a linebreak (\n) and then tell compgen that was the separator.

So this line before the compgen command was key:

local IFS=$'\n'

Next, the complete command still won't work without feeding it the -o filenames option. This seems to be critical, telling it to behave like we are dealing with filenames makes it work properly. Note that I couldn't even find the original example that led me to this solution right now. It's that undocumented.

complete -o filenames

If you'd like to see this put altogether, in a very easy to read, semi-documented autocomplete script, check it out here:
https://github.com/mike503/ec2ssh/blob/master/ec2ssh.bash_completion.sh

It's time to spend a few minutes to share something that hopefully will solve a headache of mine - and maybe one of yours.

I own Synology DS2411+ units. They're very space efficient, and quiet - the form factor is great, and something I haven't found anywhere else (WHY?!?!) - there is a premium for their specifically designed chassis and software - and in the past I've found it to be worth it. Nowadays with ZFS or Btrfs around, cheaper hardware and such... I'm not as happy, but I'm also not in the market for a new setup. :)

Anyway - one of the main things I want to do is to try to back up everything on my physical units to the cloud. The first barrier to entry is that there is only one truly "set-it-and-forget-it" *unlimited* provider that has a Linux client - and that is CrashPlan. The pricing is great, but the client is bloated, buggy and weird. I've begged Backblaze to create a Linux client for years and there was some chatter about it, but still nothing public. At this point with B2 being launched, I'd be surprised if they do it at all, or just have a B2-based app instead (in which case it will be utility based billing. I want unlimited!)

Back to our only option - CrashPlan.

Due to the terms of distribution with CrashPlan it looks like it cannot be officially packaged by Synology. Which is where the PC Load Letter option came in handy. However it had some issues over time, and the Java distribution would require reinstall periodically. Ultimately, it wasn't the most reliable solution.

So I decided - I'll use the instructions to guide me on a manual install, one that isn't subject to issues with the package manager or Java changes. After using the instructions on the Synology wiki and what I thought was a clean, successful installation that worked for a couple weeks, it too crashed. Somehow during one of CrashPlan's own updates (I believe) it wound up wiping all the .jar files away. Trying to reinstall CrashPlan (the same version) actually failed after that for some unknown reason (how?!?!)

Recently I had some issues when trying to setup a Redmine install for a client. Even following the exact instructions it wouldn't work. No idea why. So I decided to look into a Docker-based setup. Wouldn't you know, I found a package that worked perfectly out of the gate. Docker to the rescue.

I realized not too long ago that Synology added Docker to it's package center. While I dismissed it as being more bloatware and attention being paid to the wrong aspects (just make a rock solid storage platform, I don't need an app store concept on my NAS!) I decided I should take a peek at the possibility of running CrashPlan in a Docker container. That way, it was self-contained in a richer Linux environment, with it's own management of the Java runtime stuff.

As of right now, I will say - "Docker to the rescue" - again. After fixing up the right command line arguments and finding a Docker image that seems to work well, it's been running stable and inherited my old backup perfectly. I use -v to expose the /volume1 off my Synology to the container and it picks up exactly where it left off.

That's quite a lot of explanation for what boils down to the magic of it all. Here is the working image and my command line arguments to it, to expose the ports and volumes and such properly. Enjoy.

docker pull jrcs/crashplan
docker run -d -p 4242:4242 -p 4243:4243 -v /volume1:/volume1 jrcs/crashplan:latest

Add more -v's if needed, and change the ports if you wish. Remember to grab the /var/lib/crashplan/.ui_info file to get the right key so you can connect to it from a CrashPlan Desktop application (once of my other complaints with CP)

UPDATE 2017/01/07 - after running this for months, I'll share my script (I believe it actually requires installing bash from ikpg to really be solid) - I put it on my data volume (volume1) in the crashplan directory and it seems to persist. Any time I want to update the container or something seems to have crashed (since CrashPlan can crash often, thanks mainly to Java and the large amount of files and how it is RAM heavy) I can just run this. It's able to persist the relevant configuration between docker runs, and runs more stable than any other solution I've used or tried. CrashPlan is still much slower than Backblaze but it's still the only service with a Linux compatible client and unlimited data (I don't consider ACD to fit the mold exactly yet)

cat /volume1/crashplan/crashplan-docker.sh 
#!/opt/bin/bash -v

docker rm -f `docker ps -qa`
rm -f /volume1/crashplan/log/*

docker pull jrcs/crashplan:latest

# ref: https://hub.docker.com/r/jrcs/crashplan/
docker run \
-d \
--name=crashplan \
--restart=always \
-h $HOSTNAME \
-p 4242:4242 \
-p 4243:4243 \
-v /volume1:/volume1 \
-v /volume1/crashplan:/var/crashplan \
jrcs/crashplan:latest

Do you use features? It's great, but can become quite the beast when you run a "drush cc all" or a "drush fra" - and today we figured out why the "drush cc all" is an issue. Because when hook_flush_caches() runs, if you don't explicitly define this as FALSE, it will run the "fra" for you inside of your cache clear!

Add this to your settings.php:

$conf['features_rebuild_on_flush'] = FALSE;

Since we run our fra separately, we've disabled this, and noticed quite a reduction in time to "flush caches" (which was not only flushing caches but also reverting features, apparently!)

It's that unknown-to-us-before-today variable in the snippet below...

/**
 * Implements hook_flush_caches().
 */
function features_flush_caches() {
  if (variable_get('features_rebuild_on_flush', TRUE)) {
    features_rebuild();       
    // Don't flush the modules cache during installation, for performance reasons.
    if (variable_get('install_task') == 'done') {
      features_get_modules(NULL, TRUE);
    }
  }
  return array();
}

The good news however is that it does support custom run IDs to be named in the save_run() method. Awesome! Except for an odd "bug" or "feature" - a custom run ID won't display in the runs UI if it has non-hex characters in it (see the end of this post for information about that.)

This makes profiling specific requests a lot easier - you can put in a unique query string parameter and see it come up in the run ID, if the URL isn't already obvious. You could also simply define a standard query string parameter for your users/developers to use instead of the entire [scrubbed] URL as a run ID. Since I run a multi-tenant development server with a bunch of developers each having the ability to have a bunch of unique hostnames, it makes the most sense to use every piece of the URL for the run ID.

To start using custom XHProf run IDs, enable XHProf the standard way, at the earliest point in your application (at the top of the front-controller index.php, for example):

if (isset($_GET['xhprof']) && !empty($_SERVER['XHPROF_ROOT']) ) {
  include_once $_SERVER['XHPROF_ROOT'] . '/xhprof_lib/utils/xhprof_lib.php';
  include_once $_SERVER['XHPROF_ROOT']. '/xhprof_lib/utils/xhprof_runs.php';
  xhprof_enable(XHPROF_FLAGS_CPU | XHPROF_FLAGS_MEMORY);
}

The key to the custom naming is is when it saves the run output. In this example, I made a simple function to take the URL, remove all special characters and change them into dashes and remove any repetitive "dashing", and assign that as the third parameter to the save_runs() method, which is the "run ID" name.

if (isset($_GET['xhprof']) && !empty($_SERVER['XHPROF_ROOT'])) {
  function _make_xhprof_run_id() {
    if (isset($_SERVER['HTTPS'])) {
      $run_id = 'https-';
    }
    else {
      $run_id = 'http-';
    }
    $run_id .= urldecode($_SERVER['HTTP_HOST'] . '/' . $_SERVER['REQUEST_URI']) . '-' . microtime(TRUE);
    $run_id = trim(preg_replace('|([^A-Za-z0-9])|', '-', $run_id), '-');
    while (strstr($run_id, '--')) {
      $run_id = str_replace('--' , '-', $run_id);
    }
    return $run_id;
  }
  $xhprof_data = xhprof_disable();
  $xhprof_runs = new XHProfRuns_Default();
  $run_id = $xhprof_runs->save_run($xhprof_data, 'xhprof_testing', _make_xhprof_run_id()); 
}

If you wanted to use a simple query string parameter, I would still use the same type of safeguards so that the filename comes out in a sane way (no high ASCII or other characters which the filesystem wouldn't handle well) - for example, using the "run_id" parameter (I haven't tested this code, but it *should* work :))

if (isset($_GET['xhprof']) && !empty($_SERVER['XHPROF_ROOT'])) {
  function _make_xhprof_run_id() {
    // return null and it will handle it like usual
    if (!isset($_GET['run_id'])) {
      return null;
    }
    $run_id = trim(preg_replace('|([^A-Za-z0-9])|', '-', urldecode($_GET['run_id']) . '-' . microtime(TRUE)), '-');
    while (strstr($run_id, '--')) {
      $run_id = str_replace('--' , '-', $run_id);
    }
    return $run_id;
  }
  $xhprof_data = xhprof_disable();
  $xhprof_runs = new XHProfRuns_Default();
  $run_id = $xhprof_runs->save_run($xhprof_data, 'xhprof_testing', _make_xhprof_run_id()); 
}

NOTE: Both of these methods require one thing to be fixed on the display side. Currently, the XHProf display UI that comes with the stock package *will* list all the XHProf runs, but won't load them if there's non-hex characters in the run ID. I don't know why this naming limitation is being forced. I've filed a bug[1] for now to ask why, or to propose removing the restriction (ideally) - however, for now I've commented out those three lines and everything seems to work fine so far.

[1] https://github.com/phacility/xhprof/issues/58

Anyone who thinks they have privacy on a third party service needs a reality check.

As Scott McNealy said in 1999, "You have zero privacy anyway. Get over it." Anyone who still believes using a service someone else owns (especially for free) entitles them to privacy is sadly mistaken. Most services will try to do the right thing, but there will always be a chance of an accidental data breach or hack, even from the most trustworthy services. You can only hope that the company wants to do good by the consumer.

Simply put, if you don't control it, you can't expect it to be controlled for you.

There are two primary complaints I've heard during this whole Messenger debacle.

The first is that Messenger terms now allow them to spy on you and you can incriminate yourself by using it. First, companies are required to comply with law enforcement if they want to do business in this country. Second, if you are doing anything illegal, you shouldn't be discussing it on something someone else owns anyway. This has always been the case with your telephone, text messaging, etc... that's just common sense.

The other complaint is about unrestricted access to your camera and photos and contacts. Once again, that's nothing new - you authorize your apps all the time for access. Let's take some popular apps for example: Snapchat? You probably authorized it for camera, photo album and contact access. Apps need photo album access to be able to get past photos. They need camera access to take pictures/videos inside of the app. Instagram? You most likely authorized those for camera and/or the photo album too. You probably authorized LinkedIn to your contacts. You most likely authorized the original Facebook app to your camera and photo album (otherwise you can't post any images from inside the application!) - you probably even authorized it to access your contacts.

I feel bad for Facebook having to deal with such unoriginal claims. Yes, it sucks to have to install yet another app, but they have reasons for it and it works well in conjunction with the original Facebook app. It's free, and it isn't a space hog, so there's no real "cost" associated. Battery drainage is the only complaint I consider to be reasonable.

Just remember - nothing is truly private. Even your encrypted end-to-end messaging - someone can take a screenshot or save it and share it. It comes back to what Jon Voight said in Enemy of the State, "The only privacy that's left is the inside of your head."

Back to compiling eggdrop. When it gives that annoying paragraph:

configure: error:

Tcl cannot be found on this system.

Eggdrop requires Tcl and the Tcl development files to compile. If you already have Tcl installed on this system, make sure you also have the development files (common package names include 'tcl-dev' and 'tcl-devel'). If I just wasn't looking in the right place for it, re-run ./configure using the --with-tcllib='/path/to/libtcl.so' and --with-tclinc='/path/to/tcl.h' options.

See doc/COMPILE-GUIDE's 'Tcl Detection and Installation' section for more information.

Install these 2 packages (and let the dependencies also install), and then this configure string works. For some reason, we no longer have a plain old libtcl.so, and eggdrop requires BOTH of these options to be provided or neither one works...

sudo apt-get install -y libtcl8.5 tcl8.5-dev
./configure --with-tcllib=/usr/lib/x86_64-linux-gnu/libtcl8.5.so --with-tclinc=/usr/include/tcl8.5/tcl.h

Oh yeah, this does assume you're running x86_64. Because there's really no reason you wouldn't be anymore.

Old:

description "Run docker"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn

script
  /usr/bin/docker -d
end script

New:

description "Run docker"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn

env HTTP_PROXY="http://your.address:port"
env HTTPS_PROXY="http://your.address:port"

script
  /usr/bin/docker -d
end script

Here is a quick-and-dirty way to get the number of results based on an empty search (i.e. the entire index) for a single Solr core on localhost. Maybe this will help somebody else out there. I suppose it could be paramterized with hostnames, search strings, etc... and it wouldn't be that hard either from what it looks like.

Enjoy.

Filename: solr.rb

class SolrResultCount < Scout::Plugin

  needs "rubygems"
  needs "json"
  needs "net/http"

  def build_report
    url = "http://localhost:8983/solr/select?q=&rows=3&fl=bundle&wt=json"
    r = Net::HTTP.get_response(URI.parse(url))
    parsed = JSON.parse(r.body)
    report(:results=>parsed["response"]["numFound"])
  end

end

I originally tried scponly and scponlyc (which I've used in the past) and rssh, however none of them worked properly for me.

Sure enough, the openssh package from optware worked right out of the box.*

wget http://wizjos.endofinternet.net/synology/archief/syno-mvkw-bootstrap_1.2-7_arm-ds111.xsh
ipkg install openssh openssh-sftp-server

Then edit /opt/etc/openssh/sshd_config, and put in:

Match User username
        ChrootDirectory /some/directory
        ForceCommand internal-sftp

Also edit the user account in /etc/passwd, change the home dir to the /some/directory, and give it "/bin/sh" for a shell.

Viola... when sshd is restarted next time it will just work.

The guys at optware made a neat startup script that will start their sshd on boot. So nothing to do there.

Make sure to disable synology's built-in ssh (Control Panel > Terminal) or you'll probably be hitting the wrong one!

If you are concerned about privileges, the way that Synology runs its units isn't very UNIX permission friendly (most files are world writable on the filesystem, and the expectation is the daemons will properly control the access.) I wound up creating a little cron job that chmods and chowns files to keep the secondary account I've created to be a "read only" account to that directory.

* As always with my tips, YMMV - this worked fine on my Atom-based DS2411+ unit. but when I tried the same setup on a DS213, it didn't seem to work. No idea why, there aren't much diagnostics or logs provided to use. Sorry.

UPDATE: After running this on the "working" NAS unit for a bit, it stopped working. The culprit was the ChrootDirectory became owned by the user, not by root:root. Changing it back (chown root:root /some/directory) fixes that. So it looks like OpenSSH wants that in place for the chroot stuff to work. That could have been the issue mentioned in the previous paragraph (couldn't test it anymore)

It is so easy to keep adding new items, and it isn't as easy to mark them off. Tasks change status or become mini "projects" with multiple steps (get car oil changed = schedule oil change appointment, which leads to an appointment being booked, which will be the item until it can be marked complete.) At some point I really want to visit the whole "to-do item status" concept as well.

Anyway, this is how the inner workings of my engineer brain function. Some tasks are insanely simple, but mixed in with more complex ones that require perquisite tasks, specific times of the day (business hours for example), specific people, or specific locations that may or may not be possible to get to easily.

I've struggled with trying to tame the never-ending lists. Last week I had a night open and plans to be really productive and knock some things off my list - which wound up not happening, but other "productive" tasks did get taken care of. Those were not planned, but still helpful. Someone said "what a productive day" and I felt like it wasn't the "productive" I actually wanted.

Working in the world of software/web development, this kind of stuff has parallels in the engineering world. I guess you can call that "technical debt" - new stuff is coming up and old stuff isn't being taken care of.

That won't work for me. I need to be making progress, I've got a lot of tasks that need to be finished. That's why they made the list to begin with.

While I've always had the desire to finish these things, I haven't had the proper personal accountability for actually checking things off the list(s) - I love marking things done, but I don't make it a regular habit of checking them enough.

Enter Beeminder - a personal accountability system for reaching goals. A co-worker introduced me to the site. I started thinking about the usual suspects - like losing weight and thought about some of the things he had put in - like making sure he spends X hours a day being productive on personal projects. What I liked most is the work spent on the system to make it statistically sound, measurable (a goal needs to be measurable), capable of automating, and with appropriate notifications to remind me to log whatever data isn't being supplied automatically via other devices or services (your scale can talk to it, for example.)

I started thinking about other ideas to put down as goals. One of them was being "productive" by marking one thing off my to-do list per day. However, that can vary. Some items are easier than others, and what if I have a day where other "productive" tasks come up that take away from that list (as a lot of days do...) - so I began thinking more about it.

I came up with the idea that each task should have some sort of amount of effort associated to it. The effort is derived from the amount of work the task requires and/or the amount of coordination due to location, time constraints, people constraints, etc. For example, getting my passport - that has been on my list for over a year. I paid for it, filed the paperwork, just need the photo and actual submission done. I tried a few times last year but never had all my ducks in a row, and then it fell by the wayside (I had no real need for it and got busy with other things.)

Being a obsessed with completing things like I am, it's still on my list, and I want to get it done. I won't be surprised if my payment is now forfeited due to government accounting needing to close the books each year or something like that. Nevertheless, it's something I should have anyway, it's staying on the list. That would get a value of 1.0 - the highest a task can have, in my world. It requires finding a place that is open, a place that does photos, possibly trying to haggle with them to honor my payment made last year, etc. - a lot of possible effort there, along with time constraints (business hours, sort of) and location (specific locations handle specific things.)

Once I thought about it in that way, I could put in a Beeminder task of "finish at least 1.0 units of effort per day" or something. I started thinking of other tasks on my list, and it looked like that would be an easy way to try to knock things out. It almost becomes a game.

Then I started expanding the idea further. Those random unplanned tasks that come up, such as helping my Parents move? That took a lot of physical work and time. I can't really be taking care of other items if I am busy helping them out. I don't want to be penalized for it. So we can introduce the concept of "bonus units" - I could say "I didn't do anything on the list, but that was definitely worth 1.0" - and feel like the day is fulfilled still.

I believe every day people should do something to advance their life (or someone elses), and this way of tracking makes it easier to be accountable.

Extending it even more, random daily tasks that may or may not be done, based on laziness, distractions, whatever - those can have some value too. Maybe you don't go to the store that often - so going to the store is 0.2, or getting a haircut is 0.2. What about laundry? Those things do count. They are productive. Most of us probably wouldn't think of giving ourselves some credit for them when introducing something as detailed as this.

Those people who prefer the stick vs. the carrot could even put in negative units. Maybe you want to stop watching so much TV, so you deduct 0.1 units per hour. Combine that with earning 0.2 units per hour of gym time, and you are doing the equivalent of calorie counting but with productivity.

The gamification makes it kinda fun, and it might make it easier to adopt as part of a daily routine. Tying it to a system such as Beeminder could be useful too, if you like their method of being punished for non-compliance. Some of us need that, and I may wind up using their site for tracking each day's units.

I tried to come up with a name for a unit, the best I could come up with was "Personal Productivity Unit" (or PPU, since it needs an acronym) - but it was an idea that has been brewing for a while, and I think I have come up with enough structure now to give it a shot in real life and see how it fares.

Finally, I am sure I am not the first one to think of something like this, there are probably books written on this, but I haven't seen anything myself and it makes my engineering brain happy to be able to weigh tasks and set a goal based on that. If anyone else has any more ideas on the subject, I'm all ears!

I was trying to figure out the best way to launch Apache Solr on Ubuntu - and was having issues finding a nice clean way to do it. I decided after some misc init scripts, that I should look at Upstart. Thankfully someone (mentioned below) already had a working script available to start from.

Assumes:

• Solr will be running as user "solr" group "solr"

• Solr's root where start.jar is located is /home/solr

• This works well using Solr 4.3.1, Ubuntu 12.04.2 LTS (precise)

Put this in /etc/init/solr.conf:

start on runlevel [2345]
stop on runlevel [!2345]

kill timeout 30
respawn

setuid solr
setgid solr

script
   chdir /home/solr
   exec /usr/bin/java -jar start.jar
end script

Big thanks to Eric Wilson's blog for the initial script. I tweaked it for my specific user/group/location desires.

After getting a stable setup going, the last piece of it was getting a remote that they could manage. I was nervous at first because of compatibility concerns and "bleeding edge" software. However, I picked up an "Adesso ARC-1100 Media Center Remote" - which I had seen as being compatible on the XBMC forums, and out of the box, it works great. Truly plug and play.

I wanted to pick up something reasonably priced and locally (just in case I had to return it, I didn't want to hassle with RMA processes.)

The main issue with the remote is that some buttons are a bit confusing - which is tolerable, but the main offender is the power button. It works out of the box, as one would expect, except that once you power down the OpenELEC/RPi combination, you can't power it back up again (at least not with a USB-powered IR receiver...)

To solve that issue I looked into figuring out how to do some key remapping, to make the power button do something different, or nothing at all. The documentation seems straightforward but made me dizzy for a moment (and is slightly incorrect where remote.xml is), but on the first try I had success. After that, I decided to mess around and see what other buttons I could "disable" essentially (and I decided to see if <null> could be used to map the key to nothing - it seems to work)

Here is the output of that work, it looks simple enough. Put this under /storage/.xbmc/userdata/Lircmap.xml (or via the userdata SMB share)

  
    KEY_POWER 
    KEY_DELETE 
    KEY_SUBTITLE 
    KEY_VOLUMEDOWN 
    KEY_VOLUMEUP 
  

Sure enough, it all works - now I have a remote solution for them that won't let them get into trouble, and confidence now on how to map keys in the future.

After playing with XBMC on a Pivos AIOS DS, Intel NUC, and OpenELEC on an RPi, I have to say I am happy to see the extensive community involvement in the XBMC project and its derivatives.

Now, I just wish skinning it was as easy...

It is supposed to be totally unlimited as well, and they only charge $12.99 or something for 2-10 computers, vs. a per-computer model from BackBlaze. Also, they don't list support for file shares, but it had no problem backing up one of my samba mounts. (Please don't fix that if it's a bug!)

So while I still consider BackBlaze to be more efficient and easier to use (just "set it and forget it" I will say that CrashPlan has a lot more options, is an opt-in policy (akin to Mozy, etc.) instead of an opt-out by default policy (BackBlaze) and it makes it really easy to list the entire filesystem, and select/deselect at any level of it.

The other interesting/neat thing is you can set it up to backup to friends machines, local storage, attached drives or their CrashPlan Central cloud (which is what they charge the monthly for.)

Since BackBlaze isn't playing in the Linux space yet, and has special ways to check if a filesystem is "local" and such, it looks like I will be using the best of both systems for now. The Java UI does feel a bit "Java-ey" but the price, features and performance of the actual network backups seem well worth it.

So +1 to CrashPlan!

So I plead with you, companies, please invest just that little bit more to get higher quality stock. It can't be that much more, and you'll have a greater chance that people will wear those shirts sporting your logo, and not because they picked it up at the local Goodwill. :)

On another note, I apologize for not posting anything in almost seven months. I've been busy and haven't felt like wordsmithing anything I've been up to, or otherwise haven't had anything useful to share.

This is as of PHP 5.4.0 RC6. Note that I didn't put in anything related to OO, as I despise the obsession with OO now in PHP and do my best to live without it.

Misc. notable changes:

• Added built-in web server that is intended for testing purpose. (Moriyoshi)
• Changed default value of "default_charset" php.ini option from ISO-8859-1 to UTF-8. (Rasmus)
• Added array dereferencing support. (Felipe)
• Added header_register_callback() which is invoked immediately prior to the sending of headers and after default headers have been added. (Scott)
• Changed http_response_code() to be able to set a response code. (Kalle)
• Added new json_encode() option JSON_PRETTY_PRINT. FR #44331. (Adam)
• Changed silent conversion of array to string to produce a notice. (Patrick)
• Removed support for putenv("TZ=..") for setting the timezone. (Derick)
• Removed the timezone guessing algorithm in case the timezone isn't set with date.timezone or date_default_timezone_set(). Instead of a guessed timezone, "UTC" is now used instead. (Derick)
• ext/mysql, mysqli and pdo_mysql now use mysqlnd by default. (Johannes) )I think this is in 5.3 though too?)
• Expose session status via new function, session_status (FR #52982) (Arpad)
• Added support for storing upload progress feedback in session data. (Arnaud)

PHP-FPM related:

• Remove EXPERIMENTAL flag. (fat)
• Added partial syslog support (on error_log only). FR #52052. (fat)
• Lowered default value for Process Manager. FR #54098. (fat)
• Enhance security by limiting access to user defined extensions. FR #55181. (fat)
• Added process.max to control the number of process FPM can fork. FR #55166. (fat)
• Dropped restriction of not setting the same value multiple times, the last one holds. (giovanni at giacobbi dot net, fat)

Removed legacy features:

• Safe mode and all related ini options. (Kalle)
• register_globals and register_long_arrays ini options. (Kalle)
• import_request_variables(). (Kalle)
• allow_call_time_pass_reference. (Pierrick)
• Session bug compatibility mode (session.bug_compat_42 and session.bug_compat_warn ini options). (Kalle)
• session_is_registered(), session_register() and session_unregister() functions. (Kalle)
• y2k_compliance ini option. (Kalle)
• Removed magic_quotes_gpc, magic_quotes_runtime and magic_quotes_sybase ini options. get_magic_quotes_gpc, get_magic_quotes_runtime are kept but always return false, set_magic_quotes_runtime raises an E_CORE_ERROR. (Pierrick, Pierre)

The project is here: https://code.google.com/p/phpreboot/

Why is this an issue? Well, for one, it's NOT A REBOOT OF PHP. It's a frickin' Java-based re-implementation of some PHP ideas and function names with a completely different syntax, and at the end of the day, it has 99.9% nothing in common with PHP.

Why do people develop in PHP? Because it is PHP. Stop trying to make PHP more like Java, more JSON-y, etc. Why did it become the world's most popular language? Besides for being easy to pick up (too easy, sometimes, which leads to a bunch of garbage and unsecure code), it got there because of what it is.

This "php.reboot" project is just trying to use PHP's popularity and function names to get people to check it out. PHP doesn't have XML/JSON/SQL style constructs (although the new array syntax sure does look like an attempt to emulate JSON, cough), it has structure - that's what "$" and ";" are for - denoting specific constructs in the language. If people don't want to develop using "$" or ";" go switch to another language that doesn't, that is already established.

I am tired of seeing blog posts and other items pop up every so often "why PHP wants to be more like Java" or "10 things PHP can learn from Ruby" - if you're trying to adapt PHP to another language, just use the other language. Period.

Some of the ideas in this project might be neat, or good; but in the end, it's not a "reboot of PHP" and stop labeling it as one.

I, for one, am excited to see things like a roadmap and making it easier for contributors to help out.

Read Igor's post here.

He hasn't setup a web-based SVN option yet (cough, cough, I bet it would be if nginx supported SVN's DAV needs :)) - but you can hit it up using a native SVN client: svn://svn.nginx.org - note, I would only grab "/nginx/trunk" as otherwise you'll get a few years worth of code, if not the entire nine years.

At first I thought this was a joke, seeing the announcement for 1.0.0 - sounds like it just started. But one must remember, he's always used pre-1.0 for past versions.

Update:

MTecknology maintains a PPA and has already built a package for 1.0.0. The quick and dirty method to update:

apt-get install python-software-properties
add-apt-repository ppa:nginx/stable
apt-get update

This will provide you a handful of new packages, by default the "nginx" package is a dummy package that will provide "nginx-full" and should be the only one you really need to care about.

nginx-dbg - Debugging symbols for nginx
nginx - small, but very powerful and efficient web server and mail proxy
nginx-common - small, but very powerful and efficient web server (common files)
nginx-doc - small, but very powerful and efficient web server (documentation)
nginx-extras - nginx web server with full set of core modules and extras
nginx-extras-dbg - Debugging symbols for nginx (extras)
nginx-full - nginx web server with full set of core modules
nginx-full-dbg - Debugging symbols for nginx (full)
nginx-light - nginx web server with minimal set of core modules
nginx-light-dbg - Debugging symbols for nginx (light)

I didn't really need this to do much. The code is simple and there is no UI to manage it, but it works, and even gives you cute little X-Redirect headers to let you know if it was executed and if it found a match. It would be easy enough to take that if() out and have it check any URL (just be sure to remove the fallback :))

function foo_init() {
    if(stristr($_SERVER['REQUEST_URI'], '.php') || stristr($_SERVER['REQUEST_URI'], '.htm')) {
        $old = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);
        $new = db_result(db_query("SELECT new_url FROM custom_redirect WHERE old_url = '%s'", $old));
        if($new) {
            watchdog('foo', $old.' found in the custom_redirect table', NULL, WATCHDOG_INFO);
            header('X-Redirect: Found');
            drupal_goto('http://'.$_SERVER['HTTP_HOST'].$new, '', '', 301);
        } else {
            watchdog('foo', $old.' NOT found in the custom_redirect table', NULL, WATCHDOG_ERROR);
            header('X-Redirect: Not Found');
            drupal_goto('http://'.$_SERVER['HTTP_HOST'].'/', '', '', 302);
        }
    }
}

The table?

CREATE TABLE custom_redirect (
  old_url varchar(150) NOT NULL,
  new_url varchar(150) NOT NULL,
  PRIMARY KEY (old_url)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

Enjoy. Totally could have Drupal'ed that up and made a hook_install() for the schema too, right? :p

I didn't do the math, but it would have probably taken over 10-15 hours to restore the database from the mysqldump. There's a couple easy tweaks I did not use. For one, I used --skip-opt and made my mysqldump files full INSERT statements (for verbosity and the ability to "diff" them if I ever needed to) - this was stolen from a backup script I wrote.

If you read the documentation/blogs, it says to use --opt when running mysqldump for faster imports. Well, duh! While I was at it, I also tweaked a couple other things. Right now it is moving MUCH faster. What did I do?

• On the source, I used mysqldump --opt (it seemed to dump the database faster too)

• On the destination, I set innodb_flush_log_at_trx_commit to "0" in my.cnf for the time being. This server isn't used yet, so that's safe.

• I also put "SET AUTOCOMMIT=0;" at the top of the script, and "COMMIT;" at the bottom of the script. I don't need any commits until the end, this is a fresh import.

The results are not very scientific, but here's how it breaks down so far (still in the middle of the process)

• Without these tweaks, at 107 minutes it was only at 2.2gb out of 13gb.

• Without these tweaks, at 12 minutes it was at 4.5gb out of 13gb.

I think this will save my bacon, I wish I had done this sooner and not wasted that two hours originally. Someone in #mysql recommended I look at XtraBackup, but it seemed like too much to learn and attempt my first run at it while I was having to do a production migration.

This is cool, and I wasn't sure it actually got in a build or not, I remember it was mentioned or discussed, but sure enough, Jérôme coded it and got it in to FPM in core. So it is available to all, and he welcomes the feedback - see this thread on the nginx mailing list.

Due to limitations in the FastCGI protocol, you have to pass all the parameters you want as a single string, separated by "\n" - it's not the cleanest looking configuration, but that's how it is for now. I spitballed a different approach to it, but at the moment I believe it will be unlikely to get adopted.

If you're like me, you're probably looking for the code samples almost immediately. Here's your examples. Note that PHP_VALUE and PHP_ADMIN_VALUE are both legitimate keys, the difference being that PHP_ADMIN_VALUE does not allow the user to override the value using ini_set() - see the manual for more infomation.

Single value:

fastcgi_param PHP_VALUE sendmail_from=direct@prout.com;

Multiple values, manually line-broken:

fastcgi_param PHP_VALUE "sendmail_from=chroot@prout.com
precision=42";

Multiple values, with the \n in there to clearly call it out:

fastcgi_param PHP_VALUE "sendmail_from=chroot@prout.com \n precision=42";

More information is available in the feature request on the PHP bug tracker.

Due to the fact that their service always uses over 100 megs of RAM, and seems to continuously get stuck on certain files, I was planning on getting rid of it soon. Today's announcement made this decision even easier though, as now they've decided to go the way of other companies with tiered pricing models. With how cheap technology continuously gets, any company marking prices up really pisses me off.

So, I give a profane salute to you, Mozy, as you have now joined the ranks of companies I feel personally displeased with, and definitely will not recommend (not that I really did anyway.)

Even AT&T (one of the main companies I despise) let people grandfather in their unlimited plans, and cell networks take a lot more beating than a backup service with hard drive prices going down every day. Adding more servers to a rack is a lot harder than adding cell tower capacity. That type of "next month you'll be forced to change" does not sit well with me.

Dear Michael,

Thanks for being a valued Mozy subscriber. For the first time since 2006, we're adjusting the price of our MozyHome service and wanted to give you a heads up. As part of this change, we're replacing our MozyHome Unlimited backup plan and introducing the following tiered storage plans:

50 GB for $5.99 per month (includes backup for 1 computer)
125 GB for $9.99 per month (includes backup for up to 3 computers)

You may add additional computers (up to 5 in total) or 20 GB increments of storage to either of the plans, each for a monthly cost of $2.00.

While this policy takes effect for new MozyHome customers starting today, your MozyHome Unlimited subscription is still valid for the duration of your current monthly term. In order to ensure uninterrupted service, you'll need to select a new renewal plan.

<script type="text/javascript" src="http://www.google.com/jsapi"></script>
<script type="text/javascript">google.load("jquery", "1.4");</script>
<script type="text/javascript">google.load("jqueryui", "1.7.2");</script>

This is a very hacky, no-garbage-collection, but still "working good enough" script. At the advice of #chromium on freenode, when asked about cookie sharing between Incognito windows, I was told it's been discussed before, and I got the information on how to make sure that your Incognito windows don't share information or cookies by forcing separate user data directories.

I'm not really worried about privacy, I'm more annoyed that I launch separate Incognito windows and it shares cookies between them, which is sort of against the point. I have to login to the same sites over and over under different accounts for different clients, and it's a PITA.

Major things to note:

• This assumes you'll run some sort of "temp directory cleanup" tool on your own for Windows. This doesn't have any concept of "oh yeah, I have to cleanup that temp directory I made"

• None of your extensions, bookmarks, settings, etc. will be remembered in this session. It's completely barren.

• You will never (assuming the GUID is unique) get the same session more than once.

As I said, it's hacky, and you'll need to change a couple of the paths. I couldn't figure it out elegantly, and I was getting tired of trying to find script examples on the net (why is it so hard to find code that works together for Microsoft languages?)

Perhaps someday soon Chrome or someone will develop something more robust for this. For now, if you want - this does seem to work, at least on my XP SP3 system.

Note: This is vbscript. Make a file called "incognito.vbs" or something and it should work.

' keep us honest
Option Explicit

' because we have to
Dim strDirectory
Dim strTempDirectory
Dim TypeLib
Dim objFSO
Dim objShell
Dim strChromePath

' change this if you want - anything with spaces has to have be wrapped in triple quotes
strChromePath = """C:\Documents and Settings\mike\Local Settings\Application Data\Google\Chrome\Application\chrome.exe"""
strTempDirectory = "C:\Windows\Temp"

' make a clean guid
Set TypeLib = CreateObject("Scriptlet.TypeLib")
strDirectory = TypeLib.Guid
strDirectory = Replace(strDirectory, "{", "")
strDirectory = Replace(strDirectory, "}", "")
strDirectory = strTempDirectory & "\" & strDirectory

' create the directory
Set objFSO = CreateObject("Scripting.FileSystemObject")
objFSO.CreateFolder(strDirectory)

' launch it, or fail
If err.number = vbEmpty then
   Set objShell = CreateObject("WScript.Shell")
   objShell.run (strChromePath & " --incognito --no-first-run --user-data-dir=" & strDirectory)
Else
   WScript.echo "VBScript Error: " & err.number
End If

' cleanup
Set TypeLib = nothing
set objFSO = nothing
Set objShell = nothing

' quit
WScript.Quit()

Download it here.

I highly recommend upgrading to PHP 5.3.3+ with the more updated/better FPM version (among a million other enhancements) that is bundled in core now. Coincidentally, PHP 5.3.4 came out today as well, with a handful of FPM bug fixes/features (see my other blog post or the NEWS file.)

Enjoy!

• Great service

• Easy-to-use, feature-rich but not overbearing website

• Fast delivery times

• Actually reduced the price of my plan voluntarily

Today I've got some bad news though. I just received an email notifying me that my plan will increase $3/month. Normally I wouldn't be as vocal, but this is on top of the deals they've made with studios to delay releases ~30 days. Netflix claims it will receive huge discounts and the press release for Fox says it will more of their library to be streamed. It's obvious the studios are supposedly hoping this ~30 day window will encourage consumers to rent/buy DVDs normally. Most sites I've read think it will just cause piracy to rise. I'm inclined to agree.

While the movie delay originally annoyed me, it wasn't enough for me to get too angry over. I'm so busy most of the time that 30 days passes me by in the blink of an eye. I don't believe Redbox has that restriction in place, so one could go rent a movie immediately from them still if they wanted.

However, now they are increasing costs on me while delivering less service. That I can't keep quiet about.

So with a heavy heart right now, I have to -2 Netflix. I kept the original movie delay -1 to myself. But now they get both negative marks at once for this new move. I really need to start a point system.

I was worried that if FPM was to ever make it in to PHP core, it would stagnate and become part of such a big machine that it would take forever to receive updates. However, since its adoption into PHP 5.3.3, Jérôme has been hacking away on it and I was happy to see that in 5.3.4RC1, a handful of changes are being included:

• Added '-p/--prefix' to php-fpm to use a custom prefix and run multiple instances. (fat)

• Added custom process title for FPM. (fat)

• Added '-t/--test' to php-fpm to check and validate FPM conf file. (fat)

• Added statistics about listening socket queue length for FPM. (andrei dot nigmatulin at gmail dot com, fat)

• Fixed inconsistent backlog default value (-1) in FPM on many systems. (fat)

• Fixed bug #52674 (FPM Status page returns inconsistent Content-Type headers). (fat)

• Fixed bug #52498 (libevent was not only linked to php-fpm). (fat)

Thanks guys, I am counting the days until I can finally use PHP 5.3. I also discovered the new magic variable DIR exists a couple weeks ago - between everything, I am jonesing to move badly!

(For those wondering, "fat" is Jérôme's pseudonym.)

I just discovered this tool in the last hour. It not only allows you to make wireframes, but actually usable prototypes - with links to external sites and other pages inside of the prototype, pull in external images and content, has layers like photoshop ... the list goes on and on. Best part is the learning curve was quite simple. I found another tool which was an Adobe AIR-based app, if I recall, but it was a bit cryptic and hard to use. This thing allows you to even invite people to do usability testing on your prototype, record their movements, leave comments, etc. Best of all, the cost is extremely reasonable!

There's simply too much to name off and now all I want to do for the next month is prototype out all my ideas!

https://pidoco.com/

Stripping it down to bare bones would reveal a flexible and fairly well designed framework that can be used for anything really.

That's my major beef with Drupal - that it ships with so much content-specific functionality in place. For instance a node is something everyone universally agrees is a "great concept" - a basic item that is extended by custom options (CCK is the model of how easy this is) - but nodes still have a "Title" and a "Body" and publishing options and other fields that shouldn't be there in a truly generic system. The idea of one central basic element though is spot-on in my mind. With all that in mind, I begin my thoughts...

When it comes down to it, almost everything can be considered some form of a "content" item. For example, I designed something to track transactions. At first I cheated and used CCK and just made some custom fields. I also made a version using Drupal's schema API that just made a normal database table, to skip any CCK overhead and node "addons" Drupal glues on by default. However, the overhead aside, I was able to use this system and with not a lot of code, make a fully functional transaction tracking system based off of node concepts. I used CCK as a "schema designing" frontend, and could use Views to get the information back out in a variety of ways.

Is a transaction "content" in the common sense of the word? Not at all. It's a reference to someone purchasing something, in this case a user purchased another type of node - so really it was just a user reference and a node reference. I didn't care about the "Publishing options" or any of the other Drupal-related built-in node specifics, and anything else that may attach itself to nodes - Pathauto, Fivestar, etc.

I [ab]used the node-based system and CCK's UI to make myself a flexible schema to track transactions. I was also able to use pure API to create a database table and manually track the information, and skip any of the hooks related to node processing (Pathauto, Fivestar, publishing options, etc.) - which one was better? Well, I'm still deciding that. By using the node method I can easily expose it in a view. I am not sure I can using custom schema (I think someone told me I could but I haven't tried yet) - I'd really like to use Views for it, and be lazy on the delivery side of it. By going custom, I track only what I want in the database and save probably many levels of pre- and post-processing on the result set.

Getting back to my original statement - Drupal's foundation is pretty solid. It enables anything to be accomplished, it's just that damn middle layer that I wish would be more decoupled and moved into modules that can be enabled or disabled. For the most part, people who use Drupal seem to use it mainly for it's CMS/"Content Management Platform" capabilities just fine. Some of my sites require custom work though, and that is where I enjoy the framework/platform's underlying capabilities.

Typically when developing a site, there are common elements such as user login/registration/roles and management, form processing, XSS/CSRF/other attack vector protection, output encoding, data and request normalization, etc. I've built my own little library of functions and even a bit of a stub framework to use them which do a lot of this. Drupal however has already generalized most of this and provides it to you for no cost.

Frameworks in the typical definitinion such as RoR, Symfony, and half a million others give you the basics, but don't give you that much. It's still up to you to create everything else. While I may whine about Drupal's content-heavy core packaging, it does provide some built-ins that I do find useful to be included. So I guess that can be chalked up to some personal opinion - I would rather the it push the content stuff out to modules (and strip nodes down to even simpler basic structure) yet keep the user registration, login, role and capabilities and form processing available.

All in all, I am having fun looking through the directories of modules out there, and trying to approach things with a more Drupal-eqsue model in mind, understanding that I will sacrifice some performance and add some overhead, but also save myself the headache of all those tedious user submission and form processing screens and such. That's usually where typical self-coded projects seem to start trailing off for me, is winding up getting into the user interaction piece of it. Database design, delivery - that's easy. Form processing and user experience? Annoying as hell :)

OS X users using Safari wound up with a proxy issue after clicking on a button on our site, and reloading the page.

The suspect seemed simple to me - the button has to do something persistent that makes the browser behave differently the next page load - so a cookie or a session-based thing on the server side which scrambles the request. Session stuff seemed incorrect as it should show a PHP error or something, not actually make the proxy reject the request. So I came to the easy conclusion of going down the cookie route.

Sure enough, it was an onclick function which set a cookie - one that worked in every browser including Safari on Windows*, but broke on Safari on OS X. So we looked to see what the cookie was. Turns out, Safari on OS X at the moment does not like when a cookie's value ends with a comma "," - it literally breaks cookie handling altogether. It sends some request that our proxy considers invalid and rejects, and checking without the proxy involved showed us that some cookies (if not all of them) were not being sent to the server (or at least not properly) - which included our authentication cookies. The minute we nuked this "bad" cookie, our other cookies got read properly, the proxy allowed the request, etc.

So, for those people who like to see things more direct and don't like to read a lot of fluff (me included!) here's a simple breakdown:

BREAKS: CookieName=Foo,Bar,Baz,
WORKS: CookieName=Foo,Bar,Baz

Pretty simple. Odd that it appears to be inconsistent across platforms. Anyway, have fun with that little nugget of information. If I wasn't lazy I would look into exactly where to submit a bug, somewhere in the WebKit community I assume.

* I cannot confirm that Windows did not suffer from this, a coworker provided that information.

However, if you talk to people who know the internals of PHP they'll tell you there's a lot of ugly stuff in there. That it's a language based on macros, etc. I don't necessarily care about that. My experience is from a user perspective, not an internals one. That being said, just from my higher level interaction with the language, these are some of the things I'd love to change.

• Make function name conventions consistent. Some functions have underscores, some don't. strpos vs. str_replace, html_entity_decode vs. htmlentities, etc.

• Make argument order consistent for similar types of functions. Depending on what you're doing, it's one or the other. in_array($needle, $haystack) vs. strstr($haystack, $needle), etc.

• Optimize the core. Strip the core down more, and push more things into modules. Enable some of them by default, fine. But when it comes down to it, I don't need easily 30-40% of the functions that PHP has built in.

• Combine similar functions and use arguments to define the behavior. For example addslashes() and addcslashes(). Make it one function with a constant to define its behavior.

• Disable magic quotes (preferred) or enable it and don't give any option to change it. As far as I'm concerned as long as you pick one route, you can guarantee universal compatibility, whether that means using magic quotes, or not using them and expecting developers to understand input sanitization, sanity checking/type checking/all that jazz. Which I don't think is a bad thing.

• Implement a "strict" mode. "PHP is lazy" as Rasmus says which is fine and all, but I don't like the PHP name shamed with terms like "insecure" - any code can be insecure in any language, however, PHP is so easy to pick up and get things going that it makes it too easy to write crappy and insecure code. Specifics on a "strict mode"? I've got none. It's late and I can't think of how I would enforce better coding practices in core...

• Get rid of $_REQUEST. I've advocated this for years and even unset($_REQUEST) in my code. To me it's a lazy person's workaround for coding and introduces some of the same vectors that were closed when disabling register_globals. If you -really- want to have a $_REQUEST type mechanism in your code, just array_merge($_GET, $_POST, $_COOKIE, etc) in whatever oder you want. I dislike using software that uses $_REQUEST by default but doesn't actually need the flexibility of POST vs. GET vs. COOKIE and such. Know which input stream your data is coming from, if nothing else, it will at least make replay attacks and such much harder for people to craft.

• Get rid of objects and OO stuff. Yeah, I said it. Everyone loves OOP. Why? While I see the power of being able to extend classes, I also see it seeming to be the most troublesome when it comes to compatibility checks, all the APC crashing or odd bugs I've suffered from were due to it. If you look at something like Drupal, they've figured out how to extend or override using procedural code quite well. Sadly, even they're converting more things to OO as well. IMO, OOP is more suited for longer-running applications, perhaps something event driven where a new object to represent a connection is created (however, C's been doing this without dealing with objects forever, it doesn't HAVE to be OO...) Those are the two main examples I see for using OO. Disclaimer: I wasn't raised in an OO environment, this is all based on personal experience and preference. :)

I've memorized the function list for what I use pretty well (like I said, I probably only use a subset of the functions in PHP) however the most annoying thing is when it comes to the needle vs. haystack argument positioning. I usually have to reference php.net for it. Sometimes I can trial and error though. In an ideal world, I wouldn't have to.

It would be great if something like PHP 6.0 would adopt some of these practices, since it is a major version change. Perl, Ruby and Python I believe have all done similar things where a major change really was a dramatic change and required conversion of code to meet its new requirements.

I'm sure this list could grow, and I may add to it. Who knows.

Choice items such as:

• Length of account history is too short.

• Too long since most recent account established.

What the hell? Don't those basically contradict each other?

But wait, there's more:

• Too many recent inquiries.

• Length of account history is too short.

• Too many accounts with balances.

Sounds like I am being encouraged to open a fresh account, because my "most recent account" was created too long ago. But too many new accounts dings me, on the flip side. I have too many accounts at the same time though. It's a catch 22.

However, the major complaint I have is that applying for loans/doing any sort of shopping that requires someone to pull your credit winds up hurting your credit score. This isn't anything new, but I simply do not get it. Shopping around and applying to things should have absolutely nothing to do with your score. That is like being charged 25 cents for looking at a carton of milk at the store but not buying it. Or having to pay to window shop - before you know whether or not you can even afford what you're looking at. Your credit score should only be altered when your credit situation changes. Not for simply checking to see if you can save money by refinancing or shopping for auto loans.

Unless you are extended a new credit line or something else, there should be no change in score. Especially nowadays with so many companies offering specials that require you to apply, shopping for a refinance (which can be turned down easily) costs you points on your credit score. Shopping does not mean you're actually receiving anything.

Also, I am told that "inquiries within the same industry within the same 45 day period only counts as one pull". I call bullshit. I am sure they're still counting each one somewhat.

While I do believe a credit rating system is valuable in some form, ours definitely needs to be overhauled. With the economy moving so slowly, people are looking for various ways to get more money out of their home or car, applying for more credit cards, etc. Regardless of the approval, it is messing with your credit just for shopping.

Suggestions

• Having your credit pulled should have no adverse affect on your score. Period.

• The time ranges need to be adjusted. "Too long since your latest account was opened" - that's a new one for me. Really contracts the hell out of the "Length of account history is too short" notice.

• Consistency across different bureaus - some tell me I don't have enough accounts. Some tell me I have too many. It is like trying to please a committee - you can't please everyone all at once.

I've had credit being built for 10 years, a couple different credit cards, bank accounts and loans. By now I should be receiving no negative marks for anything I've done relating to too many accounts, not enough accounts, too much time, not enough time, etc. However, it seems like raising the number takes an infinite amount of time, but lowering it happens almost immediately when your credit is pulled.

Once again, a sincere disappointment with how a system is run. I also do not see the reasoning behind some of the rules, either.

Because I still have to maintain PHP 5.2.x for clients, I have decided to try building everything in self-contained in an /opt/php53 directory. So consider that my $prefix, and change appropriately.

I have the following setup, and it seems to work great so far.

/opt/php53/etc/php-fpm.conf:

[global]
log_level = notice
error_log = /opt/php53/var/log/php-fpm.log
pid = /opt/php53/var/run/php-fpm.pid
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 5s
daemonize = yes

; pools
include = /opt/php53/etc/fpm.d/pools/*.conf

One file per pool, for example, a pool named "mike" -

/opt/php53/etc/fpm.d/pools/mike.conf:

[mike]
listen = 127.0.0.1:11000
user = mike
group = mike
request_slowlog_timeout = 5s
slowlog = /opt/php53/var/log/slowlog-mike.log
pm.max_children = 5
pm.start_servers = 3
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.max_requests = 500
include = /opt/php53/etc/fpm.d/common.conf

Common elements for each pool (if these could be inherited globally, which they MIGHT be, I could just toss them in the main php-fpm.conf. Perhaps a feature request. Will post on the mailing list...)

Remember that rlimit_files needs to be something set in your sysctl.conf or on the system level or you'll get that RLIMIT_NOFILE warning. Also, depending on how you want to limit resources per pool/client, you may want to tweak things, such as request_terminate_timeout.

/opt/php53/etc/fpm.d/common.conf:

listen.backlog = -1
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.status_path = /status
ping.path = /ping
ping.response = pong
request_terminate_timeout = 120s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes
env[HOSTNAME] = $HOSTNAME
env[PATH] = /bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/opt/php53/bin:/opt/php53/sbin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

As always, YMMV.

Summary in my own words:

Never pay more than $5/foot for an HDMI cable at big box places like Best Buy, etc.

The best places to go are small shops around town - those are roughly $1/foot. There's a handful of places around me that offer plenty of options - length, color, etc. Obviously the length-to-cost ratio is different based on the length (I see some that are less than $1/foot, and some that are more than $1/foot, but all pretty close.)

When looking at an HDMI cable all you need to see is "High Speed" or 10.2Gbps on the packaging. Don't look for "Hz", "4K" or anything else. Look for either "10.2 Gbps" or "High Speed" and then find the cheapest cable for the length you need.

As it says, quality of manufacture can be an issue - but like any product, that is always a possibility. However, most cables should have absolutely no issue.

This goes for Ethernet cables too - you shouldn't have to pay more than $1/foot anymore. Even for cat6. I'm looking at a local place with 50 foot cat6 for $15. 75 foot cat5e for $11. Don't get ripped off!

Why? I find it to be the most universal and easy to support. I create things from scratch using it, and the most popular software out there uses it - Drupal, Joomla, WordPress - and frameworks including the Zend Framework.

A lot of people cook up crazy nginx configurations for this stuff, when all it really takes for at least a standard setup is a single line of configuration code. However, I just found out something that was sitting in front of me this entire time.

BAD: I used to advise people to use this:

try_files $uri $uri/ /index.php?page=$uri$is_args$args;

BETTER: When actually, even simpler, you can use this:

try_files $uri $uri/ /index.php?page=$request_uri;

So my whole blog post here is slightly wrong. If you use $request_uri, you don't have to remember the $args. It's already there. :)

Note: different packages have different parameters. CMS Made Simple uses "page", Drupal uses "q", WordPress uses "q" - some software, like WordPress, doesn't technically need anything, they can fall back by reading PATH_INFO or REQUEST_URI, and not a parameter.

In which case, all you need is:

try_files $uri $uri/ /index.php;

Don't care about directories? Drop the "$uri/" too. Save some stat calls - if you don't need em :)

We had an upcoming product release, and another team was handling all the communication, content, etc. I was simply just the "managed server" administrator type person. Helped setup nginx, PHP-FPM, configure packages on the server, stuff like that.

The other team managed the web applications and such. I had no clue what we would be in for that day.

There is a phrase, "too many eggs in one basket" - well, this was our first system and we were using it for many things, it hadn't gotten to the point where we needed to split it up or worry about scaling. Until this day, of course.

To me, the planning could have been a bit better, with the ISO image we were providing from the origin being staged on other mirrors, using torrents to offload traffic to P2P, etc. However, that wasn't done proactively, only reactively.

The story

Launch day occurs. I get an email about someone unable to SSH to a specific IP. I check it out - the IP is unreachable, but the server is up. I submit a ticket to the provider, and they let me know why - apparently the server had triggered their DDoS mitigation system once it hit something like ~500mbit; it was automatically flagged as probably being under attack, and the IP was blackholed.

Once we informed them this was all legit, they instantly worked to put us onto a switch that was not DDoS protected, and we resumed taking traffic. This was all done within 15 or 20 minutes, if I recall. I've never seen anything so smoothly handled - no IP changes, completely transparent to us.

I believe the pocket of no traffic (seen in the graph below) was when we were moved off their normal network monitoring and into a Cisco Guard setup. We were definately caught off guard; like I said, I knew we'd get some traffic, but not filling the entire gigabit port. Not a lot of providers would be so flexible about this and handle it with such grace. There are some reports of it being slow, and that is literally because the server itself has too much going on. PHP is trying to handle all the Drupal traffic, and during the night, the disk I/O was at 100% for a long period of time. Oh yeah - since this was the origin, the servers mirroring us had to start pulling stuff from us too :)

Luckily we're billed by the gigabyte there, not by 95th like most places, or this would be one heck of a hosting bill. We wound up able to reroute the bandwidth fast enough to not even be charged ANY overage for it!

All in all, without nginx in the mix, I doubt this server would have been able to take the pounding. There was no reverse proxy cache, no Varnish, no Squid, nothing of that nature. I am not sure Drupal was even setup to use any memory caching, and I don't believe memcached was available. There were a LOT of options to reduce load - the main one was just cutting down the bandwidth usage to open up space in the pipe, which was eventually done by removing the ISO file off the origin server, and pushing it to a couple mirror sites. Things calmed down then.

However, it attests to how amazinly efficient nginx is - the entire experience wound up taking only 60 something megabytes of RAM for nginx.

Want the details? See below.

The hardware

• Intel Xeon 3220 (single CPU, quad-core)

• 4GB RAM

• single 7200RPM 250GB SATA disk, no RAID, no LVM, nothing

• Gigabit networking to the rack, with a 10GbE uplink

The software - to the best of my memory (remember, all on a single box!)

• nginx (probably 0.7.x at that point)

  • proxying PHP requests to PHP-FPM with PHP 5.2.x (using the patch)

    • Drupal 6.x - as far as I know, no advanced caching, no memcached, *possibly* APC

  • proxying certain hosts for CGI requests to Apache 2.x (not sure if it was 2.2.x or 2.0.x)

    • mailman - this was a public mail list server

    • Bugzilla

  • This server was also a yum repo for the project, serving through nginx

• PHP-FPM - for Drupal, possibly a couple other apps

• Postfix - the best MTA out there :)

  • I believe amavisd/clamav/etc. was involved for mail scanning

  • Integration with mailman, of course

• MySQL - 5.0.x, using MyISAM tables by default, I don't believe things were converted to InnoDB

• rsync - mirrors were pulling using the rsync protocol

The provider

• SoftLayer - they just rock. Not a paid placement. :)

The stats

nginx memory usage
During some of that time... only needed 60 megs of physical RAM. 240 megs including virtual. At 2200+ concurrent connections... eat that, Apache.

root     13023  0.0  0.0  39684  2144 ?        Ss   03:56   0:00 nginx: master process /usr/sbin/nginx
www-data 13024  2.0  0.3  50148 14464 ?        D    03:56   9:30 nginx: worker process
www-data 13025  1.1  0.3  51052 15256 ?        D    03:56   5:38 nginx: worker process
www-data 13026  1.3  0.3  50760 15076 ?        D    03:56   6:13 nginx: worker process
www-data 13027  1.3  0.3  50584 14900 ?        D    03:56   6:22 nginx: worker process

nginx status (taken at some random point)

Active connections: 2258
server accepts handled requests
711389 711389 1483197
Reading: 2 Writing: 2040 Waiting: 216

Bandwidth (taken from the provider's switch)

Exceeded Bits Out: 1001.9 M (Threshold: 500 M)
Auto Manage Method: FCR_BLOCK
Auto Manage Result: SUCCESSFUL

Exceeded Bits Out: 868.1 M (Threshold: 500 M)
Auto Manage Method: FCR_BLOCK
Auto Manage Result: SUCCESSFUL

To give you an idea of the magnitude of growth, this is the amount of gigabytes the server pushes on a normal day:

• 2009-05-18 155.01 GB

• 2009-05-17 127.48 GB

• 2009-05-16 104.21 GB

• 2009-05-15 152.42 GB

• 2009-05-14 160.12 GB

• 2009-05-13 148.6 GB

On launch day and the spillover into the next day:

• 2009-05-19 2036.37 GB

• 2009-05-20 2481.87 GB

The pretty pictures

Click for larger versions!

Hourly traffic graph

(Note: I said 600M, apparently their threshhold from their router says 500M)

Weekly traffic graph

The takeaway

Normally I would never think a server could get slammed with so much while it is having to service so much. Perhaps if it was JUST a PHP/MySQL server, or JUST a static file server, but no - we had two webservers, a mailing list manager, Drupal (which is not the most optimized PHP software), etc. The server remained responsive enough to service requests, on purely commodity hardware.

• AT&T's got a Femtocell called a "MicroCell"

• PayPal is an example of a "micropayment" service

• Microformats are becoming increasingly popular to add more metadata into websites, mainly for richer machine processing

• MicroATX is one of many small form factors. However, not as small as Nano-ITX, Mini-ITX

• Twitter is the world's most popular "Microblogging" service

• Need to clean something fragile? Microfiber cloths are typically used to clean luxury cars, computer parts, screens, etc.

• Those small tweaks to squeeze out a little bit more performance? Micro-optimizations can be useful or can be a pain. It's up to you to decide what is worth it or not.

In this world of larger cars, larger boats, larger cruise ships, larger meals, just remember this - a lot of things are getting smaller. Typically technology... but I threw in another couple terms I seem to say often.

Update:

I forgot, microexpressions, now being made a household name thanks to the TV show "Lie to Me" (highly recommended show, btw.)

Say you have a foreign host and you don't want to have to figure out if you're on http:// or https:// and call their assets appropriately so you don't get a mixed-mode warning. You can actually use a syntax that is defined in RFC 3986, specifically section 4.2:

A relative reference that begins with two slash characters is termed a network-path reference; such references are rarely used. A relative reference that begins with a single slash character is termed an absolute-path reference. A relative reference that does not begin with a slash character is termed a relative-path reference.

Which means you can do this:

<img src="//foo.com/bar.jpg" />

and your browser will request http://foo.com/bar.jpg or https://foo.com/bar.jpg, depending on what scheme your browser is currently on.

I was hesitant at first to consider it "okay" but as it is published in the RFC and Chromium's fixed bugs relating to it, it does appear to be a properly supported method that could save you a few keystrokes. Let me know if it doesn't work for you! Be sure to give browser/OS information and conditions to reproduce.

Oh yeah, and the other host needs to be on https as well, of course. I shouldn't really have to say that, though :)

openssl genrsa 2048 > yourhost.com.key
openssl req -new -key yourhost.com.key > yourhost.com.csr

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:.
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Something Here
Organizational Unit Name (eg, section) []:.
Common Name (eg, YOUR name) []:yourhost.com
Email Address []:.

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

cat yourhost.com.crt gd_bundle.crt > yourhost.com.pem

ssl_certificate /etc/nginx/certs/yourhost.com.pem;
ssl_certificate_key /etc/nginx/certs/yourhost.com.key;

Additionally I have these SSL tweaks which seems to maintain a better SSL experience, passes McAfee Secure's SSL checks, etc.:

ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:-ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;
ssl_session_cache shared:SSL:10m;

/etc/init/my-kvm.conf:

description     "my-kvm"

start on (net-device-up
   and local-filesystems)
stop on runlevel [016]

respawn
exec /usr/bin/kvm -hda /root/virtual-machines/my-kvm.bin -no-acpi -m 128 -net nic,macaddr=DE:AD:BE:EF:18:12 -net tap -vnc :0

Enjoy.

17 Jun 2010, PHP 5.3.3 RC1
...
- Added FastCGI Process Manager (FPM) SAPI. (Tony)
...

and on an unrelated note:

- Added support for JSON_NUMERIC_CHECK option in json_encode() that converts numeric strings to integers. (Ilia)

Shouldn't this be called JSON_NUMERIC_CONVERT? or JSON_FORCE_INTEGER? It's not just a "check" - guess it's too late now? :)

WordPress 3.0...
WordPress 3.0 came out today. Tonight I'll probably upgrade this site and see how well it works. I'm going to check it in to Subversion first so I can roll back if needed.

Some key changes I wanted to talk about...

• One thing that was highlighted is the option to use custom header images - which can easily be done right now. I did it well over a year ago in a theme. With post meta you can always load metadata about a post and use it in the theme, so this update seems a bit specific to me, since themes were already customizable. Why build a feature that is so specific? Same with background images/colors...

• Custom menus/menu editor - this could get cool, the menu editor is the more exciting piece as it will allow a visual way to manage the taxonomy. Not sure how it will mix in with tags and categories though, guess that's "I'll see it when I upgrade."

• MU merge - finally, I can run multiple installs off the same WP install, hopefully, without wp-config.php hacks. How exactly it works I will have to find out.

• Custom post types - now all of a sudden you can make any sort of object with custom attributes, which opens the door to things such as the item below.

• WP e-Commerce says they're going to change from using all their extra tables to using core WP schema. That's awesome.

A couple bones to pick...

• It's not a rewrite. It's still a blogging tool that is being extended further to be a full-featured that can handle "anything" - however the tables are still named "posts" even though now you can create an arbitrary type of item. I'd like to see it renamed and normalized.

• All the plugins and themes and such are procedural code, but some inner workings such as the DB layer are OO. That seems amateur to me, and unnecessary.

I'd love to see WP get rewritten. It has a LOT of overhead and includes built in that need calling and a lot of other cruft that I stumble across. Go back to the drawing board with building a list of every feature it has, and look at it from a longer term perspective. It's great to see something keep growing, but when it comes down to it, it is still a fork of b2, which was made for blogging, not for anything and everything.

It's got the right idea with extensibility and such, but to me the core has a lot of code - and lots of code means more complicated execution paths, more "I'll just add this in instead of refactor this old code," more cruft. I'm quite sure I could get as much extensibility out of a fresh rewrite with less than half the code under the hood. Things like text styling for example should be moved to a plugin (I disable all the wptexturize filters for example... throw those in an include and make it enabled by default instead!)

Of course, WordPress does have millions of users so it has a proven track record. I can't complain that much, I do use it myself. For blogging, it's the best tool out there. For other things, it typically leverages plugins which may or may not have decent UIs or APIs to interact with. That's where it shows signs of weakness. It also isn't as strict as Drupal when it comes to code conventions either, which would greatly increase the usability of a lot of plugins.

OS 4 will give us tethering ability with AT&T. Of course, that comes with an additional cost now. If you want to "officially" tether with AT&T, you'll have to switch to one of their two data plans, and THEN buy tethering on top of that. Instead of your $30 for unlimited, you'll be paying $45 for only two gigs of bandwidth, vs. $30 currently for unlimited. Or $35 for a measily 250 megs.

They claim that 98% of its users fall within those tiers. Sounds like they're giving 98% of us a way to save a few bucks, right? Wrong. What people are overlooking is the fact that those statistics don't include the fact that you will be TETHERING. As in, your computer will be transferring data, and you're already accustomed to a much heavier data consumption through that medium. Think of how those little widgets on your laptop grab information off the net every so often, or a single Youtube video could easily be 20+ megs. It does add up, and ultimately anyone who wants to tether with AT&T should use it sparsely - otherwise, they'll be paying $10/gig for your overages.

I'll give AT&T some credit - their marketing geniuses and social engineers are gaming this system well - using the upcoming iPhone and OS for service type changes, some of which require contracts, some don't - helping them limit the impact on their network in creative ways while making you think you're getting a better deal. I especially like the fact once you drop out of an unlimited contract you can't go back, and they're trying to sway you out of it by not changing your expiration date if you drop down, and "allowing" you to switch back and forth between the two plans. Oh, and new iPad users? If you don't have a plan by the time OS 4 launches, you'll be subject to these new limits.

Let's face it - AT&T's network can't handle the iPhone. It's a blame game that ultimately both could be doing better at. I'm sure the iPhone could change it's tower-hogging style behavior, and I'm sure AT&T could be deploying all these upgrades they keep telling us about faster.

For me, if I do decide to go with the next iPhone, I won't be buying it with a contract. I'm going to stay a free agent, and not get locked in to a new agreement, and not buy their joke for insurance (monthly fee and still the cost of buying a refurbished unit for the deductable?!) - and I am getting closer every day to switching over to maybe the Evo 4G on Sprint - a hybrid solution with a limited 4g network around the area. Sprint and Verizon seem to have a much more reliable network, and since this device is still primarily a phone, voice calls do count for something. Sprint's at least would afford me faster WiMAX when I could get it, and openly promote the device's ability to share the connection with up to 8? devices.

I haven't investigated the fees, but when I can't go a single day without a dropped call, and I try to find landlines to fall back on where I can instead of my phone, something has got to change.