"Around here, however, we don’t look backwards for very long. We keep moving forward, opening up new doors and doing new things, because we’re curious… and curiosity keeps leading us down new paths." — Walt Disney

There are many posts on this blog about Wii Transfer, the little Mac app that launched commercially almost by accident, and convinced me that it would be worthwhile to invest time in this side business called Riverfold Software. Early posts like the launch post in 2006 or this one about the first 75 days, and this one covering the price bump in version 2.5. But the app has been fading over the last couple of years, no longer as relevant today as it once was. It's time to let it go.

I'm retiring Wii Transfer to focus on my other apps. It's not that it doesn't sell; it still does. It's just that it's not an app I actually use anymore. By officially shelving the whole project, I hope to remove a psychological burden of sorts — to no longer worry that I'm ignoring an active product.

It also doesn't fit into a new theme I have for Riverfold: apps that are all about keeping and remembering what matters. For Clipstart, that's family videos. For Tweet Library and Tweet Marker Plus, that's old tweets. Wii Transfer is about… listening to music on your Wii? It doesn't fit, and in the world of the Apple TV and Roku, modern streaming technology has passed the app by.

If anyone is disappointed that Wii Transfer will no longer receive updates, of course I offer refunds. I won't be selling or open sourcing the app, preferring instead to continue to support existing customers myself for as long as they want to use the app. And I'll keep the automatic bookmark service running that makes setup easy, as well as the Mii rendering service, so nothing breaks.

I put a lot of work into Wii Transfer over its 5-year lifespan. It's not easy saying goodbye, especially to some of the unique things that only Wii Transfer could do, such as exporting Miis as images. Maybe I can bring that back one day. For now, I'm following the path started by my apps Tweet Library and Clipstart, for which there are many new things still to do.

I've been using Twitter's Bootstrap in an internal project at VitalSource for a few months, and over the weekend I finally switched to using the CSS framework in Tweet Marker too. The layout now works in more browsers and provides a much better foundation for design changes. It also allowed me to integrate this excellent date picker.

Here's a short screencast video showing the date picker in a new browsing feature in Tweet Marker Plus. I'm very happy with how this turned out — both the look and functionality. On the server the date ranges are implemented with a Sphinx query, so they can be combined with search terms to help find old tweets.

A few things have changed since I last talked about hosting. Tweet Marker passed 200,000 total users to the API. There are more apps and more platforms. And of course Plus launched with new requirements for database and search indexing of tweets.

Here's a graph showing monthly hosting costs for the last year, stopping just short of $600/month for April:

The first dip was when I moved away from Heroku's dedicated PostgreSQL database, to Redis on EC2. The more recent increase is when I updated capacity for Tweet Marker Plus.

Tweet Marker currently runs on both Heroku and Amazon EC2. On Heroku, there are 6 dynos: 5 web dynos with 3 Unicorn processes each, and 1 background worker. I also run hourly scheduled tasks that add a small number of extra dyno-hours, and sometimes I'll fire up additional temporary workers.

For Amazon, I run with 3 medium server instances: 1 for Redis master, 1 for Redis slave, and 1 for search with MySQL and Sphinx. The search server is partitioned across multiple EBS volumes, each one mounted as a separate MySQL database and Sphinx index. It is possible for me to move a database or search index shard to a different EC2 instance if I need to, as well as move customers between shards.

The volumes look like this in Amazon's admin UI:

I picked 20 GB shards because it seemed like about the point where the database would be too big to be fast, given the modest hardware. It's enough to hold several months of tweets for all the users in a shard. I estimate how many users should be in each shard, and when it reaches that number I roll new accounts to the next shard, and so on.

Backup dumps for the Redis database and MySQL get sent to S3 every hour and every day, where I keep the last 24 hourly backups and the last 31 daily backups. I also do occasional EBS snapshots.

I don't currently need a MySQL slave for backups. If I lose a drive, when I restore the last hourly backup, Tweet Marker Plus will automatically add any missing tweets lost during the downtime to bring things back to a current state.

Overall I'm happy with this setup. It's as simple as I could design it. Hosting is not cheap, but I think I can run for the rest of the year with very few changes and mostly fixed costs. I also plan to switch to reserved instance pricing at Amazon, which should be a significant discount.

If you've made it this far, you probably care enough about servers and Twitter that you should consider signing up for Tweet Marker Plus yourself. Check out the details here.

I released a small bug fix update to Clipstart today, version 1.4.2, to fix an issue with YouTube uploads when using your Gmail address sign-in instead of the YouTube account username. This version should also show up in the Mac App Store after it goes through Apple's approval process. You can see the full release notes for recent bug fixes here.

As I said earlier this year, there will only be a couple more releases of Clipstart in the Mac App Store. My current plan is to switch completely over to direct-only sales with version 1.5. The new versions run without prompting for registration if you've already purchased and run a copy from the Mac App Store.

You may not notice it right away, but Tweet Marker Plus is the start of a big migration for my Twitter projects. The backend infrastructure for tweetlibrary.com will be moving there, so that Tweet Marker can have access to published collections. And a major web feature to complement Tweet Library — originally written for tweetlibrary.com last year but never released — will be launching on Tweet Marker Plus instead.

Essentially, Tweet Marker will be the web app and web services. Tweet Library will be the native iOS client only.

This has a few pretty big advantages for me:

• One web app to manage instead of two.
• Other Twitter apps can access the collections publishing API.
• Tweet Marker is a better brand than Tweet Library.

More people know about Tweet Marker. It just makes sense to build any new web features on top of that existing name.

The first step in this transition is ready now: published collections can be accessed via tweetmarker.net. For example, see this collection of tweets from the game Millinaut by Shaun Inman, Neven Mrgan, and Alex Ogle.

VoodooPad 5 is out this weekend, with a new file format that plays nicely with Dropbox. Gus Mueller says:

"No more connecting to WebDAV servers and having to deal with authentication issues and strange HTTP errors. Now you can just put your VoodooPad 5 document in a Dropbox folder and VP will detect when pages have been updated."

This is another good example of where web APIs like Dropbox can be more useful than iCloud. Multiple people can collaborate on a VoodooPad document, and the direct download and Mac App Store copies of VoodooPad can sync together.

Also new in version 5 is native support for Markdown and an ePub export option. The workflow for help documentation that I posted 5 years ago carries over to VoodooPad 5 just fine, too.

There has been some good commentary on Sergey Brin's interview with the Guardian. It's probably best summarized in John Gruber's comment:

"The assumption here is that the only way to search is through Google, and that the 'open Internet' is only what Google can index and sell ads against."

This resonates with me because I think Google has put enough ads on the internet. It's impossible to take anything Google says at face value if they talk of "open" but their intentions say "ads". But here's the thing: Brin is actually right. There is great data hidden behind apps that should be indexable.

In the race to win the App Store, we forgot about the web. Think about Instagram. Millions of photos are being shared that are inaccessible via a search engine. These photos can't be found again and aren't discoverable. When you search the web, how does it make sense that public photos on Flickr show up, but public photos on Instagram do not?

We shouldn't have to choose between Apple's closed systems and Google's ad-driven business. I want to talk about improving the web without automatically being pro-Google. This tweet from John Marstall made me realize it:

"I'd prefer to separate 'indexable' from 'indexable by Google.' Yes, they're mostly the same for now. Might not always be.".

We need competition in web search. More than Bing. A new search engine is the number 1 item from Paul Graham's frighteningly ambitious ideas:

"The best ideas are just on the right side of impossible. I don't know if this one is possible, but there are signs it might be. Making a new search engine means competing with Google, and recently I've noticed some cracks in their fortress."

Of course it's crazy, but so was the 1st search breakthrough, lightning-fast AltaVista, and so was the 2nd major innovation, Google itself. It's time for a search engine that isn't all about ads. It's time for search that understands apps and embraces data from web services as much as it does from web sites. It's time for the 3rd act in search.

This MacStories article covers the progress that developers have made adopting iCloud over the last 6 months. Over the next 6 months, we should have an even better appreciation for what iCloud is and isn't good for.

For iOS backups and iTunes Match, iCloud is fantastic. For private, app-specific data that doesn't make any sense away from a single developer's native Mac and iOS apps, it's also excellent. There's no question that using Macs, iPhones, and iPads today is a significantly better experience thanks to iCloud.

But there are two fundamental limitations in iCloud that make it inappropriate for a bunch of syncing uses:

• No way to access it from other platforms or web apps.
• No way to share data between apps from different developers.

I don't expect Apple to change this. Again, for what iCloud was made for, these limitations are fine. It simply means that iCloud cannot replace web APIs that do solve these two problems.

Here's what I wrote not long after announcing the Tweet Marker API:

"The primary goal with Tweet Marker is to enable different Twitter apps to work together. iCloud is designed for storage and syncing only between apps from the same developer, so it's not appropriate as a replacement architecture for Tweet Marker."

Think back to the so-called Web 2.0, which gave us web services to access previously closed-off data. This eventually led to truly dominant syncing APIs like Dropbox, Simplenote, and Instapaper. At the same time, we all have more devices than ever before. Syncing exploded on iOS in text editors and note taking apps. Without a proper shared file system between apps, or even an event or scripting system, these iOS apps looked to web APIs as the only way to communicate.

That has turned out wonderfully. With web APIs as the only solution, we see more compatibility between apps and more web services popping up all the time. If you create a new web app, it's dead without an API. Every success of the modern web, from Flickr to Twitter, has an API that is available from any platform.

So then what about iCloud? If Web 2.0 made data more accessible, iCloud takes that same data and... keeps it closed. It's a step forward on user convenience and a step back on interoperability.

If you're a developer considering iCloud support, just make sure your data fits there. Ask yourself if your data is all about your app, or if it's bigger than your app. Developers who are willing to take a risk on building an open API instead of iCloud could see new opportunities: web-based views of their data, compatibility with other apps, and syncing on the Mac outside of the App Store.

A couple years ago, Shawn Blanc said about cloud syncing:

"And my next wish? A cloud-based service like Instapaper, but for to-do items. I want it to be available in apps like Tweetie, Reeder, and more, so when I click on 'Do Later' it sends the link or item of note into a running to-do list (that syncs with Things, of course)."

Imagine if Things or OmniFocus or another tasks app opened up a slice of their private syncing API to make the Instapaper of to-do inboxes. Now take other APIs for all of the useful apps we use. Not just to-do apps as Shawn mentions, but RSS, photos, blog drafts, sketches, and more.

What I wrote above about Tweet Marker is still very much true. Since then, Tweet Marker has become the standard for last-read syncing between Twitter apps, with support for 15 apps and many more in development. I want to see more syncing platforms like this. Let's think big and make apps work together.

When I created Tweet Marker Plus, I thought I was creating a new way to search Twitter. Limit the search to just people you follow and you can store more tweets, and more relevant ones. But as I've been adding new features to it, I'm realizing that Tweet Marker Plus is really a new kind of Twitter client — a client that has search and filters at its core.

Here's what the sidebar looks like in my Tweet Marker Plus account:

Seems simple enough. But quickly switching between saved filters is very powerful. Because Tweet Marker is routinely fetching new tweets in the background, even when you haven't opened your web browser in days or weeks, there are no gaps in the timeline. When I use a filter, it's showing me everything that any of the people I follow have said since I first started using Tweet Marker Plus.

I'm excited about this. I'll keep adding features and growing the storage, to make Tweet Marker Plus the best value $2/month could possibly get you.

I love reading about how big sites use Amazon EC2. If this post from Instagram is still accurate, they must be at something like $50k/month in hosting costs. Their user base has doubled in the 4 months since they posted that.

My setup for Tweet Marker is trivial by comparison, but to me — without Instagram's $7 million in funding — it's a very big deal. What I wrote back in October about moving to Redis is still mostly true, although I've added MySQL and Sphinx to the mix. I now run with 3 Amazon EC2 "medium" instances and have 5 web dynos (with 3 Unicorn processes each) at Heroku.

I put everything from donations back into the servers. For Tweet Marker to work, it had to be rock solid. It had to scale. It's only the first week for Tweet Marker Plus subscriptions, but already I have a good feeling that it'll all pay off.