8. RESTRICTIONS ON USE

You may use this Site only for purposes expressly permitted by this Site. You may not use this Site for any other purpose, including any commercial purpose, without YOBI’s express prior written consent. For example, you may not (and may not authorize any other party to) (i) co-brand this Site, or (ii) frame this Site, or (iii) hyper-link to this Site, without the express prior written permission of an authorized representative of YOBI. For purposes of these Terms of Use, “co-branding” means to display a name, logo, trademark, or other means of attribution or identification of any party in such a manner as is reasonably likely to give a user the impression that such other party has the right to display, publish, or distribute this Site or content accessible within this Site. You agree to cooperate with YOBI in causing any unauthorized co-branding, framing or hyper-linking immediately to cease.

Far be it from me to violate their Terms, which is why the name of their site, above, is not hyperlinked.

I thought this particular idiocy had been eliminated years ago.  If you don’t want people to link to you, why the heck are you on the Web at all?  I think somebody needs to rein in the lawyers again.

The “y”, “e”, “4″, and “u” are simple.  But is that “w” or “W”?  “z” or “Z”?

I wasn’t going to send a comment.  And, seeing as how I can’t reliably determine what to type here, I wouldn’t even attempt it.  Knowing my luck, I’d get into an infinite loop of failures and ambiguous prompts.

Pubsubhubbub was originally designed to support the Atom Syndication Format.  But there’s no widely accepted standard for publishing media information with Atom.  It looks like most sites that publish media use RSS 2.0 and the Media RSS extensions.

I touched on the development of syndication formats in a 2004 blog entry.  Atom “won” by becoming a standard, but RSS (mostly RSS 2.0) is so prevalent that it’s unlikely to be pushed out any time soon.

The developers of Pubsubhubbub included support for RSS in July of last year, and the latest specification (0.3) includes support.  This is both good and bad:  good because now all those sites that use RSS 2.0 can participate in Pubsubhubbub, and bad because those of us building readers still have to deal with two competing formats.

Also on the bad side, and especially pertinent to me, is the lack of any kind of media extensions for Atom.  There’s been some chatter in the past about using the Media RSS extensions inside of Atom, but it appears that nothing ever came of that.  Atom has an “enclosure” element that adds some media capability, but it’s pretty minimal.  The Atom Publishing Protocol specification allows for media, but not to any great extent.   There’s also a proposed standard for Atom Media Extensions, although I find no mention of it on the IETF’s site.

So I’m left with this choice:  publish RSS 2.0 with Media RSS extensions (both non-standard), or publish a severely crippled Atom standard standard document.  Since it’s likely that RSS 2.0 will continue to thrive and it looks like there are no standard media extensions to Atom, it looks like I’m stuck with RSS.  Unfortunate, but that’s the way it goes.

We love standards.  That’s why we have so many of them.

We actually had something in a little more than a year, but it wasn’t very interesting.  It worked, but nobody would have been very interested in it if we had released it to the world.

By then we had grown to four people.  We stuck with it, improved our crawler and indexing technology, and released two or three other incarnations to very limited audiences.  Those attempts, too, were less than successful, but they gave us a lot of valuable information about what people like (and don’t like), and how users want to view online video.

It’s been a long road, and sometimes a bit discouraging, but we’re finally able to present an early version of our new product, Podly.TV.

Take it for a test drive.  Browse the channels.  If you don’t see anything you like, do a search and create your own channels.  It’s totally free.  Anybody can sign up and create a personal channel list.  We’re adding new videos constantly, and new video sources on a regular basis.

Google’s technology is impressive, no doubt. They’ve come a long way in the 12 years or so since two college kids named Sergi Brin and Larry Page came up with a way to greatly improve the quality of Web search results. They met quite a bit of resistance when they went looking for funding to build a company. Everybody thought that Yahoo owned search, and nobody thought you could make money with search. “You’re going to spend millions of dollars to build a phone book of the Web? How will you make money?”

Around the same time, there was a small group at Microsoft who wanted to build search. Microsoft’s corporate leaders shut that down pretty quickly, for much the same reason: “there’s no money in search.” In addition, and perhaps more importantly, Microsoft hadn’t really embraced the Internet. Sure, Internet Explorer was in ascendance, mostly due to Netscape’s incompetence, and other parts of the company were making noise about using the Web, but at heart Microsoft remained a shrink-wrap software company. Their business was selling Windows and Office. They embraced the Internet to the extent required to sell those products.

Microsoft eventually embraced search, first grudgingly–”It’s something we have to provide”–and finally, after realizing that there was money to be made, by committing serious resources. But by then it was too late.

It was too late because Google had figured out how to make money with search: first by displaying advertisements on search pages, then with Adwords, Adsense, and other cooperative advertising programs. Google was transformed from a search engine with some incredibly impressive technology into an advertising company that understands how to make billions of dollars a few pennies at a time.

And Google is an advertising company. Make no mistake. Google is in the business of placing ads on your screen, and doing so in a manner that makes you more likely to click on the ad. That means making them as relevant as possible and walking that fine line between visiblity and unobtrusiveness. It also means getting their ads everywhere, and everything that Google does furthers that goal, directly or indirectly.

Google’s technology has two jobs: deliver ads, and to increase their audience. I know very little about how they deliver ads–that’s their proprietary process and, one might argue, the heart of their business. But they’re transparent about how they increase their audience. They provide arguably the best results of any general search engine available. With YouTube, they dominate Web video. They have a whole bunch of other free services and software–translation tools, Google Chrome Web browser, Google Maps, Google Earth, Google Books, Patent Search, Blogger, Mail, SketchUp, Images, and many more–that make it easier to use the Internet or provide online replacements for traditionally client-bound tools. By making it easier to use the Internet, they get more people on the Internet.

Google also produces and makes available an incredible amount of program source code that developers can use or include in their products for free. Just check out Google code sometime. It’s full of proven working code that Google paid their employees to develop, and is now giving away for free. It’s not that they’re altruistic. They know that by making it easier for developers to create quality Web sites, their audience is growing.

Two recent (well, one not so recent) developments show Google’s commitment. First, the Chrome Web browser. This is Google’s free browser, which is arguably the best on the market today. One might ask why Google would go to the expense and effort of creating a new browser and then make large parts of its source code available (see the Chromum project)? I can’t say for sure, but here’s what I think.

I think that Google wants to do things with the Web that other browsers (Internet Explorer, Firefox, Opera, Safari, etc.) don’t currently support. Although it’s often possible for Google to convince the people in control of those browsers to support new features, Google is left waiting for support. If they control the browser, then Google can start pushing new technologies on their own schedule.

Whatever the reason behind it, Google Chrome is building market share. It used to be that Microsoft’s Internet Explorer had 70 to 75% of the browser market, followed by FireFox in the 20 to 25% range, and everybody else was down in the noise. The most recent numbers I have put IE below 60% for the first time, Firefox still hanging in there around 20 to 25%, and the rest being shared by Opera, Safari, and Chrome. Except Chrome is taking market share, most of which is coming from Internet Explorer.

The more recent development is Google’s support of the WebM project, a high-quality, open, and free video format. I cannot overemphasize the importance of this development. WebM combines a container format with free video and audio codecs so that anybody can create and distribute video royalty-free without having to worry about patents or other intellectual property concerns. Google spent something like $100 million to obtain the rights to the VP8 video codec in order to make this possible. Then they turned around and made it freely available to anybody. Why? Because ubiquitous free video gives Google a huge increase in surface area–a larger audience–that they can exploit for the purpose of delivering ads.

From the outside, Google’s business plan really does look as simple as, “Make the Web easy to use so that we can deliver more ads to more people.”

In the process, Google is steamrolling over a number of entrenched companies who thought they had it made. Consider Adobe, whose Flash player is currently The Standard for online video. Back in 2007, Flash 8 had something like 95% (perhaps higher) penetration. That is, 95% of computers connected to the Internet had Flash installed. Why? Because of YouTube. When Adobe released Flash version 9, it achieved more than 90% penetration in just a few months, again in large part (perhaps primarily) because YouTube went to Flash 9 for their video. Adobe owned Web video.

But Adobe dropped the ball. For reasons I’ll never understand, Adobe still clings to the idea that Flash is for creating rich Web apps. The ability to do rich client things in a Web page is cool, and there was a time when Flash was the best way to do it. But browsers and computers are more capable now. I know from experience that it’s now much easier to build rich applications with JavaScript than it ever was with Flash. And all you need is a modern browser. There’s no need to download and install a Flash control to do it.

After Google’s WebM announcement last week, Adobe made a press release saying that they’ll support WebM “in a future version.” YouTube will continue to use Flash for low-quality videos. Starting soon, though, higher quality video will be delivered with WebM. You have to be blind not to see what’s coming: the eventual removal of Flash support on YouTube. But it’s already over for Adobe Flash. They will only see decreasing market share. And Adobe has nobody to blame but themselves. They ran into much the same thing clinging to their old .FLV format when the rest of the world was moving to .MP4. The reason? They make money by selling very expensive software packages that create video files. Much like their PDF tools, they give away the reader and charge a lot of money for software that creates the files that their free players read.

With WebM, all that goes away. There are already FFmpeg patches for WebM, and likely will be some very good free tools.

Microsoft, too, is getting steamrolled by Google. After Google’s WebM announcement, Microsoft said that they’re very excited about the new technology and that Internet Explorer 9 will fully support it as long as the user has installed the proper codec. If you’re not familiar with the world of codecs, don’t feel bad. Understanding codecs is not something a user should have to do. Finding and installing the proper codec can be incredibly frustrating and fraught with danger. If you go looking for a codec for Media Player, for example, you’ll find yourself confused and in very real danger of inadvertently downloading and installing some malware.

For Microsoft to say, “as long as the user has installed the proper codec” is like GM saying that the new car they sell you will be fully functional as long as you find and install a compatible engine.

And don’t expect Microsoft’s Media Player to support WebM any time soon. According to Microsoft’s own Information about the Multimedia file types that Windows Media Player supports, they don’t even support MP4. Granted, that article was written two years ago, but it covers Media Player 11, which is the most current version. That article says, “You can play back .mp4 media files in Windows Media Player when you install DirectShow-compatible MPEG-4 decoder packs. DirectShow-compatible MPEG-4 decoder packs include the Ligos LSX-MPEG Player and the EnvivioTV.” In other words, you have to install a codec made by a third party in order to play a video format that the rest of the world embraced five years ago.

The announcement of WebM is also pushing innovation in another area: the server. The day after the WebM announcement, somebody was streaming WebM from the Cherokee Web server. One day! This has some very interesting ramifications. An open media format combined with an open Web server (like Apache) means that a free media server is not far behind. There goes Adobe’s Flash Media Server business. And quite possibly Microsoft’s Home Media Server, especially if somebody releases an easy Linux configuration that includes this hypothetical (but soon to be realized) media server, backup and data recovery, and document management.

It’s interesting to note that Google hasn’t had to “target” any of these companies in order to take them out. In fact, Google probably isn’t even interested in “taking them out.” Google is just doing what it needs to do in order to grow the business. If it means investing hundreds of millions of dollars so that more people will come online to watch video, then so be it. If Google makes a few pennies every time somebody watches a video online, that hundred million bucks will be returned in short order.

The really funny part here is that both Microsoft and Adobe had to see it coming. It’s not like Google made a surprise announcement last week: there was a big splash when they acquired the VP8 technology a few months back, and Google has been telegraphing this move since at least 2007, when they paid $1.5 billion for YouTube. That kind of investment says, “We want to own Web video because we think we can make money at it.” No, Microsoft and Adobe saw this coming and knew that they were powerless to stop it. But rather than embrace VP8 and try to find a way to work with it, they clung to their own product plans hoping that some imaginary Maginot Line would block Google’s advance. Adobe, Microsoft, and other companies whose businesses are built on artificial scarcity (selling bits) are living in the past and will continue to see their market share stolen by companies like Google that can provide better products for free.

You’re going to see this same thing play out all over again in the world of television. Google recently announced a deal with Intel and Sony that will put Google TV on Sony television sets. Today, something like 25% of all new televisions sold are Internet ready. Google is ready to go there, and not just because it increases the surface area for their Web advertisements, but also because it gives Google a platform from which to launch an assault on the television advertising market ($70 billion annually in the U.S. alone).

Google’s competition for that market is a handful of old media companies and Madison Avenue advertising firms, both of which have grown fat and complacent. Sure, they’ve been hit by Internet advertising over the years, but it’s been more of a slow leak in a dike rather than a tsunami that overwhelms the entire system. Those companies probably aren’t smart enough to see it coming yet, but when they do see Google riding the wave, they’ll probably all hunker down behind the dike and hope for the best. And then complain bitterly (read: try to win through litigation) when they discover that they lost the war while they were sitting there with their thumbs up their butts trying to decide if they should do anything.

Remember, you heard it here first.

I’m not trying to paint Google in a bad light at all. On the contrary, I have nothing but admiration for them. They’re going about their business. If the entrenched companies can’t keep up, it’s not Google’s fault. While the old media companies are refining the horse-drawn carriage, Google is hard at work on the V8 engine. In the process, Google is making all manner of things available to Internet users and developers, and actually encouraging us to build products that leverage the free services that the company offers. Given the choice between begging for access from the old media companies or accepting the bounty freely offered by Google, I’ll throw in with Google.

The document is freely available online as a PDF, although it can be difficult to find.  I just went searching for it again and couldn’t find the full version.  If I remember where I found it, I’ll post a link here.

Adobe Flash Player Update 3 (9, 0, 115, 0) and higher can play some MP4 files.  The subset of MP4s that Flash can play is described in Video File Format Specification Version 9.  That document gives you an idea of the MPEG-12 file format, although you probably want the full spec. if you’re implementing a reader.

The file format is quite flexible–perhaps overly so–but reasonably easy to parse once you grok the basic structure.  I coded up a quick MP4 reader in a day, and within two days had my web crawler extracting metadata from files I located online.  But then I ran into a problem:  my movie player would sometimes hang when trying to play a file.  The really weird part was that the movie would play fine if I downloaded it first.  It was only when trying to play from online that I experienced the problem.

It didn’t take long to find the problem.  Or at least part of the problem.

Data in the Part 12 file format is organized in “boxes.”  Those boxes contain all manner of information:  a file header, overall movie information, information about the individual tracks, synchronization data for different tracks, etc.  Part 12 describes the overall structure of the file and the contents of the “moov” box that contains basic movie metadata (number and types of tracks, duration, codecs required to play the tracks, etc.).

Another box, called “mdat,” contains the actual movie data:  the video and audio information that will be played.

In order to start playing a movie, a player must have the metadata.  The player can’t play the first frame until it knows how to decode that frame.  The movie data, on the other hand, can be delivered relatively slowly:  at whatever the playback speed is.  In other words, playing a movie consists of these steps:

Read the metadata.
Determine if the movie is playable with this player.
repeat
    Read movie data (audio, visual, etc.) frame
    Render frame
until end of movie

So it makes sense to organize data in the movie file to facilitate that.  Right?  In fact, the Part 12 document makes two very pertinent recommendations:

2)  It is strongly recommended that all header boxes be placed first in their container: these boxes are the Movie Header, Track Header, Media Header, and the specific media headers inside the Media Information Box (e.g. the Video Media Header).

8)  It is recommended that the progressive download information box be placed as early as possible in files, for maximum utility.

The emphasized “recommended” is in the original document.

There are good reasons for these recommendations, as I discovered in the first problem file I looked at.  In that particular file, the “mdat” box, which contains the frame data, is placed at the front of the file:  immediately after the file header.  “mdat” is 89 megabytes long.  It’s followed by the the “moov” box that’s a little less than two megabytes.  A movie player has to download 89 megabytes of stuff before it can get to the metadata that tells the player how to play the movie.  89 megabytes might not sound like much, but at 10 megabits per second (which would be a very fast residential connection here in the U.S.), it’s a minute and a half.  Nobody’s going to wait a minute and a half for their video to download.

I suspect that whoever made these movies has no idea that they’re effectively unplayable over the Internet, and might not even care.  I care, because I’m not going to download the entire movie just to see if I’m really interested in watching it.

What surprises me is that video player software doesn’t recognize this and skip over the movie data to get to the metadata.  The HTTP 1.1 specification makes it very easy to get a partial file.  The movie player should see that the “mdat” box comes before “moov”, and make another request to get “moov”.  It could then go back to “mdat” after digesting the metadata.

I wonder how hard it would be to create a tool that fixes those backwards movies . . .

Dragonwood appears to be very commonly used for the trunks and larger branches of artificial (silk) trees.  It’s also commonly used to make perches for pet birds, and I gather somewhat less commonly used to make cat trees and cheap furniture.  That’s all interesting, but I couldn’t find a picture of a dragonwood tree or anything that gave me the botanical name of the silly thing.  The best I could find is that it grows in Florida.

Somebody else on the forum posted an answer this afternoon, identifying the wood as Lyonia Ferruginea (rusty staggerbrush), a shrub or small tree that grows in Florida, Georgia, and South Carolina.  In case you’re interested, that person also indicated that it’s good carving wood.

I’m really surprised that this one stumped me.  The common name dragonwood (less often, “dragon wood”) is used in a lot of places, but I was unable to find a any reference that showed its botanical name.  I figured I could find it just like I can type “bottle brush tree” and get the botanical name.  No such luck.

One resource said that “dragonwood” was a corruption of the original “draggin’ wood”, which describes how they get the wood out of the thicket after it’s cut.

Hopefully anybody else looking for a description of dragonwood will find this post and not have to wade through a few dozen pages of links to fake plants and parrot cage goodies.

Other sites are much worse in that they provide no metadata in the HTML.  For example, one site’s video page is very code-heavy.  Requiring that the page be reloaded every time you request a new video would require a lot of network traffic.  Their design instead uses JavaScript to request a particular video’s metadata from a server.  Loading a new video involves downloading just a few kilobytes of data.

I spent some time this afternoon searching through the a video page HTML and the associated JavaScript, looking for the magic incantation that would get me the data I’m looking for.  The amount of code involved is staggering, and I quickly went crosseyed trying to decipher it before I hit on the idea of hooking up a sniffer to see if I could identify the HTTP request that gets the data.

It took me all of five minutes to download and install Free Http Sniffer, request a video from the site in question, and locate the magic line in the 230 or so requests that the page makes when it loads.  Problem solved.  Now all I have to do is write code that’ll transform a video page url into a request for the metadata, and I’m set.

I have no idea why I didn’t think of the sniffer earlier.  I’d used one before for a similar purpose.  I suspect I’ll be making heavy use of it in the near future as I expand the number of sites that we crawl for media.

So I canceled the notification and clicked on something else.  It reminded me again.  Okay, so I added those applications back to my feed.  Won’t let a little bug stop me from posting a comment on a friend’s wall.

Except now I keep getting this:

I can still post comments (the dialog is not modal), but it’s just … weird.

Later:  Closing the Facebook tab in my browser and re-loading fixed the problem.

Until 10 days ago, that report was delivered daily, without fail.  But since we changed our colocation setup and got new IP addresses, the mail has been sporadic:  bouncing eight of the last ten times I’ve tried to send it.  The error message I get back from Yahoo’s server says that the message is rejected “for policy reasons.”  Digging deeper, I find that Yahoo’s filter seems to think that there are “links to potentially objectionable material or malicious software.”

When contacting Yahoo, I gave them considerable detail, including the text of the message, a full description of the problem, and the reason why I thought that their filter was being over zealous.  Their responses have been canned boilerplate paragraphs, first asking for information that is not relevant or that I’ve already supplied, then explaining the policy:  the same policy that’s on their Web site and that I told them I already understood.  I have yet to receive a response from Yahoo to indicate that they’ve actually read and understood any of the information that I’ve sent to them.  I’m convinced that if I sent a message requesting a ham and swiss on rye, they’d reply by asking me to forward the full headers from the email in question.

I would strongly discourage anybody from using Yahoo for their business email.  Their response to this simple request has convinced me that Yahoo’s incompetence is not limited to search (which they’ve finally agreed to farm out to Microsoft), but permeates the entire organization.  If you want reliable email and intelligent, helpful support, find somebody other than Yahoo to host it.