Google’s technology is impressive, no doubt. They’ve come a long way in the 12 years or so since two college kids named Sergi Brin and Larry Page came up with a way to greatly improve the quality of Web search results. They met quite a bit of resistance when they went looking for funding to build a company. Everybody thought that Yahoo owned search, and nobody thought you could make money with search. “You’re going to spend millions of dollars to build a phone book of the Web? How will you make money?”

Around the same time, there was a small group at Microsoft who wanted to build search. Microsoft’s corporate leaders shut that down pretty quickly, for much the same reason: “there’s no money in search.” In addition, and perhaps more importantly, Microsoft hadn’t really embraced the Internet. Sure, Internet Explorer was in ascendance, mostly due to Netscape’s incompetence, and other parts of the company were making noise about using the Web, but at heart Microsoft remained a shrink-wrap software company. Their business was selling Windows and Office. They embraced the Internet to the extent required to sell those products.

Microsoft eventually embraced search, first grudgingly–”It’s something we have to provide”–and finally, after realizing that there was money to be made, by committing serious resources. But by then it was too late.

It was too late because Google had figured out how to make money with search: first by displaying advertisements on search pages, then with Adwords, Adsense, and other cooperative advertising programs. Google was transformed from a search engine with some incredibly impressive technology into an advertising company that understands how to make billions of dollars a few pennies at a time.

And Google is an advertising company. Make no mistake. Google is in the business of placing ads on your screen, and doing so in a manner that makes you more likely to click on the ad. That means making them as relevant as possible and walking that fine line between visiblity and unobtrusiveness. It also means getting their ads everywhere, and everything that Google does furthers that goal, directly or indirectly.

Google’s technology has two jobs: deliver ads, and to increase their audience. I know very little about how they deliver ads–that’s their proprietary process and, one might argue, the heart of their business. But they’re transparent about how they increase their audience. They provide arguably the best results of any general search engine available. With YouTube, they dominate Web video. They have a whole bunch of other free services and software–translation tools, Google Chrome Web browser, Google Maps, Google Earth, Google Books, Patent Search, Blogger, Mail, SketchUp, Images, and many more–that make it easier to use the Internet or provide online replacements for traditionally client-bound tools. By making it easier to use the Internet, they get more people on the Internet.

Google also produces and makes available an incredible amount of program source code that developers can use or include in their products for free. Just check out Google code sometime. It’s full of proven working code that Google paid their employees to develop, and is now giving away for free. It’s not that they’re altruistic. They know that by making it easier for developers to create quality Web sites, their audience is growing.

Two recent (well, one not so recent) developments show Google’s commitment. First, the Chrome Web browser. This is Google’s free browser, which is arguably the best on the market today. One might ask why Google would go to the expense and effort of creating a new browser and then make large parts of its source code available (see the Chromum project)? I can’t say for sure, but here’s what I think.

I think that Google wants to do things with the Web that other browsers (Internet Explorer, Firefox, Opera, Safari, etc.) don’t currently support. Although it’s often possible for Google to convince the people in control of those browsers to support new features, Google is left waiting for support. If they control the browser, then Google can start pushing new technologies on their own schedule.

Whatever the reason behind it, Google Chrome is building market share. It used to be that Microsoft’s Internet Explorer had 70 to 75% of the browser market, followed by FireFox in the 20 to 25% range, and everybody else was down in the noise. The most recent numbers I have put IE below 60% for the first time, Firefox still hanging in there around 20 to 25%, and the rest being shared by Opera, Safari, and Chrome. Except Chrome is taking market share, most of which is coming from Internet Explorer.

The more recent development is Google’s support of the WebM project, a high-quality, open, and free video format. I cannot overemphasize the importance of this development. WebM combines a container format with free video and audio codecs so that anybody can create and distribute video royalty-free without having to worry about patents or other intellectual property concerns. Google spent something like $100 million to obtain the rights to the VP8 video codec in order to make this possible. Then they turned around and made it freely available to anybody. Why? Because ubiquitous free video gives Google a huge increase in surface area–a larger audience–that they can exploit for the purpose of delivering ads.

From the outside, Google’s business plan really does look as simple as, “Make the Web easy to use so that we can deliver more ads to more people.”

In the process, Google is steamrolling over a number of entrenched companies who thought they had it made. Consider Adobe, whose Flash player is currently The Standard for online video. Back in 2007, Flash 8 had something like 95% (perhaps higher) penetration. That is, 95% of computers connected to the Internet had Flash installed. Why? Because of YouTube. When Adobe released Flash version 9, it achieved more than 90% penetration in just a few months, again in large part (perhaps primarily) because YouTube went to Flash 9 for their video. Adobe owned Web video.

But Adobe dropped the ball. For reasons I’ll never understand, Adobe still clings to the idea that Flash is for creating rich Web apps. The ability to do rich client things in a Web page is cool, and there was a time when Flash was the best way to do it. But browsers and computers are more capable now. I know from experience that it’s now much easier to build rich applications with JavaScript than it ever was with Flash. And all you need is a modern browser. There’s no need to download and install a Flash control to do it.

After Google’s WebM announcement last week, Adobe made a press release saying that they’ll support WebM “in a future version.” YouTube will continue to use Flash for low-quality videos. Starting soon, though, higher quality video will be delivered with WebM. You have to be blind not to see what’s coming: the eventual removal of Flash support on YouTube. But it’s already over for Adobe Flash. They will only see decreasing market share. And Adobe has nobody to blame but themselves. They ran into much the same thing clinging to their old .FLV format when the rest of the world was moving to .MP4. The reason? They make money by selling very expensive software packages that create video files. Much like their PDF tools, they give away the reader and charge a lot of money for software that creates the files that their free players read.

With WebM, all that goes away. There are already FFmpeg patches for WebM, and likely will be some very good free tools.

Microsoft, too, is getting steamrolled by Google. After Google’s WebM announcement, Microsoft said that they’re very excited about the new technology and that Internet Explorer 9 will fully support it as long as the user has installed the proper codec. If you’re not familiar with the world of codecs, don’t feel bad. Understanding codecs is not something a user should have to do. Finding and installing the proper codec can be incredibly frustrating and fraught with danger. If you go looking for a codec for Media Player, for example, you’ll find yourself confused and in very real danger of inadvertently downloading and installing some malware.

For Microsoft to say, “as long as the user has installed the proper codec” is like GM saying that the new car they sell you will be fully functional as long as you find and install a compatible engine.

And don’t expect Microsoft’s Media Player to support WebM any time soon. According to Microsoft’s own Information about the Multimedia file types that Windows Media Player supports, they don’t even support MP4. Granted, that article was written two years ago, but it covers Media Player 11, which is the most current version. That article says, “You can play back .mp4 media files in Windows Media Player when you install DirectShow-compatible MPEG-4 decoder packs. DirectShow-compatible MPEG-4 decoder packs include the Ligos LSX-MPEG Player and the EnvivioTV.” In other words, you have to install a codec made by a third party in order to play a video format that the rest of the world embraced five years ago.

The announcement of WebM is also pushing innovation in another area: the server. The day after the WebM announcement, somebody was streaming WebM from the Cherokee Web server. One day! This has some very interesting ramifications. An open media format combined with an open Web server (like Apache) means that a free media server is not far behind. There goes Adobe’s Flash Media Server business. And quite possibly Microsoft’s Home Media Server, especially if somebody releases an easy Linux configuration that includes this hypothetical (but soon to be realized) media server, backup and data recovery, and document management.

It’s interesting to note that Google hasn’t had to “target” any of these companies in order to take them out. In fact, Google probably isn’t even interested in “taking them out.” Google is just doing what it needs to do in order to grow the business. If it means investing hundreds of millions of dollars so that more people will come online to watch video, then so be it. If Google makes a few pennies every time somebody watches a video online, that hundred million bucks will be returned in short order.

The really funny part here is that both Microsoft and Adobe had to see it coming. It’s not like Google made a surprise announcement last week: there was a big splash when they acquired the VP8 technology a few months back, and Google has been telegraphing this move since at least 2007, when they paid $1.5 billion for YouTube. That kind of investment says, “We want to own Web video because we think we can make money at it.” No, Microsoft and Adobe saw this coming and knew that they were powerless to stop it. But rather than embrace VP8 and try to find a way to work with it, they clung to their own product plans hoping that some imaginary Maginot Line would block Google’s advance. Adobe, Microsoft, and other companies whose businesses are built on artificial scarcity (selling bits) are living in the past and will continue to see their market share stolen by companies like Google that can provide better products for free.

You’re going to see this same thing play out all over again in the world of television. Google recently announced a deal with Intel and Sony that will put Google TV on Sony television sets. Today, something like 25% of all new televisions sold are Internet ready. Google is ready to go there, and not just because it increases the surface area for their Web advertisements, but also because it gives Google a platform from which to launch an assault on the television advertising market ($70 billion annually in the U.S. alone).

Google’s competition for that market is a handful of old media companies and Madison Avenue advertising firms, both of which have grown fat and complacent. Sure, they’ve been hit by Internet advertising over the years, but it’s been more of a slow leak in a dike rather than a tsunami that overwhelms the entire system. Those companies probably aren’t smart enough to see it coming yet, but when they do see Google riding the wave, they’ll probably all hunker down behind the dike and hope for the best. And then complain bitterly (read: try to win through litigation) when they discover that they lost the war while they were sitting there with their thumbs up their butts trying to decide if they should do anything.

Remember, you heard it here first.

I’m not trying to paint Google in a bad light at all. On the contrary, I have nothing but admiration for them. They’re going about their business. If the entrenched companies can’t keep up, it’s not Google’s fault. While the old media companies are refining the horse-drawn carriage, Google is hard at work on the V8 engine. In the process, Google is making all manner of things available to Internet users and developers, and actually encouraging us to build products that leverage the free services that the company offers. Given the choice between begging for access from the old media companies or accepting the bounty freely offered by Google, I’ll throw in with Google.

The build is an MSBUILD script that compiles all of our projects and dependencies, and then copies the results to a staging directory from which we can run unit tests or build distribution packages for our internal customers (see below).  To this point, everything can be done with a minimal batch file script, the MSBUILD program supplied with the .NET development tools, and of course the subversion command line client.  We have one other tool called sendEmail that notifies me of the build status.

I’d like to notify everybody when the build fails, but doing so requires that I tell them why it failed.  And the generated build log is very large:  about 120 kilobytes, most of which is irrelevant.  The important information is typically the last 10 or so lines of the file, and that’s what I’d like to send to people when the build fails.  Those lines say, in effect, “The build failed for these reasons.”  A programmer who receives that message can quickly determine if it’s his responsibility, and take steps to fix the problem.

The only trouble I have is that there is no simple way with Windows-supplied tools to extract those pertinent lines from the file.  At least, I can’t think of a way.  But the GNU awk (gawk) can do it trivially.

When the build fails, the last thing that MSBUILD outputs is a line that says, “Build FAILED”, followed by some lines that describe the error or errors.  So all I need is a program that will go through the file, locate the “Build FAILED” line, and then output that line and all following lines to the end of the file.  It’s been 20 years since I did any awk programming, but this script was simple:

gawk "{ if (/^Build FAILED/) { doit=1 } if (doit) print $0 }" < buildlog.txt

Done and done.

The only problem I have now is deciding whether I want to install the full GNU Tools for Windows package on my server, or if I should just grab Gawk for Windows.  The full package is probably the right way to go because I suspect I’ll be needing some other tools in the future.

Either way, I’m annoyed that Windows doesn’t include these simple text processing tools.  I can perhaps understand why they don’t exist in desktop versions, but we do these types of things on servers all the time, and the standard server install should include a more robust toolset.

Above I mentioned “internal customers.”  In reality, we are our own customers.  There are only five of us here, and one of us doesn’t use the tools that the build creates.  In light of that, it’d be easy to take a more cavalier attitude towards our build process.  I’ve found, though, that things run smoother if, as a programmer, I think of the users of my software (in this case, the crawler subsystem and the tools that process the collected data) in much the same way as I would an external customer.  Even though the primary user of the crawler is me.  I wear a number of different hats around here (as does everybody else–we’re a startup, after all), and it’s useful to think of Jim the SysAdmin as a separate person from Jim the Programmer.  That way, when we can afford to hire a system administrator to take those duties from me, the systems will already be in place for him to step right in.

Like source code version control, a formal build process is one of those things that you don’t need to implement until the size of your project team exceeds zero people.

The document is freely available online as a PDF, although it can be difficult to find.  I just went searching for it again and couldn’t find the full version.  If I remember where I found it, I’ll post a link here.

Adobe Flash Player Update 3 (9, 0, 115, 0) and higher can play some MP4 files.  The subset of MP4s that Flash can play is described in Video File Format Specification Version 9.  That document gives you an idea of the MPEG-12 file format, although you probably want the full spec. if you’re implementing a reader.

The file format is quite flexible–perhaps overly so–but reasonably easy to parse once you grok the basic structure.  I coded up a quick MP4 reader in a day, and within two days had my web crawler extracting metadata from files I located online.  But then I ran into a problem:  my movie player would sometimes hang when trying to play a file.  The really weird part was that the movie would play fine if I downloaded it first.  It was only when trying to play from online that I experienced the problem.

It didn’t take long to find the problem.  Or at least part of the problem.

Data in the Part 12 file format is organized in “boxes.”  Those boxes contain all manner of information:  a file header, overall movie information, information about the individual tracks, synchronization data for different tracks, etc.  Part 12 describes the overall structure of the file and the contents of the “moov” box that contains basic movie metadata (number and types of tracks, duration, codecs required to play the tracks, etc.).

Another box, called “mdat,” contains the actual movie data:  the video and audio information that will be played.

In order to start playing a movie, a player must have the metadata.  The player can’t play the first frame until it knows how to decode that frame.  The movie data, on the other hand, can be delivered relatively slowly:  at whatever the playback speed is.  In other words, playing a movie consists of these steps:

Read the metadata.
Determine if the movie is playable with this player.
repeat
    Read movie data (audio, visual, etc.) frame
    Render frame
until end of movie

So it makes sense to organize data in the movie file to facilitate that.  Right?  In fact, the Part 12 document makes two very pertinent recommendations:

2)  It is strongly recommended that all header boxes be placed first in their container: these boxes are the Movie Header, Track Header, Media Header, and the specific media headers inside the Media Information Box (e.g. the Video Media Header).

8)  It is recommended that the progressive download information box be placed as early as possible in files, for maximum utility.

The emphasized “recommended” is in the original document.

There are good reasons for these recommendations, as I discovered in the first problem file I looked at.  In that particular file, the “mdat” box, which contains the frame data, is placed at the front of the file:  immediately after the file header.  “mdat” is 89 megabytes long.  It’s followed by the the “moov” box that’s a little less than two megabytes.  A movie player has to download 89 megabytes of stuff before it can get to the metadata that tells the player how to play the movie.  89 megabytes might not sound like much, but at 10 megabits per second (which would be a very fast residential connection here in the U.S.), it’s a minute and a half.  Nobody’s going to wait a minute and a half for their video to download.

I suspect that whoever made these movies has no idea that they’re effectively unplayable over the Internet, and might not even care.  I care, because I’m not going to download the entire movie just to see if I’m really interested in watching it.

What surprises me is that video player software doesn’t recognize this and skip over the movie data to get to the metadata.  The HTTP 1.1 specification makes it very easy to get a partial file.  The movie player should see that the “mdat” box comes before “moov”, and make another request to get “moov”.  It could then go back to “mdat” after digesting the metadata.

I wonder how hard it would be to create a tool that fixes those backwards movies . . .

grep -o "http://www.example.com/feeds/rss/[^.]\+.rss" feedIndex.html
  | sort -u | xargs wget

What makes that possible is the GNU tools–a standard set of tools that mimic and extend the standard tools that have been available for Unix-based systems for decades.

Although the Windows command line supports piping, it doesn’t include a comprehensive set tools that were designed to work together the way the GNU tools were designed.  The Windows toolset is primitive, and not up to solving this simple task.  I used GNU grep for Windows to extract the URLs and save them to a file, TextPad to sort and then manually remove duplicates, and finally GNU wget for Windows to download the files.

This isn’t the first time I’ve had to resort to a hodgepodge of tools to solve a problem that I could solve without trouble if I had the GNU tools.  But in the past, downloading and installing all the GNU tools for Windows was a giant pain in the neck with version conflicts and such.  That’s not a problem any longer.  Today I discovered the getgnuwin32 project, which automates the process of downloading, installing, and maintaining a full set of GNU tools for Windows. 

The few tools I’ve used so far work exactly as expected.  Time (and some effort:  it’s been a while since I used the GNU tools) will tell if this is as useful as I hope it is.

Update (later the same day):
There is one slight problem:  some of the GNU tools have name conflicts with the Windows tools.  sort is a good example.  If I tried the above command line on a Windows machine, it would try to invoke the brain damaged Windows SORT utility, which is so bad that whoever wrote it should die from embarrassment.  It depends on where in your path you put the GnuWin32\bin directory.  Either way you go, name conflicts are going to give you some headaches.

I’m thinking that, since most programmers don’t even know that the Windows command line exists, I’ll put GnuWin32\bin ahead of the Windows directory that contains the standard tools.  Or maybe I should just delete or rename SORT and any other tools that have conflicting names.  It’s not like I ever run batch files that I get from other people.

A little debugging with Process Explorer revealed the culprit:  the USB driver.  Odd, that.  Still going with the bad mouse theory, I figured that the mouse had somehow caused the driver to freak out.  So I rebooted the computer.  That solved the problem.  For a bit.  Then the mouse stopped working.

Now here’s the odd part.  As long as I leave the mouse plugged in to the USB port, everything else is fine.  None of the USB ports work, but there’s no excess CPU usage.  But as soon as I unplug the mouse, CPU usage goes to 100%.

I’ve looked around online and have tried most of the solutions others with this problem have tried:  uninstalling the devices and letting them re-install on reboot, updating the driver, etc.  All to no avail.  The mouse works fine for a while when I first restart, but it stops working after some unpredictable amount of time.

The one solution I haven’t tried yet is re-installing the operating system (Windows XP Pro).  I hesitate to go to that effort if it’s not required, but I have no idea what else to try.  The USB ports on this Dell 630 notebook are built into the motherboard, so there’s no chance of just replacing them.  And by the time I pay for a new motherboard and the labor to install it, I’m out about the same amount it would cost me to just buy a refurbished replacement computer.

I can limp along without USB ports for a while, but it’s not a good long-term solution.

I’d sure like to hear from anybody who’s had a similar problem, and learn how you solved it (if you did).  I suppose I’ll try the full disk wipe and OS restore, as much as I hate to do it.  I can’t think of how that’d solve the problem, but I also can’t think of any other possible solution.

How long?  More than a minute.  Seriously.  And during that time, any other process that is trying to access files on the NAS gets really slow.  Sometimes, deleting the file causes the NAS to become unresponsive so long that other processes’ IO requests time out and the program crashes.  That is not a happy state of affairs when I’m running a job that takes 36 hours.

It appears that the slowdown is due to indirect block pointer updates in the ext3 file system, as described in this post.

Is this a fundamental shortcoming of the ext3 file system?  If it is, what are my options?  The Thecus supports a file system called ZFS, but from what I’ve read about it online, I don’t want to go down that path.  I wonder if a firmware upgrade would solve my problem.

I tried drilling holes in the case (after opening it and removing the drive, of course) on one of those Seagate FreeAgent drives.  Getting the thing apart was quite a chore, and I had a fun time making a mess drilling holes in the case.  The unit tested fine afterwards when copying small files to it, but it went unresponsive after about three gigabytes of the large file.  It’s difficult to say what went wrong.  I suspect that the USB-to-SATA electronics, which were marginal to begin with, finally gave up the ghost.  At some point I’ll pull out the 1 TB Seagate drive that’s in there and see if I can use it as a normal SATA drive.

Yesterday I picked up two Antec MX-1 external drive enclosures and fitted them with 500 GB drives.  I got them installed last night, and initial results are positive.  I’ve heard that there have been some fan failures with the Antec enclosures, but a search didn’t reveal an inordinate number.  For the price (about $55 each, with tax), I might pick up a third just to keep on hand in case a fan does fail.

The drive comes with USB and eSATA cables.  I was all ready to go eSATA until I discovered that my server doesn’t appear to have a spare SATA port inside.  I suppose I could go eSATA at the office and USB at the datacenter.  I might still do that, although it’ll have to wait until I can take down that office server.  It serves other important duties here, so I can’t just shut it down without affecting a lot of other things.

In any case, I think (hope) that my removable drive troubles are over, at least for a while.

It used to take about three and a half hours to copy data (a couple hundred gigabytes) from the server to the removable drive.  Recently it’s been taking on the order of 10 to 12 hours.  At first I thought it was another idiotic problem with caching, similar to the problem I had copying large files between servers, except this copy would eventually complete.  The odd thing was that when I started the copy it would proceed at the expected rate and at some point slow to a crawl.

So I wrote my own program that reads a gigabyte at a time from the local drive and then writes it to the USB device, timing each write operation.  Running locally (at the office), the program reported a steady 24 MB/sec write speed, and copied the entire file at that rate.  Run at the data center copying the same file, the program reported the same 24 MB/sec for the first 20 gigabytes or so.  Then it slowed to about 4 MB/sec.

That smacks of a thermal problem.  Either the drive electronics or the server’s USB port was overheating.  I quickly eliminated the server’s USB port as the problem by hooking up a different USB device and checking to see that the server could pass more than 50 gigabytes of data without trouble.

So the problem is with the FreeAgent drive.  If you spend a little time searching online, you’ll see that other people have experienced overheating problems with the FreeAgent drives.  And looking at the design, I can see why:  the only ventilation is at the bottom of the device where the electronics are.

The picture on the left, above, shows the drive as we typically would place it in the rack at the data center.  It sits on top of one of our servers.  The spot where it’s sitting is directly above one of the disk drives.  That spot is cool to the touch when I tested it yesterday.  Note, however, that you can’t see any ventilation holes.  Those are on the other side of the enclosure, as shown by the red arrow in the picture to the right.

Since air enters the cabinet from where I was standing taking this picture, and flows towards the back, mounting the drive as shown on the left doesn’t allow for very good airflow.  So yesterday I placed the drive in the cabinet as shown on the right.  Then I ran my test program.  I was able to write about 90 gigabytes before the drive slowed down.  I’m convinced now that it’s a thermal problem.

I don’t quite know where to go from here, though.  I think the first thing I’ll try is lifting the drive higher off the surface it’s sitting on.  That should allow for better airflow, and perhaps will be enough to keep the electronics cool.  (The problem, according to what I’ve found online, appears to be the USB to SATA conversion electronics at the base of the drive enclosure.)  If changing the drive location doesn’t solve the problem, I’ll have to find a different model of removable drive that has better ventilation or better heat tolerance.  Perhaps it’s time to visit Fry’s and see about buying an enclosure that’s designed for use in the warm environment of a server rack.

I don’t know how, but I somehow managed to get the Malware Defense “anti-spyware” program on my system at home.  Fortunately for me, it doesn’t do anything malicious like delete files or install botnet sofware.  It just continually pops up virus warnings and giving opportunities to install.  For a price, of course.  If you pay, they go away.

The removal instructions I came across weren’t complete, as I completed those steps, rebooted the system, and the thing came right back.  I finally tracked down and eliminated the richtx64.exe trojan, which I think is what was re-running Malware Defense.

I’ve been running my computer for years without any kind of active anti-virus or such, and this is the first time I’ve ever been infected.  Now I’m not sure what to do.  I certainly won’t go back to Norton after the troubles I’ve had with them, and I don’t hear good reports about McAfee’s offering, either.  Is there a good anti-virus, anti-malware package that works, is inexpensive, and doesn’t take inordinate amounts of CPU time?

Update 12/28:

It took a while, but with some research and downloading and running a few cleanup utilities, it looks like I was successful in disinfecting the computer.  The thing kept getting re-infected whenever I’d reboot, and it would prevent me from installing or running common anti-malware utilities.  I found a program called rkill that kills common malware processes, and then I could install and run cleanup software.  This morning, a complete scan with Malwarebytes’ Anti-Malware reported zero problems.  I then installed Microsoft Security Essentials from a file that I downloaded from a different (uninfected) computer.  It reports no problems.

Darrin Chandler brings up an interesting point in the comments:  it’s all a matter of weighing the risks.  I’ve gone years without any kind of malware problems.  Even when I had anti-malware applications installed, they never reported that they’d blocked anything.  And those programs are very quick to notify whenever they see anything even vaguely suspicious.  So, as Darrin points out, my risk of being infected is pretty small.  However, the cost of being infected is fairly high.  It cost me most of a day to get rid of it.  And I was fortunate that it doesn’t seem to have deleted any files.  I have no idea if it copied anything from me.  I’m not too worried since I don’t keep financial information on this machine.

I’m hoping that Microsoft Security Essentials works well and doesn’t cause problems by being too chatty or sucking down too many resources.  We’ll see how it goes.

I discovered useful thing #2 when my XmlReader threw an exception trying to parse the JSON I fed it. I originally thought that the problem was with the JSON-to-XML conversion. But then I fed the JSON to JSONLint.  It turns out that the string "It\'s an error" contains an error.  Escaping the apostrophe is an error in JSON.  There are only a handful of characters that can be legally escaped.  It’s nice to know that the site was in error and not my JSON-to-XML converter.  Either way, I still have to gracefully handle the error.

I had hoped to use the Windows command FINDSTR as a substitute for grep.  No such luck.  FINDSTR has two problems that make it marginally useful at best.  First, there’s no switch that corresponds to grep’s --only-matching (-o) option.  If you specify --only-matching, then grep outputs only the text that matches the query expression rather than outputting the entire line that contains the match.  FINDSTR lacks that option, making it useless for many of the things I do.

The other problem is very odd.  Both grep and FINDSTR are line-oriented tools.  But FINDSTR’s definition of a line is inconsistent when working with files whose lines end with just a line feed.  For example, if I’m looking for all lines that contain the text “.xml”, I’d write this:

FINDSTR /R "\.xml" file.txt

The /R switch tells FINDSTR to treat the search string as a regular expression.  I could have done a literal search in this instance, but I want to illustrate the error.  FINDSTR correctly finds and outputs all of the lines that contain the string “.xml”.

What I really want, though, is just those lines that end with “.xml”. So the command would be:

FINDSTR /R "\.xml$" file.txt

FINDSTR doesn’t find any lines that end in “.xml” unless I convert the file so that it has CR/LF line ends. grep correctly handles both line end conventions. Since I can’t guarantee the format of the files I work with (I often am working with files that I download with wget), FINDSTR is practically useless if I’m doing regular expression searches.

My advice, download GNU Grep for Windows.