* I am sorry for the language below, but I am very frustrated right now *

I f*cking hate software. It never, ever, ever, EVER works as it should be. After 7 hours trying to find an acceptable solution to this, I am so frustrated right now that I could smash whatever is in front of me.

So, as you remember from my yesterday’s blog post, I am trying to export 1080p for use with the Xbox360 and the PS3, from interlaced HDV videos. The only utility that came close but not quite, is SUPER. SUPER is just not as flexible as it should be, because it only allows for 29.97 fps encoding, while my camera also supports 24 fps too. Everything else seems to be ok with that app.

My previous efforts included using the freeware XViD4PSP and the very popular MediaCoder which failed miserably in many levels: from encoding errors, to wrong aspect ratios., to crashes (they all use ffmpeg, mencoder, x264 btw). So, I went the commercial route. I tried ArcSoft’s MediaConverter 2.0 which came with my hardware h.264 encoder stick, but it failed miserably with HD resolutions. I downloaded the new version of MediaConverter, 2.5 but this doesn’t get the aspect ratios correctly either (it doesn’t “stretch”, even if the “stretch” option is checked), neither it de-interlaces. Another commercial utility I tried which was possibly riddled with spyware but I had no choice but to try it, MP4Converter, wouldn’t encode at all, plus it wouldn’t go more than 3 Mbps which is too low for 1080p.

Want to go with QuickTime Pro? Slow-ass h.264 encoder (worse of all in terms of encoding speed so don’t ask me to buy FinalCutPro and a new Mac either), with no de-interlacing support and with keyframe problems (visible “ticks” every second or so in the video no matter what keyframe settings you set), as of with versions from since 6 months ago onwards. Apple stills sleeps over the issue.

PS3Video9 you said? Terrible interface aside, it won’t de-interlace, no matter if you add (as suggested) the -deinterlace ffmpeg option in the CLI. It’s like ffmpeg, to which PS3Video9 is a front-end, doesn’t take it into account at all. FFmpeg “writes your request in its balls”, as we say in Greece very accurately. Either that, or its de-interlacing sucks balls.

Overall, from my experience with at least 7 shitty utilities which for some reason are very popular, their main problem seems to be flexibility and open mind when they develop their software. They just don’t think too much about the problem they are trying to solve. If your video is already 4:3, good luck trying to stretch it. If it is widescreen, non-interlaced (which that would mean that this would possibly be a pirated movie/show), you will be in better luck. But if you have your OWN footage from your own HD camcorder (in other words, if you want to convert legal home videos), you are f*cked. It seems that people are creating encoding front-ends for pirated DivX movies rather than interlaced home footage (that is usually anamorphic too).

So, mediocre software, all over the place. I have not seen a SINGLE application that is able to “understand” a 1440×1080 HDV interlaced file (with the anamorphic flag set *or* not), and be able to transform it to 1920×1080p widescreen at the requested industry standard fps settings or 10 Mbps bitrate. There is ALWAYS something missing from each app, or something not working right. ALWAYS.

God damn it, I f*cking hate software. If this problem is not solved within 1-2 months, I will sell my camcorder. There is no f*cking point getting sunburns while shooting under the f*cking sun if I can’t enjoy my f*cking footage as I should be in a f*cking 1080p TV. SUPER does most of what I need except providing an option for 24fps, but thing is that I want to shoot 24fps with my camcorder and it’s shit like that that keep me from doing it. I am extremely frustrated right now. Things just don’t work properly. I have tried everything and this ordeal makes me have ZERO tolerance for software. > : (

Update: Finally, some half-assed progress on PS3Video9. Add this to your C:\Program Files\Red Kawa\Video Converter\Profiles\Video\PS3.xml .

<profile name="PS3-Eugenia" encoder="PS3 1-Pass - FFmpeg MI" position="0" duration="0" vcodec="MPEG-4 AVC" vprofile="Main Profile" vlevel="Level 4.1" vmode="ABR" vbitrate="8192" vquality="0" vquantizer="0" vwidth="1920" vheight="1080" vaspect="Original" vframerate="29.97" varenabled="true" varmaxres="1920x1080" varmod16="false" vartotal="false" vafenabled="true" vcroptop="0" vcropbottom="0" vcropleft="0" vcropright="0" vpadtop="0" vpadbottom="0" vpadleft="0" vpadright="0" vcli=" -deinterlace " vavsenabled="true" vavsautofps="false" vavsmanualfps="0" vavsconvertfps="false" vbufsize="20000" vminrate="0" vmaxrate="25000" vkeyint="300" vthreads="1" vcabac="true" acodec="AAC-LC" amode="ABR" abitrate="128" achannels="2" asamplerate="0" avol="100" acli="" />

Now, in order for ffmpeg to de-interlace, your source video file MUST be in the YUY2 colorspace. So, as per the tutorial yesterday, install the Huffyuv lossless codec, change its settings as described on one of the last paragraphs to use the YUY2 colorspace, and then use that exported .avi file to re-encode to 1080p using my profile above. Apparently, FFmpeg is extremely picky about YUY2 (which is not of the highest-quality btw) and it won’t even support that in some other lossless codecs that use that same colorspace (e.g. Lagarith). Not sure if the resulted file will playback on XBox360 (I will need testing), but it should play ok on the PS3. Anyways, at last, something that kinda works, I can definitely live with the YUY2 restriction as long as everything else works ok. It wasn’t easy to figure this out btw. It took 8 hours of trial and error.

Update 2: I talked too early. Apparently ffmpeg’s deinterlacer SUCKS. This leaves me with NO application that can create good 1080p h.264 videos out of an HDV interlaced/anamorphic source with reasonable fps/bitrate/aspect-ratio-resize flexibility. I talked to JBQ about it and he agrees that the features I need to work flawlessly are pretty basic and he said that what’s wrong is the lack of process that all these developers have. There were no product requirements ever specified when they had the “cool idea” to start developing an encoder front-end.

It’s official: Software sucks.

Update 3: Apparently, in order to de-interlace correctly with FFmpeg, the -deinterlace CLI option must be the FIRST argument in the ffmpeg command line! After I did that, I got a correct de-interlaced output! I hope the PS3Video9 developer fixes this.

I also downloaded and checked Adobe Premiere Pro CS3, only to find out that the trial version has no HD support (stupid Adobe, how can I test my hardware and my exporting options now?). However, by simulating the resolutions it seems that Premiere is able to create a correct HD h.264 output.

Update 4: So, to get it right, use ffmpeg from the command line. Install PS3Video9, just don’t use its UI (at least until the deinterlace bug is fixed). Copy your Huffyuv YUY2 interlaced .avi file on C:\Program Files\Red Kawa\Video Converter\Tools\FFmpeg\ and navigate there with a DOS command prompt. I use the follow commands for my projects, which should be relevant to all people with HDV and DV *NTSC widescreen* camcorder footage who want their videos to play nice on their EDTV/HDTV via the PS3 or the XBox360.

1080/30p (for 24 fps use “8192″ bitrate instead of “9216″ below):
ffmpeg.exe -deinterlace -i "C:\Program Files\Red Kawa\Video Converter\Tools\FFmpeg\XXXX.avi" -y -f mp4 -title "XXXXX" -vcodec h264 -level 41 -s 1920x1080 -r 30000/1001 -b 9216k -bt 9216k -bufsize 20000k -maxrate 25000k -g 300 -coder 1 -acodec aac -ac 2 -ab 128k "C:\Program Files\Red Kawa\Video Converter\Tools\FFmpeg\XXXX.mp4"

720/30p:
ffmpeg.exe -deinterlace -i "C:\Program Files\Red Kawa\Video Converter\Tools\FFmpeg\XXXX.avi" -y -f mp4 -title "XXXXX" -vcodec h264 -level 41 -s 1280x720 -r 30000/1001 -b 4096k -bt 4096k -bufsize 15000k -maxrate 16000k -g 300 -coder 1 -acodec aac -ac 2 -ab 128k "C:\Program Files\Red Kawa\Video Converter\Tools\FFmpeg\XXXX.mp4"

480p/30p:
ffmpeg.exe -deinterlace -i "C:\Program Files\Red Kawa\Video Converter\Tools\FFmpeg\XXXX.avi" -y -f mp4 -title "XXXXX" -vcodec h264 -level 41 -s 852x480 -r 30000/1001 -b 2048k -bt 2048k -bufsize 10000k -maxrate 11000k -g 300 -coder 1 -acodec aac -ac 2 -ab 128k "C:\Program Files\Red Kawa\Video Converter\Tools\FFmpeg\XXXX.mp4"

For 23.976 fps instead of 29.97 fps, change the “-r 30000/1001″ to “-r 24000/1001″.

“The war is hardly the only area where the Bush administration is trying to expand its powers beyond all legal justification. But the danger of an imperial presidency is particularly great when a president takes the nation to war, something the founders understood well. In the looming showdown, the founders and the Constitution are firmly on Congress’s side.” More here.

The Founding Fathers are probably turning on their graves right now over Bush. Thing is though, that what he has done is not illegal. He made sure that he passes bills that gave him that jurisdiction. And somehow he passed these bills without anyone noticing. Now, no matter what Bush does, he’s on the clear: he has his own laws on his side. The real problem in the situation is not Bush (he’s going away soon), but with these laws in place, future presidents can be as boneheaded too.

This whole deal kinda reminded me how the democracy in ancient Rome fell. It was little by little, year by year, within 50 years. It was not a one-thing-off shake off thing and so for the next 100 years Roman citizens still thought that they actually had democracy. But they didn’t. History seems to be repeating itself.

Update: Wow, I had no idea about this book (found it by googling after writing the above).

The most cost-effective way to play your own HDV files as HD videos on your TV is either the Sony PS3 (or XBox360) or the Ziova CS505. With the latest firmware the PS3 supports playback of h.264 files up to 1080/30p. The Ziova CS505 goes up to 1080i for some filetypes, and 720p for others (no h.264 support btw). So basically, if your TV is 1080i and you don’t have lots of money, go for the Ziova product. If you have a 1080p TV, you have more money and more time, and you would find the Blu-Ray and gaming abilities useful, get the PS3.

I have ruled out the following ways to playback your personal HD files on your TV:
- A Media PC: can be more expensive than the PS3 before its specs can handle 1080p, takes too much space and if you run it with Linux instead of Windows it can be difficult to configure properly. It can be a flexible solution with lots of extra features, but it can also be a nightmare. Decide with care.
- AppleTV: Good price, nice interface, but it goes only up to 720/24p, and most HDV camcorders grab 30fps, not 24. AppleTV would be THE device to go, if and only if, Apple upgrades it to support full HD resolutions one day.
- HDV Camcorder: you will have to use tapes all the time and deal with 2-hour battery life each time. Yuck.
- HD-DVD/Blu-Ray burner and separate TV player: way too expensive, buggy burner software up to this day.
- DivX-enabled upscaling DVD players: *Most* of these models only have NTSC resolution DivX support, not HD. Don’t fall into the trap of their “1080p upscaling” marketing because that doesn’t usually apply for your own mpeg-2/4 files.

Now, here is how you could export your HDV home videos to HD, from let’s say, the $130 consumer NLE Sony Vegas Movie Studio 8 Platinum (VMS8-P), which is the best deal for the money compared to other consumer NLEs in terms of feature-set. VMS8-P is not the easiest to operate and it has its own quirks and bugs, but overall, I found it to be the best bang for the buck if you need flexibility and power. Most consumer NLEs don’t allow you to save on anything above 1080i-HDV (that’s 1440×1080/60i) and that’s ok for Ziova (export either as WMV or MP4 or MP2 in 1080i or 720p, Ziova should be able to handle it) but that’s not as optimal for the PS3 which is able to output 1080p progressively, so we will need to pull a few tricks and re-encode from 1080i-HDV resolution to true 1080p without loss of quality.

First thing to do is install the Huffyuv lossless codec and MediaCoder. The Lagarith codec would have been my personal choice instead of Huffyuv, but it is not supported by most freeware encoder front-ends, so we are going with Huffyuv for this tutorial. Edit your HDV video clips as you normally would on VMS8-P. When your video editing is finished, select “render as” from the “File” menu, select the .avi method and the “1080/60i-intermediate” template. Click “Custom”. From the “Video” tab there is a “Video Format” drop-down menu. From there, select the Huffyuv lossless codec. Encode your finished work using these settings and a huge.avi file will be produced. It is not uncompressed, but because it’s lossless, the filesize will be big (about 2 GB per minute of video length).

Open MediaCoder, load the newly created .avi file, and from the “Extensions” menu select the “Game Consoles / PSP”. Select “yes” to revert settings, select either the default MPEG4 or AVC versions (note: the AVC HD versions produced by MediaCoder don’t playback on Quicktime/VLC so you might want to go with the default mpeg-4 encoding option to be on the safe side), and then close the popup PSP window that MediaCoder has opened. From the “Video” tab at the bottom of MediaCoder’s main window change the bitrate to 8192 Kbps or more. From the “Picture” tab on the Resize option type 1920×1080, select your source frame rate (usually 29.970 on NTSC), Aspect Ratio 16:9. Then, click on “Effects” button below and from the “De-Interlace” option select “Linear Blend” (not sure if it’s the best option) and then click “ok”. Select 128 Kbps from the Audio tabs on the right by changing the “target quality” to “streaming CBR”. Then, start encoding. Depending on the speed of your PC and how lengthy your video was, you should be having a 1080p file now.

Instead of using MediaCoder, you could use XviD4PSP, which is more well-tested against the PS3. However, in order to force that utility to de-interlace, you first will have to click “configure” on the Huffyuv lossless codec setup under Vegas’ .avi export dialog, and tell it to use YUV2 instead of its default. Then, you export normally from Vegas and you load the .avi file on XviD4PSP. *If* it has black bars left and right of the image (if it shows up with wrong aspect ratio) then crop it by 240 pixels on the left and 240 pixels on the right. Then on the next screen you select the PS3 preset format, and you adjust the bitrate to 8192 Mbps (or more), resolution 1920×1080, Aspect Ratio 16:9. On the “Advanced” tab click on the “De-Interlace” checkbox and then click “ok” and then “Start” to encode. That resulting HD .mp4 file is compatible with the PS3, Quicktime and VLC.

You can burn a CD or DVD with these files, as normal files in the optical drive filesystem (not as DVD-video), or copy them to PS3’s hard drive or a USB stick or via networking, and then enjoy them on your PS3 and HDTV.

Update: Using the second method described above, I have created this 5 MB test 1080p file for testing with the PS3 or XBox360, shot with my HV20. Let me know if it works on your console (if you have the latest firmware installed).

Update 2: It seems that only the SUPER encoder will create valid –and with fewer headaches– 1080p files for the XBox360 and PS3. Works well with avi source files using Huffyuv btw, but it will only accept 29.97 fps as output, no matter if your source video is 24fps — which makes it useless for people with Canon HV20 cameras who shot in 24f mode.

Update 3: Read the Update on this blog post for the best way yet to convert interlaced HDV files to PS3/XBox360.

I frequent a number of video-related forums lately, and this question pops up all the time: what’s the best practice for videos to upload on YouTube and other video sharing sites.

Youtube encodes your video in 320×240 QVGA format and it has a time limit of 10 minutes and 100 MB size per clip. Other sites have similar size or time limitations, but few don’t. Revver re-encodes on 480×360 and Veoh on something close to 500 pixels wide. Here are a few pointers:

1. If your clip is smaller than 10 minutes, you can safely decide to encode it and upload it in VGA 640×480 format. If it’s bigger than 10 minutes, please note that YouTube won’t accept your video. Regardless, if your video is too long select the QVGA exporting size.

2. If your video is widescreen, make sure you use your encoder’s options to create a letterbox. You can choose to not letterbox and instead export using a widescreen size (e.g. 640×360 or 320×180), but depending on the kind of codec you will use, some video sharing sites won’t be 100% compatible recognizing it and they will “stretch” your video vertically making it look ugly. To be on the safe side, learn how to letterbox your videos.

3. If you decided to export in VGA size, use 1024 kbps video bitrate and 64kbps audio. If you went for QVGA, choose 320 kbps video and 64kbps audio.

4. The preferred codec exporting format is mpeg4. Choose either XVid or MP4-Simple profile, with CBR bitrate encoding (VBR bitrate can cause compatibility problems). Avoid WMV, as depending on the encoder used it has A/V sync issues when re-encoded by most video sharing sites.

Useful applications to do the job properly (with letterbox support) is MediaCoder on the PC side and QuickTime Pro on the Mac. FFmpegX won’t letterbox.

Haha… So basically, Apple bent over to Cingular to not include support for native apps because –as it’s well known– carriers are afraid of native app stability but mostly security problems, and now they get their full blown security hole from the browser. I can only suggest two things to ATT for that (expected) overreaction over native apps: either remove the browser from the iPhone (*tsk*), or free the damn SDK and let users have native apps instead of useless web “apps”.

At the beginning of our relationship we were like most young couples: we would occasionally argue about stuff and we would have about 2-3 major fights per year. But as years go by, things are getting so much smoother. Sure, there is some arguing every now and then that fades away 5 minutes later, but we haven’t had a big fight for more than a year now.

Either we are getting old and too bored to fight (scientifically: sex drive dying), or we learn about and love each other more and more every day. It’s so great being married to the most beautiful man in the world (even if he was a bit grumpy in the pictures ).

A view of the Stanford campgrounds at Palo Alto, CA, USA. We spent about an hour there yesterday with JBQ and on the way back we stopped at Suraj, an Indian restaurant with lovely lamb dishes, for dinner.

Video shot with a Canon HV20 HD camcorder, a fluid tripod head, an ND-6 filter, a polarizer and Tiffen’s “soft” HDTV/FX-3 filter at 1/60 shutter speed but with variable exposure & aperture settings. 720p HD version of the video available here (”save as” link, 180 MB), or view it via Flash video below.

The video portion is licensed under the Creative Commons ‘Attribution’ 3.0 License. Feel free to chop, remix and re-use the visuals the way you want to as long as the copyright is kept intact and credit is given where is due. However, the audio portion of the video (song “Meant to Be” from the album “Woods of Chaos”) was used after specifically receiving permission by its artist, Rob Costlow. To license Rob’s beautiful music please check RobCostlow.com or Magnatune.

Update: Of course, that’s “Stanford Campus”. not campground.

Below is a list of gadgets that you should be owning if you already haven’t, given that their prices are fair for the features they bring:

* If you already don’t own a smartphone, get one. The cheapest, but at the same time most well-crafted all-around smartphones are the Nokia Symbian S60 3.1 ones. I would recommend either the Nokia 6120 or the Nokia 5700 as they are the cheapest of all for the features they offer — while running the latest version of the operating system.

* A camcorder. If you just need “a” camcorder, get the Canon ZR800 ($230). If you need HD, then don’t even think getting anything else other than the Canon HV20 ($800). Consumers get it because the Indies get it, and the Indies get it because it’s cheap for what it does.

* The iPod Shuffle clone! It works as well as the real Apple Shuffle, but it costs $22 rather than $79.

* Kodak EasyShare C875 8 megapixel. I own this camera and I love its UI and ease of use. Picture quality is not bad at all either. It sells for less than $140.

* Sony MDR-W20 in-ear-but-over-the-head headphones for sports. I can’t find any picture of it online, but we bought a pair yesterday at Frys for just $14, and quality/loudness was extremely good. Better than some other headphones that sell for much more. It’s the only “in-ear” model that actually fits in my ears.

I see this or this picture and I am getting jealous. There is no consumer small-sized camcorder or even a prosumer one that will give you that background blur out of the box without some serious/expensive tweaking with 35mm adapters and lenses. I (and most other enthusiasts) need a consumer-priced HD camcorder that will give us out of the box that elusive “film look” instead of “Johnny records his dog taking a piss at his living room” look. It is the holy grail of indie filmmaking.

Yeah, I know. The consensus is that cameras that cost less than $20,000+lenses don’t give you that film look out of the box. Which is why there is a whole “geek” filmmaker community that tweaks the hell out of their cheaper cameras to get close to that elusive professional movie look. Now, these guys are just *accepting* the situation, sit tight and do with whatever hardware and software filters they got in their disposal. I can’t just do that. Being the perfectionist that I am I want a product that is able to do exactly what I need. The real problem is that most people think that it is not physically possible to create a small camcorder with good background blur and the right contrast values to achieve the “film look”. That’s what they think.

But in reality, IT CAN be done. And it can be done by using a big APS/DSLR-sized sensor (or bigger) instead of the tiny sensors currently found on the consumer/prosumer camcorders. If you have a big sensor and a good-enough built-in lens, you can achieve the film look just fine — and without ugly hacks. Problem is, nobody manufactures such a camera, possibly because of the following five reasons:

1. In order to achieve good background blur in a small-sized body you need to sacrifice the zoom. For example, while even the cheapest camcorder will be able to do 24x optical zoom, such a camcorder I am proposing wouldn’t go above 4x. It’s how it is. But the thing is, that’s acceptable! Such a camcorder would be used by filmmakers and enthusiasts, not tourists who would want to zoom to the Great Wall of China from 2 miles away. IMO, Companies are *afraid* to release a camcorder with only a 4x zoom, it’s a risk of their image.

2. Creating a big sensor, is more expensive than creating a smaller one. You see, the bigger the chip physically, the more “dust” & “errors” it can accumulate during manufacturing resulting in a higher amount of “bad” chips straight out of the factory. However, while price would be higher, don’t expect it to be more than 30% than let’s say, Canon’s tiny Digic-II.

4. They are afraid that such a $1000 camera would kill their $3500 cameras.

5. They are idiots. They never did a market research to try fill up that niche, or their ‘ingenious’ engineers never thought about the problem.

To backup my claims, here are two digital *still* cameras that have the hardware features I am after: the Sony DSC-R1 and the Sigma DP1. These guys recognized that there was such a specific niche in the digital photography and they created the appropriate cameras — although they are not well-sold because in the digital photography space you can get a DSLR plus the right lens that will give you background blur for less than $1000, so it makes sense to go for a DSLR rather than these specialized digicams. But in the camcorder space there is no “DSLR equivalent” for a thousand bucks, so such a product would immediately catch on with enthusiasts. And yet, no one has created anything like that…

In my opinion, such a camcorder would easily sell for $1000 with a bad-ass body look similar to this and the following features: 1920×1080 24/30p, 1440×1080/60i, 1280×720 24/30/60p, Compact Flash as a medium, h.264 recording format (even if it doesn’t create as good quality as HDV’s mpeg2), no need for viewfinder, flexible focus, image stabilization, enough manual options and other standard conveniences. As long as I get my background blur, field view and depth out of the box, I don’t need support for external lenses, although if the right steps would be taken during design, they could make it easier for accessory companies to create 35mm adapters for lenses — although this could just be an extra option.

For those who don’t have enough experience with optics are probably thinking “Eugenia is dreaming, if such camcorder doesn’t exist there is a good reason why it doesn’t“, but that isn’t true. I am ranting over this because the technology to create such a camcorder (on the cheap even) exists and there is an established niche market for it too. It’s just that no one took the steps to put the pieces together and create one for some strange reason. If someone decides to go for it, it would be similar to what Google did with Gmail. Here you had about 100 email services online competing with each other giving from 1 to 10 MBs of online email space, and then, there you had Google coming out of the blue giving away 1 GB. They blew their competition away without using any alien technology, but just by going “big”. I am just waiting for that company — or department at Canon — to go for it too.

The RED guys have announced that they are going to be releasing a “pocket professional camera” next year, so let’s hope that it’s designed with that in mind — and in the right price.

Update: Good discussion with directors and cameramen about the issue.

I just found out that Quicktime crashing when loaded in a Firefox/browser on the secondary rotated screen is primarily NVidia’s fault. Apparently when you rotate the screen, the driver does not enforce hardware acceleration on that screen. Joost, Flash and games are choppy when played on my secondary rotated screen. The bug exists for at least 2 years, and a freeware utility that supposedly helps setting the right attributes to Windows to use acceleration, it doesn’t. ATi’s driver didn’t have this dual-monitor/rotation bug (it had others).