Caliban – Opinion and Righteous Anger

Another day, another MythTV experience.

I spent most of yesterday afternoon and evening coding my new TV programme data grabber. This one grabs programme timetable information from Chello, which is the name of UPC‘s cable TV (and cable Internet) offering.

The standard XMLTV grabber for The Netherlands isn’t very good, so I quickly replaced it with a different one, written in Python. This one is much better and is kept nicely up to date by its author.

Anyway, even that one has a major shortcoming, namely that it doesn’t know anything about many of the channels in UPC’s digital cable package. There are many cable companies operating all over the country and each one offers a different selection of channels, so finding a good data source for all of them is difficult.

So, after several hours of Ruby coding, utilising the excellent REXML library to produce XMLTV format data, I had a tool I could use to populate my MythTV database with programme information for the missing channels. As a bonus, it even grabs channel icons for use in the on-screen display and programme guide.

I started by adding data for just BBC 3, BBC4, The Travel Channel and a few others. Then, I checked MythWeb in my browser and, sure enough, The Travel Channel was now displaying a lot of Globe Trekker episodes. I quickly created a rule to record all showings of this, one of our favourite programmes, on any channel that happens to be broadcasting it.

Later in the evening, I added data for a few more channels, such as Discovery Travel & Living. The MythTV system immediately applied the Globe Trekker rule I created earlier in the evening to ensure that the new episodes on this channel will also be recorded.

Confident that the new grabber is working properly, I have now added its invocation to cron and look forward to using it to exploit the full power of our shiny new PVR system.

When I’ve tidied up the code a little, I’ll make this script available for anyone who cares to use it.

mythfilldatabase is the program that actually takes all of the XMLTV data and pumps it into the MythTV database. The grabbers that this program will accept data from are hardcoded into the binary. Since I didn’t want to patch the source and recompile, I decided to add the programme data using a temporary file. A wrapper script takes care of the whole process:

xml_file=$( mktemp -p ~/.xmltv upc.XXXXXXXXXX )

video_source=2

tv_grab_nl_upc > $xml_file

mythfilldatabase –file $video_source -1 $xml_file

rm $xml_file

And Bob’s your father’s brother!

If anyone needs tv_grab_nl_upc and can’t wait for it, drop me a line and I’ll send it to you.

Another week has gone by and a few more hours of sleep have been grabbed at various intervals.

In the intervening time, work has continued on the interminable odyssey that is the MythTV box.

The major breakthrough came earlier this week when I finally got the IR blaster to talk to UPC‘s digital cable decoder. Unfortunately, it’s still a bit hit and miss, because the IR blaster‘s signals sometimes go astray.

For example, Amsterdam’s local station, AT5 is on channel 714 of UPC’s digital cable, so the blaster sends the digits 7, 1 and 4 with a 1.3 second interval between them. Sometimes (and the percentage is way higher than I would like), one of these digits gets lost somewhere, so the set-top box might get tuned to channel 71, 74, 14 or even 7, 1 or 4 if two digits happen to be lost.

When changing to a single digit channel, sometimes that solitary digit gets lost, so the channel doesn’t get changed at all. In the end, though, it doesn’t matter. Whether the channel doesn’t get changed at all or it does get changed, but to the wrong channel, you ultimately don’t get the recording you wanted, and that’s all that matters.

That’s not the only problem with the channels from the digital cable package, however.

The Dutch TV schedule grabber pulls its data from TVgids.nl, which works quite well for the common channels available in the analogue package. Unfortunately, though, TVgids.nl doesn’t carry programme schedule information for the less common channels, such as BBC3, BBC4 and The Travel Channel. Wah!

So, my next project will be to write a new grabber in Ruby that pulls programme data for these and the other digital-only channels from the most logical source, UPC’s own Chello site. This means scraping a lot of HTML and outputting XMLTV format files. I’ll probably start writing this in Ruby some time in the next few days.

The playback freezes are still with us, but they’re infrequent, so fixing them is a low priority. Skipping back five seconds is sufficient to restart playback, anyway.

I’m also working with the mod_mce author to fix the system freezes caused when trying to use a wireless Microsoft MCE keyboard. No luck yet, but we’re slowly localising the cause of the problem.

This week also saw the purchase of some more hardware. First of all, I bought a Logitech Harmony 885 universal remote control. It’s still in the packaging, however, as I haven’t had a chance to set it up. I’ll also need to borrow Sarah’s Windows laptop for that.

Secondly, I bought a Logitech Z-5450 5.1 speaker system. It’s not high-end audio by any stretch of the imagination, but it’s better than the TV speakers and connects directly to the MythTV box’s soundcard, which enables us to make full use of the MythMusic module to stream our OGG and MP3 filles from the server in the cellar. It’s great to have the ability to play music in the living-room again.

The Z-5450 also allows us to digitally connect our DVD player and set-top box, so when we want to play a disc or watch digital cable straight from the source, we get nice crisp audio to go with it.

Lastly, I bought another PVR-350 TV capture card, along with a dual tuner PVR-500 card.

The PVR-350 was to replace one of the two already in the box, which was suffering from diagonal, wavy lines in the picture. These were annoying me more and more, so I decided to replace the card.

Unfortuantely, the new PVR-350 had a newer, different tuner chip on it, which didn’t immediately work. When I finally found out how to get it to work, the new card turned out to suffer more from the wavy lines than the one it was supposed to replace.

As a result, I returned that PVR-350 today for another one, and purchased yet another one on top, in the hope of increasing my chance of bringing home a good one. Thankfully, one of them gave a perfect picture, so that one is staying and the other two are going back tomorrow.

As for the PVR-500, this also turned out to have newer tuner chips than older renditions PVR-500. Instead of Philips tuner chips, these have a Samsung tuner. A kernel patch was required to get this card to fire up properly, but once that was applied, it worked like a charm… and thankfully without any wavy lines!

So, the MythTV box is now a quadruple tuner affair, three of which receive analogue coax inputs, the remaining one getting its signal over S-Video from UPC’s digital set-top box. That means we can now record up to four simultaneous broadcasts whilst watching a previously recorded programme. Is that cool or what? Initial tests show that the 500 Gb SATA drive takes all of this in its stride.

As you can surmise from all of this, it’s been a week of devouring documentation, becoming extremely frustrated with poor quality hardware whose components seem to change on an almost daily basis, software experimentation, and trailing around the computer shops of Amsterdam, most of whom probably hate me by now, as I am constantly purchasing and returning TV capture cards.

PC hardware is utter shit, it has to be said. If my experiences earlier this year and in the last week are representative of the overall state of the industry, then a good third of all PC hardware sold is either substandard or just plain broken. And that’s before you figure in the frustration of building in hardware that Linux supposedly supports, only to discover that the manufacturer has changed the specs of the device without giving any outward indication of this on the packaging. The net result: hardware that doesn’t work and lots of wasted time and energy spent troubleshooting and shuttling back and forth by bike between home and the computer shops.

It’s been worth it, though. Our MythTV box is now very powerful and contains TV capture cards that actually provide a decent picture.

Once I get to grips with adding programme schedule information for the missing digital channels to the database, we’ll be able to schedule recordings from any channel, no matter how obscure. At that point, this box will be approaching its maximum utility; it’s already way beyond the functionality of any of the dismal commercial PVR appliances available in this country.

Picture quality on the MythTV is now a lot better. I had been ignoring the transcoder configuration options of the front-end, because my tuner cards, PVR-350 cards, do the MPEG2 encoding in hardware. However, it tuns out that the transcoder options aren’t just for software encoding; on the PVR-350, they also control how the firmware is used for hardware encoding.

In short, we’d been encoding the analogue TV signal with a resolution of 480×480 and a 4:3 aspect ratio. Once I had changed this to 720×576, which is the standard PAL resolution, and used a 16:9 aspect ratio for our widescreen LCD TV, things looked a lot better. Things could still be better, of course, but the quality is now quite satisfactory.

I also changed the encoding algorithm to DVD Special 2, which is apparently the one that Microsoft MCE uses, so in theory the PVR-350’s firmware is optimised for this method. All in all, I think the picture quality is now as good as I’m going to be able to make it.

Unfortunately, I’m still having no luck getting the system to change the channel on the set-top box. I posted an appeal on a site devoted to discussing UPC’s products. That yielded an lircd.conf for the Motorola DPI4000 set-top box, which apparently uses the same IR codes as the Thomson DCI-52UPC that I have. Unfortunately, it didn’t work for me, so the search for a solution goes on.

I also bought a Microsoft MCE IR keyboard a couple of days ago. Linux support for this is still in its infancy, but support for it theoretically does exist in the form of the mod_mce kernel module.

Well, I can get it compiled and inserted into my running kernel, but at the first press of a key on either the keyboard or the remote-control, the entire system freezes. It’s not pingable and the TV picture freezes; even the blinking dots separating the hours from the minutes on the VFD display cease to blink. Whatever that module is doing, it’s not good. So now I have an extra problem to solve; otherwise I’ve wasted €90 on a new wireless keyboard (and it really is a great-looking keyboard).

All in all, we’re getting there. We cleaned up the database to remove duplicate channel listings in MythWeb’s programme guide. My SQL fell short on this task, so Sarah put together a complicated join for me. She’s still a dab-hand at the old SQL.

Speaking of MythWeb, I’ve opened up a port on the firewall to forward traffic from the Internet to the Web back-end. This will be handy when we’re away, because we’ll be able to schedule the recording of new programmes from wherever we happen to be in the world. I was always envious of my TiVo-hacking friends in the States who had this. Naturally, all traffic from outside the home network needs to be authenticated, so digest authentication in Apache 2.2 is taking care of that. I had to figure out how to authenticate outside traffic whilst allowing inside traffic straight through without a password, but that took just a few minutes, thanks to Google.

And that’s where we are right now. We still can’t use the MythTV box for digital cable recordings and we’re still suffering from irritating playback freezes. If I can fix those two problems, plus get the new IR keyboard working, I’ll be a very happy man, indeed.

The box is already providing us with a lot of TV viewing pleasure, however. It’s immeasurably better than the calamitously bad Philips DVDR7300H that we had been using up until last week, although we’ll keep that around for its HDMI DVD-playing capability. In the absence of TiVo, building this box was really the only choice, unless I wanted to buy or build an MCE-based box, but the idea of an inflexible, closed source Windows box was obviously repugnant.

Finally, I must comment on how great it has felt to once again have a technical project to immerse myself in. Choosing the hardware was tedious and putting it all together was an experience frustrating beyond belief (even causing me to shelve the project for a good six months), but once the operating system was installed and I could begin work on the actual MythTV installation and configuration, I was in my element again.

Again, once I iron out the last few serious wrinkles, I’ll be feeling quite satisfied.

I’ve made a lot of progress on the MythTV box on the last few days.

Firstly, I didn’t trust the SCART to S-Video/2 x RCA cable that I’d bought, suspecting that it may be wired to have SCART be the input, not the output. So, I went out and bought a switchable input/output SCART connector, along with an S-Video cable and a double RCA jack lead. Using this, I was able to conjur up a black-and-white image from our digital set-top box on /dev/video1, our second PVR-350 TV card.

Interestingly, the ivtv driver needed its input set to 2, not 1 (as documented), to produce colour S-Video output. Perhaps 1 is for NTSC, rather than PAL, but who knows? Most of this stuff assumes NTSC and other American details, so I’ve often found myself having to go through a few extra hoops. Anyway, the cable box now gives an image on the MythTV box, so that’s progress.

For the last couple of days, I’ve been beating my head against the wall, trying to get to the next stage, which is to have the MythTV box be able to change the channel on the digital set-top box. Getting this to work is a nightmare.

The basic description of what is needed for this is simple: connect an IR blaster to the MythTV box and have it act like an oversized remote-control to the set-top box. The reality, as is so often the case with Linux, is that there’s a lot more to it than this.

The IR blaster was purchased months ago from a bloke in the US, using PayPal. It connects to the RS232 serial port.

Now, the IR blaster needs to be driven using lirc, but an instance of that is already running to handle input from the PVR-350’s remote-control. lirc isn’t teriibly sophisticated, so multiple instances of the daemon, lircd, would conflict with each other. So, I had to produce a modified build of lirc that would handle talking to the set-top box.

Ha! Easier said than done. Kernel sources were required for this, so I downloaded the Fedora source RPM of those and dropped them on the box. However, compiling lirc against these gave module address mapping conflicts when I tried to insert lirc’s modules into the running kernel. In the end, I had to partially build a Fedora kernel to reproduce the module map that Fedora was actually using and thus expecting.

That was more work than it sounds, because I don’t use custom kernels any more (I just don’t need them any more and compiling them for the sake of it is something I’m too old, fat and lazy for these days).

Anyway, finally I had a second instance of lirc that would use different device nodes than the original. Next, I set about testing the software, which eventually worked and appeared able to drive the IR blaster.

The next problem was that there is no existing lirc remote-control config for the Thomson DCI-52UPC, which is the piece-of-shit set-top box provided by that wayward gang of goons known as UPC. This meant I needed to create one.

Creating such a config entails using the device’s remote-control to teach lirc abput the required IR codes to drive the unit. Sounds easy, I know.

Well, the first problem was that my PVR-350’s IR unit expects only its accompanying remote, which uses the RC5 protocol. UPC’s set-top remote-control clearly doesn’t, so I can’t use the PVR-350 as a receiver for learning the codes.

Now, the VFD display on the Origen AE case that houses the MythTV box also happens to have an IR unit, so I downloaded some software from IRTrans to make that work. Rather surprisingly, I was able to get this IR unit to the stage that it would recognise keypresses from the DCI-52UPC’s remote-control, which was pleasing.

Unfortunately, the software used to talk to this IR unit is not lirc, so although it was listening to the keypresses and recording them, it was doing so using a format that lirc can’t use to relay to the set-top box.

So, I went out and purchased another IR receiver this afternoon, a Microsoft MCE unit. Although it’s sold as a Microsoft unit, the kernel reports it as a Philips eHome Infrared Transceiver. It’s a USB device and was recognised as soon as I plugged it in.

Within a few minutes, the system was configured to start recording keystrokes from the UPC remote-control, using lirc’s irrecord utility. A few minutes after that, I had an lircd.conf file containing raw IR codes for the set-top box. Optimistically, I thought I was now in the home straight.

Well, I fired up the new instance of lircd and told it to send some numeric keystrokes to the UPC box. The IR blaster’s visible LED flickered red with each keystroke, but the UPC box just sat there, ignoring whatever the blaster was sending.

I’ve spent the evening reading about lirc and infrared protocols, but I have to admit to being stumped at this stage of the game. I don’t know what to try next at this point. How hard can it be to send IR signals over a small transmitter and have those interpreted by a set-top box? Too hard for me, clearly.

But it’s not all doom and gloom. Yesterday, I finally figured out how to make the VFD display on the case work. It connects internally over USB, but it needs to be driven as a serial device, so first of all you need the ftdi_sio kernel module to do the conversion.

Then, you run irserver, which can be downloaded from IRTrans. It’s a low level daemon that accepts commands over TCP and passes them on to the VFD unit.

MythTV, on the other hand, knows nothing of such things and expects to be able to use LCDproc as its API. Happily, the people at IRTrans have a patched version of LCDproc available for download. Once that is installed, you can fire up LCDd, which then talks to irserver, which, with a little bit of help from the ftdi_sio module, drives the VFD unit.

Simple, as only Linux can be. Anyway, the result is nice: channel and programme information is now displayed on the case’s VFD unit, which lends the box a slick and professional look.

I’ve also replaced XMLTV’s tv_grab_nl with a Python version I found on the Web. This one is much better, as it grabs channel icons and reliably downloads programme descriptions and genre information. The old one wasn’t

robust and couldn’t deal with the contradictory data supplied by tvgids.nl (e.g. overlapping start and finish times).

Finally, I’ve put in a cable amplfier/signal splitter unit, so that we can have more inputs to the MythTV box. The idea is to add a third PVR-350 tuner card at a later date. At that point, all of the machine’s PCI slots will be in use.

There are sundry other niggling little problems with the box, ranging from playback freezes (which can be remedied with a push of the skip button) to kernel oopses. All in all, the system has fantastic potential and is an endless source of hacking pleasure and frustration, but it’s definitely not production quality material at the moment.

I really do have to get the bloody thing to change the channel on the digital set-top box, or it will have limited utility. If I can just get that part to work, I’ll be most of the way there and can then sit back, breathe out and relax a little again.

The whole experience gives me a huge amount of respect for TiVo. How they can put together a reliable box for $200 is beyond me.