Marin TV Programming Issues

March update - Our tired server returned refreshed after a brief vacation in the Midwest and we upgraded software across the board which of course caused new minor conflicts we are still sorting out - but your Marin TV is back to normal.

- original post -

You may have noticed some programming disruptions this week on Marin TV Channels 26 and 30 - please be aware that we're working on it! We hope to have everything back to normal next week. That's the short explanation - for the long one, read on . . .

Community Ch 26: Submitted Series and Specials are now playing out as scheduled without interruption.
Government Ch 27: Programming has been unaffected.
Education Ch 30: Regular programming has been temporarily shifted to UCTV and NASA TV programming.

On Tuesday, Feb 24, all our channels went black around 10am. This had never happened. We quickly investigated and determined the master control router had gone on the blink. This is the switcher that controls the routing of all the audio and video signals that pass through to Comcast, AT&T and the web - imagine a box with about 100 cables spilling out the back. It's controlled by our scheduler server which is programmed every day to select between the various signals at specific times. Anyway, a quick restart and the router was seemingly back to normal . . . for about 15 minutes until it went down again. Some quick diagnostics and troubleshooting revealed a bad power supply was the likely culprit which precipitated a desperate call to the manufacturer for a replacement. As Megan and I scratched our heads deciding which cables to reroute to bypass the problem so we could have a signal on air, we heard a loud pop followed by the very troubling smell of something electronic that was cooked beyond well done - this was definitely not going to be a good morning. Some poking around (and targeted smelling) revealed that our playback server for 26/30 also decided to blow a power supply, and it was only 11am.

We mulled over some options, including just leaving town, but instead we decided to manually reroute the signals to 26/30 from our bulletin system to at least keep a signal on air (there's some obscure FCC issue about that). Parts were quickly ordered for the playout server for good measure. All the parts arrived express the next day and were carefully inserted into the disassembled equipment (yes - all those cables had to come off). By noon, regularly scheduled programming was back on the channels. Not bad, a complete catastrophe averted in 24 hours, we were feeling pretty good . . for a couple hours anyway, until it became apparent that the playout server still wasn't very happy. It was no longer talking to the scheduling server which tells it what to play and when, and worse, if we restarted it, it would freeze on the Microsoft startup screen (probably the biggest insult of all).

It was time for the paid techies to earn their keep, our trustworthy integrator and the manufacturer techs. They quickly went to work from afar to remotely figure out what was happening. This part is where Wednesday runs into Thursday into Friday and they eventually concede defeat and tell us to just box it up and send it to them. I never care for that outcome, when boxed this heavy thing barely fits in my car and the shipping is over $400. Until this time, the server mostly played programs as scheduled or it was sitting frozen on that microsoft logo doing nothing, hence the unpredictable cable schedule of the last several days. The determination was that a yet to be identified hardware failure (likely from the spontaneous BBQ event) was the culprit and not the software. So it is now off to the manufacturer who has all the parts to build it anew. And that's where we are as of Friday.

Megan has since moved the community programming and scheduling over to our government play out system which will carry on until we get the other server back. This won't include the fill programming used to fill the gaps, but we will have the satellite feeds and their recordings back on track (Democracy now, etc). The Edu channel will be the most affected in the coming week - we've diverted that entirely to UCTV (University of California) programming 24/7 until the server returns - but that is great programming to watch too. You may also see some live government meetings there in the evenings next week - we were planning to start carrying more live meetings and this is currently the only channel that can accommodate that as the government channel is too full.

Many don't realize that government services and programming accounts for nearly half the activities of our staff now. Folks come to the center and see all the community folk, courses and other activities and think that is what CMCM does - not even close, maybe 1/3. We currently carry the meetings for 7 of the 11 municipalities/County of the MTA on the government channel and in most of those cases we installed and operate the equipment in those city chambers as well. The channel also runs the meetings of several JPAs and other agencies that we cover too. On the education front, we've worked closely with many schools and programs to develop programming, run a summer sports broadcast camp and are on the cusp of launching a year-round youth media program. So, there's always a lot of exciting things happening at CMCM, some you don't see and some you don't want to see - like the technical difficulties of this week.

Thanks for your patience and we'll check in again next week when everything is all back to normal!