More Dead Than (CBS Outdoor) Alive
Adrian J Cotterill, Editor-in-Chief
“Death comes alike to the idle man and to him that works much” so said Achilles to Odysseus (Homer, Iliad 9.320) and despite lots of running around (perhaps a little like headless chickens at times) the technology and engineering folks at CBS Outdoor in London seem to have messed up like crazy what should be one of the showpiece digital screen networks on the planet.
We reported yesterday that much of the CBS Outdoor London Underground (Tube) digital screen network had been dark since Sunday AND not surprisingly CBS Outdoor took major issue with this – especially with our use of the word ‘much’.
We had a conference call with a CBS Outdoor spokesperson this morning who wanted to explain some of the issues that we were about to raise (we sent them a DRAFT of this post for review) and correct any inaccuracies. It’s likely however that CBS Outdoor will still be unhappy with what is written here. Remember this is our point of view, our thoughts. We don’t purposely write to cause trouble (though it may indeed look like that a lot of the time).
We had initially heard that almost 400 screens on the Tube network were off, “absolutely not so” say CBS Outdoor who claim a 94% success rate.
CBS Outdoor proceeded to tell us why the screens at Green Park and South Kensington (more on that later) weren’t on. They didn’t however manage to tell us why, that by our reckoning, there were only TWO tube stations that had ALL of their screens (the DEPs) working on the escalators (Baker Street and London Bridge). Every other DEP has an issue with at least one screen on one side. If I was paying for my brand to be on the DEPs I would not be happy with that.
- CBS tell us that all the Green Park screens are off because of London Underground enabling work. This of course does happen from time to time.
- CBS tell us that all the South Kensington screens are off because they are upgrading the system. This makes sense but our point was that one half of the escalator panel didn’t work most of the time previously – hence the need for swopping out and the upgrade.
Here’s the pinch, we can argue about the exact numbers and what is officially on and officially off (or as CBS Outdoor term it ‘Out of Charge’, i.e. is not sold or charged to brands) BUT like many regular London commuters since March 2009 we have seen what can only be described as a massive downturn in the reliability of the digital signage estate.
Walk around the tube and it seems that almost everywhere there is a screen off (for whatever reason) and as previously mentioned walk up an escalator and it’s very unlikely you will see all the screens in action at the same time.
There have been rumours in a lack of technical management and in-house systems knowledge within CBS Outdoor since the departure of several key people in the last 12 months or so and we have seen more than enough systems with our own eyes displaying errors, playing back incoherently or just showing a blank screen.
Rather than a fault with the infrastructure or deployed technologies (hands up here – we are often the first to beat up on Windoze or ‘crappy’ signage systems) that doesn’t seem to be be the issue this time. In fact it would be easy to blame TELentice or Windoze but they are not the culprits – we believe that it is more likely to be human error and lack of process.
CBS Outdoor told us (this is amazing, so hold your breath) that they PAY PEOPLE to go down into the Tube 3 times a day to physically check the screens – in fact they made a big thing about spending GBP 250,000 a year to do this. This is crazy, absolutely crazy!
WIth a properly built and properly managed digital network you can monitor all this remotely. It seems to us that the issues might be with the daisy chaining of the devices (one player goes down on a DEP and it brings down several other screens attached to it), We’d also say that the synchronicity of the DEPs doesn’t seem to work properly either and we expect that is another reason for the need of Eyeball Mark 2.0 tracking!!!
The big problem here is obviously a lack of experienced management at the helm that has seen this once groundbreaking technology reduced to at best an unreliable medium. Long gone it seems is any attention to detail. Again, CBS Outdoor told us “not so” and they vouch for their people 100%. We’d say walk around the Tube and see for yourself.
We believe that CBS Outdoor need to quickly appoint a head of digital (someone who understands all of the intricacies of networks such as these) and stop relying on the IT network folks who obviously don’t have a clue as to how and why the original system was built. Quite literally this once great network seems to have slid from bad to worst since they got rid of a number of engineers who originally built the system.
September 17th, 2009 at 12:42 @571
Somebody please help those poor CPUs at Victoria Tube Station find the boot device. The BIOS message looks so sad.
September 17th, 2009 at 14:01 @626
Anyone who bases a network of signage equipment on an operating system as fragile as Windows deserves all the problems they will (inevitably) get. It’s a home computing platform, designed for versatility and ease of use, and was never intended for 24×7 operation in a commercial environment. You may as well enter Le Mans with a Renault Clio.
Having to spend (waste) 250,000 pounds a year employing people to constantly wander around checking to see which screens have broken down is utter madness. I assume CBS’s advertisers are being paid the compensation they are enitled to for this entirely avoidable fiasco?
September 17th, 2009 at 14:33 @648
What is troubling and upsetting about this is that they are hurting the perception of digital signage for everyone. I can only imagine a pitch happening for another OOH Digital company only to have an ad exec say, “But it seems like the systems don’t work all the time and there are always outages. I think we’ll wait until the technology is better refined.”
September 17th, 2009 at 15:20 @681
Chris, a properly configured Windows XP OS (Embedded or Proffessional) can be very reliable. It’s all to easy to blame Windows for all stablity issues, when in reality it is up to the software/hardware engineers to correctly configure the OS and write stable signage sofware etc…as is the case with any alternative OS or software….
September 17th, 2009 at 16:48 @741
I’ve also noticed the reliability going down. I started counting the number of escalators I saw each day with every screen working in sync and never got above 5 a week. I travel through Bank, Bond St, Baker St and Piccadilly Circus regularly. Way too many BIOS “no boot device found” errors which leads me to believe the problems are not Windows (Windows works absolutely fine on our players) but hard drive errors. Maybe there are ventilation problems down there? It wouldn’t be surprising in that hot dusty environment.
September 17th, 2009 at 17:47 @783
Bryan may have a point in that it’s the hard drive, rather than the operating system. Again, though, that begs the question “why use a hard drive?”. The hard drive and the power supply are always the two weakest links in any computer, which is why the more modern signage systems no longer use hard drives and why high availability systems like those used in telecom applications have dual power supplies. The Digital Signage industry is still pretty primitive when it comes to selecting hardware and software architectures, and as we see more and more systems out there, we see more and more system failures which are nearly all down to poor choice of system.
September 17th, 2009 at 18:34 @815
Don’t always blame the vendor. Some customers demand cheap over reliability. Sure, you can say “Well we won’t install on that crap!” but a project is a project. SAD but true.
September 17th, 2009 at 18:35 @816
“You may as well enter Le Mans with a Renault Clio.”
It would be less thrilling going down the Mulsanne, but at least you’re not going to take off and land in the trees as Mark Webber discovered many years ago.
On a serious note, we’re the distributor of what I believe is the most flexible and capable Macintosh digital signage solution available in the market today and our opinion is that the Windows digital signage market suffers from what we call “the tyranny of choice”. This is a counter-argument to that which Windows devotees use against the Mac platform, namely that it doesn’t have as wide a range of software or peripherals and that you have to play the game by Apple’s rules.
Well, that may be true and a compelling argument when you’re talking about USB joysticks or the latest first-person shoot ’em up game, but it’s less compelling when all you want to do is have your digital signage solution start up in the morning, send a message to its affiliated screens to get out from under the duvet and then start the very serious business of telling people to buy more cornflakes.
At that point, Apple’s model of “you can have any video card you like so long as it’s one we like the look of” begins to look more attractive: I know my solution works with Apple’s video cards (all half a dozen of them, if that) and so long as I hold off any software updates until they’ve been properly tested, I’l be fine.
As for hard drives and power supplies, you’re probably onto something: I have a personal dream that Apple would offer an SSD in a Mac mini as it would save me having to train hardware monkeys to split the thing open with a putty knife.
OK, it’s more expensive in a capex sense, but let’s go ask the guys at CBS Outdoor whether they’d prefer to spend a slightly larger amount of money at the front-end or be going through the commercial/political grief they’re probably suffering at the moment.
September 17th, 2009 at 18:59 @832
Billy B (773) probably hits the nail on the head.
Why do so many LCD screens die in a DOOH environment?
Could it be that people who rarely get the “total cost of ownership” argument (let’s say, beancounters and purchasing types to name two easy targets) get involved and don’t realise that displays that are designed to run in portrait orientation cost more for a reason?
September 17th, 2009 at 18:59 @832
XP machines are completely fine if they are setup properly , well tested, and of a decent quality. It will always be easier/cheaper to find good XP technicians than MAC/Linux ones. Regarding hardware, sure SSD and fanless solutions are nice but still expensive vs the good old platter drive, anyway at the end of the line you do your best with the budget & time you have. Don’t criticize CBS unless you know what was their budget on this project. I also suspect that the quality of the power lines in the subway is quite different than what you can find at your office desk.
However having to spend 250,000 pounds to go just see (not service !) the screens in person is ridiculous with the technologies available on the market.
And yes, it’s bad for the whole industry.
September 17th, 2009 at 23:12 @008
Does anyone know if these systems use Windows XP and hard drives?
Regarding de famous 250k pounds, google says that the Fujitsu solution has a Monitor module: http://www.fujitsu.com/au/services/solutions/telentice/
The article says that the issue seems to be a human error and lack of process and not hardware or software related.
In fact, I believe that this network has been fully operational for several years and is now when the problems has arisen…
September 17th, 2009 at 23:49 @034
“XP machines are completely fine if they are setup properly , well tested, and of a decent quality.”
As opposed to being stable and secure out of the box: our guide to configuration is turn on the firewall, turn off automatic updates and turn off the system’s desire to sleep.
“It will always be easier/cheaper to find good XP technicians than MAC/Linux ones.”
And it’s always so much more necessary to find them.
XP is now over eight years old. If you need a non-security hotfix to solve an issue with XP, you’d better have some decent agreements in place because Microsoft aren’t going to be helping you out unless you have.
Personally I’d rather spend the money on pre-sales people or marketing than creating a social security program for Windows techies and I’m sure the same is sure for my customers, but whatever floats your boat.
BTW, is it also easier/cheaper to find good Vista technicians. How about Windows 7 techies? Just out of interest, let’s pick a mainstream DOOH product like Scala. What’s the support like for Vista?
“anyway at the end of the line you do your best with the budget & time you have”
Agreed. But I’ll still give you dollars to doughnuts that most DOOH project sponsors don’t do due diligence (try saying that quickly five times after a pint of creme de menthe) on the total lifecycle cost of ownership of their proposed networks.
“Don’t criticize CBS unless you know what was their budget on this project.”
Fair enough again. But I’m really not sure that was Adrian Cotterill’s point. I think his point could be summarised as “CBS Outdoor: please deal with reality before it deals with you”. But I could be wrong.