# Feds spend billions to run museum-ready computer systems



## Mark Holtz (Mar 23, 2002)

From Associated Press:

*Feds spend billions to run museum-ready computer systems*


> The government is squandering its technology budget maintaining museum-ready computer systems in critical areas from nuclear weapons to Social Security. They're still using floppy disks at the Pentagon.
> 
> In a report released Wednesday, nonpartisan congressional investigators found that about three-fourths of the $80 billion budget goes to keep aging technology running, and the increasing cost is shortchanging modernization.
> 
> The White House has been pushing to replace workhorse systems that date back more than 50 years in some cases. But the government is expected to spend $7 billion less on modernization in 2017 than in 2010, said the Government Accountability Office.


FULL ARTICLE HERE


----------



## Cholly (Mar 22, 2004)

The crazy thing is that some agencies are still running Windows 3.1 on systems dating back to the late 70's/early 80's. And there are still a lot of systems running Fortran and Cobol software from as much as a half century ago on ancient main frames!


----------



## Drucifer (Feb 12, 2009)

They finally come to the conclusion it will be cheaper to upgrade then to maintain a sloppy disk system.

I wonder if this just our government or most governments on earth.


----------



## texasbrit (Aug 9, 2006)

This is all a bit misleading. The real issue here is not the hardware it's the software. It will cost many (hundreds of) millions to rewrite the software to run on current hardware, essentially they will be brand new systems and many multiyear projects,some of which if experience is anything to go by will be failures. How many bugs are acceptable in software to control nuclear missile launches.....??


----------



## dpeters11 (May 30, 2007)

texasbrit said:


> This is all a bit misleading. The real issue here is not the hardware it's the software. It will cost many (hundreds of) millions to rewrite the software to run on current hardware, essentially they will be brand new systems and many multiyear projects,some of which if experience is anything to go by will be failures. How many bugs are acceptable in software to control nuclear missile launches.....??


It would need to be something like they used for the Space Shuttle. But even that software wasn't perfect.

http://www.fastcompany.com/28121/they-write-right-stuff


----------



## 4HiMarks (Jan 21, 2004)

texasbrit said:


> This is all a bit misleading. The real issue here is not the hardware it's the software. It will cost many (hundreds of) millions to rewrite the software to run on current hardware, essentially they will be brand new systems and many multiyear projects,some of which if experience is anything to go by will be failures. How many bugs are acceptable in software to control nuclear missile launches.....??


It's also other hardware. I used to work for a scientific agency which had lab instruments that cost 100's of $1000's each, which worked perfectly fine, except the control software was written for DOS or Windows 95. In some cases, they would have had to buy a whole new instrument to modernize the controllers. When the agency went to a "shared services" model for help desk in the early days of the GW Bush administration, they then needed to create a whole new "scientific computing" division to maintain them, since Shared Services could only service a standard desktop configuration.

This is another version of the mainframe/terminal - server/client - local/cloud argument: Consolidation vs. Decentralization. Either one can be a cost saver, or a waste of money, depending on the way it is done.


----------



## Cholly (Mar 22, 2004)

Having worked for IBM as a programmer for 25+ years, I worked with several generations of computers, , from special purpose computers succh as missile guidance computers, air defense computers and aircraft computers to civilian computers such as the IBM PC, 7090/7094, 1130, Series/1 and System/360. That encompassed a myriad of programming languages, from computer specific assembly language to higher level languages such as Fortran and PL/1 in addition to good old Basic. There was no ral commonality in those days and a lot of the equipment from that time frame (1960's through 1980's) lives on today, running the same old code.
Fortunately, today, much of the programming is done in versions of the C programming language, predominantly in C++,, as long as there is a compiler available for the computer being used Having been retired for almost 30 yars, I haven't kept abreast of industry trends, but I can well understand the problems that exist in modernization of software and hardware (been there, done that). :grin:


----------



## James Long (Apr 17, 2003)

What are they calling antique? If the system was built on UNIX would it cost less to maintain the systems or more?

There are some systems I would not WANT to operate on Windows 10 or other general purpose operating systems. Constant patches and the potential for downtime to fix problems found in the OS that do not relate to how the OS is being use but must be done to stay current. And updates that add OS features that will never be used.

Not every new operating system is an improvement.


----------



## KyL416 (Nov 11, 2005)

And would you want a modern OS with vulnerabilites running critical areas like nuclear weapons? (Especially if it's a vulnerability that Microsoft doesn't deem urgent enough for an immediate update, so it remains at risk until the next second Tuesday of the month, or in the case of Apple, you remain at risk until they decide to release the next OSX update)

There's also the way things get funded in government, it takes an act of Congress or one of their committees to get approval for most things, you're forced to give it to the lowest bidder, and things completely unrelated to the upgrade process can halt the funding without warning.


----------



## phrelin (Jan 18, 2007)

Let's all keep in mind that the feds spent about $174 million on the HealthCare.gov website which fumbled badly at the outset.

The problem is complex. Here in California we are the home of Intuit, the generally successful maker of general purpose and special purpose accounting systems software. But we don't let that Silicon Valley expertise get in the way of doing things wrong.

in June 2006 we had this headline *California Starts State Payroll System Overhaul* for a story telling us: "California State Controller Steve Westly today announced the start of the long-awaited 21st Century Project to replace the State's 1970s-vintage payroll system."

In December 2012 this headline appeared *Overhaul of California government payroll system at risk of collapse* for a story telling us:



> One of the state's biggest technology endeavors, a $371-million overhaul of the government payroll system, is beset with problems and "in danger of collapsing," according to the state controller's office.
> 
> The company hired for the project is in over its head and may be unable to deliver on its promise to update a payroll system so old that even simple salary adjustments can tie it in knots, the controller's chief administrative officer said in a letter.
> 
> The state has spent at least $254 million so far on contractors, staff salaries, software and more for the system upgrade, which is five years overdue and has nearly tripled in cost since lawmakers authorized it in 2005.


On Valentine's Day 2013 there was this headline *California's Payroll Project Debacle: Another $50 Million Up in Smoke* followed in November 2013 with *California sues SAP over payroll system failures*. In February 2016 we were enthralled to read *California controller seeks $4.8 million to fund litigation over failed payroll system* in which we learned:



> Locked in a finger-pointing lawsuit over a failed state IT project, State Controller Betty Yee has asked lawmakers for $4.8 million to cover court court costs through the end of this year.
> That's on top of $6.8 million spent on litigation during the last six months of 2015. The project, MyCalPays, was suspended in February 2013.
> 
> ...Then-Controller John Chiang axed the project three years ago after an eight-month test run fouled up 1,300 state employees' pay in a variety of ways: wage underpayments and overpayments, employee-insurance and pension deduction errors and missed child-support payments.
> ...


They do have a the new payroll system operating, but this in the state where Silicon Valley is located and one of the world's largest companies in the accounting software business is located in Mountain View and offers up this image:










I can't understand why someone didn't just ask them for help in the beginning ... except I do understand.


----------



## trh (Nov 3, 2007)

Yes, and one of the system we want to upgrade controls nuclear weapons. I'm not sure I trust the system to get the upgrade done properly.


----------



## James Long (Apr 17, 2003)

Nobody wants to see the blue screen of nuclear destruction.


----------



## Wilf (Oct 15, 2008)

James Long said:


> Nobody wants to see the blue screen of nuclear destruction.


----------



## inkahauts (Nov 13, 2006)

This is why IMHO it's better for companies and government to spend more each individual year or two and upgrade their systems yearly or bi annually to something close to one or two steps behind the latest rather than wait for 20 years. The work it requires to upgrade such a giant leap all at once is tremendously higher than what it takes to do consistent incremental upgrades...


----------



## dennisj00 (Sep 27, 2007)

Reminds me of the kerfuffle of activity in the late 1990s to 'upgrade' systems to handle the year 2000+.

Some 'professionals' actually thought elevators would fail or planes would fall out of the sky because it couldn't handle y2K.


----------



## Cholly (Mar 22, 2004)

trh said:


> Yes, and one of the system we want to upgrade controls nuclear weapons. I'm not sure I trust the system to get the upgrade done properly.


I'd taken early retirement from IBM's Federal Systems division and was rehired as a supplemental employee for a year and a half because no one in my former area had my skills set as a Series/1 Programmer. Within a few months afte the end of my supplemental employment, I received a phone call asking me to come back again to work on updating the software for the Series/1 computers in Minuteman silos. I declined.


----------



## djlong (Jul 8, 2002)

dennisj00 said:


> Reminds me of the kerfuffle of activity in the late 1990s to 'upgrade' systems to handle the year 2000+.
> 
> Some 'professionals' actually thought elevators would fail or planes would fall out of the sky because it couldn't handle y2K.


Hey, F-35s nearly fell out of the sky because they couldn't handle crossing the International Date Line.

I remember doing my first Y2K project. *In 1994* - the hospital where I worked was rather forward thinking.


----------



## James Long (Apr 17, 2003)

I remember standing watch on Friday December 31st 1999 with my then employer present because of his fears that everything would go dead at midnight plus one second. A year later the millennium started with the business completely empty until normal working hours.

(And by "everything go dead" I mean massive power failures, planes falling from the sky, etc. He was a die hard Y2K fearmonger who was not persuaded until the moment actually passed.)


----------



## scooper (Apr 22, 2002)

I spent Dec 31, 1999 at a James Taylor concert where I wouldn't be surprised if half the audience were techies with pagers and cell phones waiting for the call that never came...


----------



## dpeters11 (May 30, 2007)

dennisj00 said:


> Reminds me of the kerfuffle of activity in the late 1990s to 'upgrade' systems to handle the year 2000+.
> 
> Some 'professionals' actually thought elevators would fail or planes would fall out of the sky because it couldn't handle y2K.


Certainly things like that were just fearmongering. But I've heard others go too far the other way that all the work was for nothing as nothing at all happened. We would have had issues if it was ignored, just not to the level of planes crashing.

But then recently I read of an incident of a piece of equipment failing during a heart procedure because it was scheduled to do an AV scan.


----------



## phrelin (Jan 18, 2007)

Here's another story this week, this time about the California Department of Veterans Affairs, *Audit: Vets agency wastes $28 million on failed computer system*.



> The California Department of Veterans Affairs has spent nearly $28 million on a system that launched years later than planned, wastes staff time and has not been fully implemented, according to an audit released Thursday by state Auditor Elaine Howle.
> 
> The audit marks the latest in a long string of California government technology failures. The auditor previously found data security weaknesses and unsatisfactory oversight on technology projects. Additionally, a payroll system update spiraled into chaos, licensing board software was delayed, and a tax and fee system stalled.
> 
> Howle's latest audit found the Department of Veterans Affairs started with a plan to implement a comprehensive computer system so veterans who receive rehabilitative, residential and medical services would get "consistent and integrated care" no matter which facility they visited throughout the state. The idea was approved in 2006.


A PDF of the audit report can be *downloaded here*. The newspaper story does not report that the audit also laid blame on the California Department of Technology. It's really quite depressing.

I know that somewhere out there in Silicon Valley is someone with both the systems analytical skills and an awareness of systems being put into place in the senior care home business who probably could have recommended and implemented a system that would work for these veterans homes for $2.8 million plus hardware costs. But they wouldn't be willing to go through the governmental RFP/contracting processes developed for acquiring vegetables for those same homes. :nono2:


----------



## P Smith (Jul 25, 2002)

Cholly said:


> I'd taken early retirement from IBM's Federal Systems division and was rehired as a supplemental employee for a year and a half because no one in my former area had my skills set as a Series/1 Programmer. Within a few months afte the end of my supplemental employment, I received a phone call asking me to come back again to work on updating the software for the Series/1 computers in Minuteman silos. I declined.


When I decide to post here, I thought to explain exactly same point.

There are no more SW developers who could create working system for System/1 or OS/360 ! Especially whole World's life critical as nuclear missiles control !!

Look at that source code what the coders making now


----------



## dennisj00 (Sep 27, 2007)

They could easily replace the 8" floppy drives with a raspberry pi / usb stick floppy emulation for about $100.00 per drive. ($1 million or more per drive in Gov. terms.)

But that poses a big security threat. Much easier to hide a USB stick than an 8" floppy.


----------



## Drucifer (Feb 12, 2009)

phrelin said:


> Here's another story this week, this time about the California Department of Veterans Affairs, *Audit: Vets agency wastes $28 million on failed computer system*.
> 
> A PDF of the audit report can be *downloaded here*. The newspaper story does not report that the audit also laid blame on the California Department of Technology. It's really quite depressing.
> 
> I know that somewhere out there in Silicon Valley is someone with both the systems analytical skills and an awareness of systems being put into place in the senior care home business who probably could have recommended and implemented a system that would work for these veterans homes for $2.8 million plus hardware costs. But they wouldn't be willing to go through the *governmental RFP/contracting* processes developed for acquiring vegetables for those same homes. :nono2:


Yep, the government in trying to protect itself is turning off businesses that can do the job to those that are only out to make money.


----------



## inkahauts (Nov 13, 2006)

I will say this. A new system would be difficult to implummet that was truly good no matter who did it.


----------



## P Smith (Jul 25, 2002)

inkahauts said:


> I will say this. A new system would be difficult to implummet that was truly good no matter who did it.


if that's mean people back then (in 60s) was more forgiven to bell and whistles and accepting minimal implementations (eg DOS prompt on a b/w non-graphic console), focused just on functionality, reliability ..I'm with you.


----------



## Cholly (Mar 22, 2004)

More often than not, new software is developed based on a best guess scenario. Hardware is developed an lays dormant because specifications are ill written,necessitating total redesign of contrrol programs. One wonders about the POS chip readers that are yet to be activated in countless stores across the country. Even those that are in use tend to have occasional hiccups.


----------



## James Long (Apr 17, 2003)

P Smith said:


> if that's mean people back then (in 60s) was more forgiven to bell and whistles and accepting minimal implementations (eg DOS prompt on a b/w non-graphic console), focused just on functionality, reliability ..I'm with you.


People were easily impressed in the 60s. No need for bells and whistles, just some functionality. They didn't have the processor speeds to add much more than making it work. And that was all the users expected. Programs did not even have to work well, they just had to not have too many failures.

Today it is all about the GUI. Text layout on a console is rejected (whether it is green on black, amber on black or white on black - black on white and full color being more modern schemes). Font? Isn't normal and bold enough font choices?

From a practical standpoint the 60s code only had to run on the machine (and connected terminals). Today's code has to run on an operating system. Every program is something based ... I have several at work that are Linux based but most require Windows based machines as servers and clients. (The Linux machines normally having web interfaces that work with Firefox or IE.)

Today's processors are much faster but we have managed to bog them down with operating systems and bells and whistles.


----------



## Cholly (Mar 22, 2004)

James Long said:


> From a practical standpoint the 60s code only had to run on the machine (and connected terminals). Today's code has to run on an operating system. Every program is something based ... I have several at work that are Linux based but most require Windows based machines as servers and clients. (The Linux machines normally having web interfaces that work with Firefox or IE.)
> 
> Today's processors are much faster but we have managed to bog them down with operating systems and bells and whistles.


True enough. The programs I wrote during the 60's ran standalone for the most prat, although OS/360 had come upon the scene.Programs I wrote for the IBM 7090/7094 werestandalone, as I recall -- Written either in Fortran or Fap, compiled and saved on mag tape. MSystem/360 programs ran under the control of the OS, as I recall (My memory there tends to be foggy at best).


----------



## phrelin (Jan 18, 2007)

inkahauts said:


> I will say this. A new system would be difficult to implummet that was truly good no matter who did it.


Yes, but there are at least a half dozen well-designed, comprehensive, tested multi-facility "senior care home" software packages already in use nationally that can be purchased sort of "off the shelf." Some are fully integrated and some integrate established medical systems. While I'm sure I could figure it out, I would struggle to find ways to spend $28 million of the State's money implementing them.


----------



## 4HiMarks (Jan 21, 2004)

James Long said:


> Today it is all about the GUI. Text layout on a console is rejected (whether it is green on black, amber on black or white on black - black on white and full color being more modern schemes). Font? Isn't normal and bold enough font choices?


True enough, and half the time there is no text at all, now. Nothing but nameless icons you have to guess at the function of. On a phone, there isn't even a mouseover tooltip to help you. Or if there IS text, it is so teeny tiny that middle-aged eyes can't read it.


----------

