— Domani Spero
[twitter-follow screen_name=’Diplopundit’ ]

The first announcement about the troubled Consular Consolidated Database (CCD) went out on Wednesday, July 23:

The Department of State Bureau of Consular Affairs is currently experiencing technical problems with our passport/visa system.  This issue is worldwide and is not specific to any particular country, citizenship document, or visa category.  We apologize to applicants who are experiencing delays or are unable to obtain a passport, Consular Report of Birth Abroad, or visa at this time. We are working urgently to correct the problem and expect our system to be fully operational again soon.

The AP reported on July 23 that unspecified glitches have resulted in performance issues since Saturday, which would be July 19.

On July 25, CA announced:” Our visa and passport processing systems are now operational, however they are working at limited capacity. We are still working to correct the problem and expect to be fully operational soon.”

A State Department official speaking on background told us the same day that this issue was not/not caused by  hackers. We were told that the CCD crashed shortly after maintenance was performed and that the root cause of the problem is not yet known.

On July 27, CA released an update:

As of July 27, the Department of State has made continued progress on restoring our system to full functionality. As we restore our ability to print visas, we are prioritizing immigrant cases, including adoptions visas. System engineers are performing maintenance to address the problems we encountered. As system performance improves, we will continue to process visas at U.S. Embassies and Consulates worldwide. We are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.

On July 29, CA posted this on FB:

The Department of State Bureau of Consular Affairs continues to make progress restoring our nonimmigrant visa system to full functionality. Over the weekend, the Department of State implemented system changes aimed at optimizing performance and addressing the challenges we have faced. We are now testing our system capacity to ensure stability. Processing of immigrant visas cases, including adoptions, remains a high priority. Some Embassies and Consulates may temporarily limit or reschedule nonimmigrant visa interview appointments until more system resources become available to process these new applications. We sincerely regret the inconvenience to travelers, and are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.

The CA Bureau’s Facebook page has been inundated with comments. There were complaints that at one post the visas were printing fine and then they were not. There were complains from people waiting for visas for adopted kids, for fiancees, for family members, for family waiting at the border, for students anxious to get to their schools, people worried about time running out for diversity visas, applicants with flights already booked, and many more.  One FB commenter writes, “I feel that the problem most people have is not that the system broke, but the lack of clear, meaningful information so people can make appropriate plans.”

Other than what the CA Bureau chose to tell us, we cannot pry any substantial detail from official sources.  We, however, understand from sources familiar with the system but not authorized to speak for the bureau that the CCD has been having problems for sometime but it got worse in the last couple weeks.   If you’re familiar with the highs and lows of visa operation, this will not be altogether surprising.  Whatever problems already existed in the system prior to this “glitch” could have easily been exacerbated in July, which is the middle of the peak travel season worldwide. A source working in one of our consular posts confirmed to us that the system is back running, but not at the normal level and that the backlogs are building up. Another source told us that Beijing already had a 15k NIV backlog over the weekend.  We haven’t yet heard what are the backlogs like in mega visa-issuing posts like Brazil, Mexico and India.

We understand that everyone is currently doing all they can to get the process moving, but that some cases are getting through the system, while some are not. No one seems to know why this is happening. These machine readable visas are tied to the system and there are no manual back-ups for processing these cases (more of that below).

So who owns CCD?

The Consular Systems and Technology (CA/CST) manages the CCD.  We have previously blogged about its troubled past:

• OIG Review of Visa Lottery Snafu Details Failure of the Consular Bureau’s CST Office
• State Dept CA CIO Kirit Amin Rants on FSOs, Cronyism, Nepotism, “Doormats”… Two Days Later, Apologizes
• Inadequate Oversight of State Dept’s Consolidated Consular Database– Is This WikiLeaks Waiting to Happen, Again?

CST is currently headed by a new Director, Greg D Ambrose who reports to the CA Bureau’s Assistant Secretary.  It looks like despite the 2011 OIG recommendation, the CST deputy position remains vacant. We should also note that the  Asst Secretary for Consular Affairs Janice Jacobs retired this past April.  No replacement has been nominated to-date and Michele T. Bond has been Acting Assistant Secretary since Ms. Jacobs’ departure.

Last September, Mr. Ambrose was with FedScoopTV and talked about Consular One, the future of consular IT.

CST Just Got a New Data Engineering Contract

In Many 2014, ActioNet, Inc., headquartered in Vienna, Virginia,announced a 5-year task order for data engineering, supporting CST.

ActioNet, Inc. announced today the award of a five (5)-year task order entitled Data Engineering (DE) in support of Department of State (DOS). This task order will provide data engineering and database infrastructure support services necessary for planning, analysis, design, and implementation services for the Bureau of Consular Affairs.  These service also include contract and program management support to ensure that innovation, efficiency, and cost control practices are built into the program. […] The Office of Consular Systems and Technology (CST) within the Bureau develops, deploys and maintains the unclassified and classified IT infrastructures that help execute these missions. The Bureau currently manages over 800 servers worldwide, in order to comply with the fast paced changes inherent to data processing and telecommunications, CST requires that contractor services provide for rapid provisioning of highly experienced and trained individuals with the IT (information technology) backgrounds and the security clearances required of CA’s environment of workstation-based local and wide-area network infrastructures.

Due to limited information available, we don’t know if the new Consular One and/or the new DE contract are related to ongoing issues or if there are hardware issues, given the multiple legacy systems, but we do know that CST has both an impressive and troubled history. Let’s take a look.

Records Growing by the Day

The 2010 Consular Consolidated Database (CCD) Privacy Impact Assessment (PIA) describes (pdf) the CCD as “one of the largest Oracle based data warehouses in the world that holds current and archived data from the Consular Affairs (CA) domestic and post databases around the world.”  According to the PIA, in December 2009, the CCD contained over 100 million visa cases and 75 million photographs, utilizing billions of rows of data, and has a current growth rate of approximately 35 thousand visa cases every day. The 2011 OIG report says that in 2010, the CCD contained over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day.

That was almost four years ago.

A Critical Operational and National Security Database with No Back-Up System?

According to publicly available information, the CCD’s chief functions are 1) to support data delivery to approved applications via industry-standard Web Service queries, 2) provide users with easy-to-use data entry interfaces to CCD, and 3) allow emergency recovery of post databases.  The CCD also serves as a gateway to IDENT and IAFIS fingerprint checking databases, the Department of State Facial Recognition system, and the NameCheck system. It  provides access to passport data in Travel Document Issuance System (TDIS), Passport Lookout Tracking System (PLOTS), and Passport Information Electronic Records System (PIERS).  The OIG says that the CCD serves 11,000 users in the Department and more than 19,000 users in other agencies, primarily the Department of Homeland Security (DHS) and various law enforcement elements, and is accessed more than 120 million times every month.

Given that the CCD is considered “a critical operational and national security database,” there is surprisingly no redundancies or any back-up system.

Resurrect the Standard Register protectograph aka: `Burroughs visas’?

No one is actually suggesting that but when the CCD system is down, there is no manual way to issue a visa. No post can  handprint visas  because security measures prevent consular officers from printing a visa unless it is approved through the database system. Here is a quick history of the handprinted ‘Burroughs visas’ and the machine readable visas via the GPO:

November 18, 1988, mandated the development of a machine-readable travel and identity document to improve border entry and departure control using an automated data-capture system. As a result, the Department developed the Machine Readable Visa, a durable, long-lasting adhesive foil made out of Teslin.

Before MRVs, nonimmigrant visas were issued using a device called a Standard Register protectograph, otherwise known as a Burroughs certifier machine. It produced what was colloquially known as a “Burroughs visa,” an indelible ink impression mechanically stamped directly onto a page in the alien’s passport. Over time, Burroughs machines were gradually replaced by MRV technology, which is now used exclusively by all nonimmigrant visa issuing posts throughout the world.

Burroughs visas contained a space in which a consular employee was required to write the name of the alien to whom the visa was being issued. An alien’s passport might also include family members, such as a spouse, or children, who also had to be listed on the visa. In March 1983, in order to expedite the issuance of nonimmigrant visas and to improve operational efficiency, the Department authorized the use of a “bearer(s)” stamp for certain countries so that consular officers would not have to spend time writing in the applicant’s name (and those of accompanying family members). MRVs, however, must be issued individually to qualified aliens. Consequently, the “bearer”annotation has become obsolete.

The problem with the old Burroughs machine, besides the obvious, was maybe — you run out of ink, the plates are ruined/broken or you need it oiled. We could not remember those breaking down. With the MRV technology, all posts are connected to a central database, and the new machines by themselves cannot issue visas.  Which brings us to the security of that system.

Management Alert on Information System Security Program

The State Department PIA says that “To appropriately safeguard the information, numerous management, operational, and technical security controls are in place in accordance with the Federal Information Security Management Act (FISMA) of 2002 and information assurance standards published by the National Institute of Standards and Technology (NIST).” Must be why in November 2013, the Office of the Inspector General issued a Management Alert  for significant and recurring weaknesses found in the State Department’s Information System Security Program over the past three fiscal years (FY 2011-2013).

In 2011, State/OIG also issued a report on CA’s CST division and has, what appears to be a lengthy discussion of the CCD, but almost all of it but a paragraph had been redacted:

That OIG report also includes a discussion of the Systems Development Life Cycle Process and notes that decision control gates within CST’s SDLC process are weak. It cites a couple of examples where this manifested: 1) the development of the Consular report of Birth Abroad (CRBA) system. “The ownership of development and deployment shifted throughout the process, and the business unit’s requirements were not clearly communicated to the development team. As a result, CST designed and tested the CRBA for a printer that did not match the printer model identified and procured by the business unit;” 2)  the Crisis Task Force application, for which CST was tasked to enhance its Web-facing interaction. “The deployment of this application has been challenged by the lack of project ownership and decision controls, as well as by the incomplete requirements definition. The use of incorrect scripts that were provided by the CM group has further delayed the Crisis Task Force application’s deployment.”

If there’s somethin’ strange in your CCD, who ya gonna call? (Glitchbusters!)

The Consular Consolidated Database (CCD) is central to all consular operations. It is run by CST where according to the OIG, “the smooth functioning of every part of the office depends on its contractors.” And because it runs such an important element of U.S. national security systems, if all CST’s contractors, all 850 of them quit, this critical consular data delivery to the State Department and other Federal agencies would screech to a a halt.

To carry out its mandate, CST must provide uninterrupted support to 233 overseas posts, 21 passport agencies, 2 passport processing centers, and other domestic facilities, for a total of 30,000 end users across 16 Federal agencies and in nearly every country. CST faces 24/7/365 service requirements, as any disruption in automated support brings operations to an immediate halt, with very serious implications for travelers and the U.S. image.
[…]
CST is led by a director and is staffed by 68 full-time equivalent (FTE) employees (62 Civil Service and 6 Foreign Service). There are 12 positions (3 Foreign Service and 9 Civil Service) currently vacant. CA recently authorized CST 19 additional FTE positions. There are also more than 850 contractors operating under nearly 30 different contracts. In FY 2010, CST’s annual operating budget was approximately $266 million.

If CCD is compromised for a lengthy period such as the last couple of weeks, what is the back up plan to keep the operation going?  Obviously, none. It’s either down or running under limited or full capacity.  No one we know remember CCD problems persist this long.  Right now, we know from a reliable source that the system is not down, and some cases and going through but — what if the CCD is completely down for two weeks … four weeks … wouldn’t international travel come to a slow stop?

What if CCD goes down indefinitely whether by hardware or software glitch or through malicious penetration by foreign hackers, what happens then?

Currently, it appears nothing can be done but for folks to be patient and wait until the fixes are in.  We know they’re working hard at it but there’s got to be a better way.   Perhaps we can also agree that this has very serious national security implications on top of disgruntled travelers and a grave impact on the U.S. image overseas.

 Related items:

May 2011 |  Inspection of The Bureau of Consular Affairs, Office of Consular Systems and Technology (CST) Report Number ISP-I-11-51

-11/30/13   Audit of Department of State Information Security Program (FISMA) (AUD-IT-14-03)  [3610 Kb]  Posted January 29th, 2014

-01/13/14   Management Alert on OIG Findings of Significant, Recurring Weaknesses in Dept of State Info System Security Program (MA-A-0001)  [6298 Kb]  Posted on January 16, 2014

 Related articles

• US visa chaos for thousands of Irish waiting for travel documents (IrishCentral) (irishcentral.com)
• Joseph Parker Also Facing Visa Battle To The U.S. (boxingscene.com)
• Tech hiccup delays US visa processing (arabtimesonline.com)
• US consular database crash: Not a good time to renew your passport (siliconangle.com)
• Technical problems hold up US visa processing (thanhniennews.com)
• ‘My papers, please?’ US passport database crash leaves travelers stranded (go.theregister.com)
• Parker still fighting for US bout (3news.co.nz)
• Technical glitch affects US visa processing (manilatimes.net)
• ‘My papers, please?’ US passport database crash leaves travelers stranded (go.theregister.com)
• Parker still fighting for US bout (3news.co.nz)
• Technical glitch affects US visa processing (manilatimes.net)