Tag Archives: Consular Systems and Technology

State Dept’s Consular Database Problems Dash Plans, Dreams … Also Cost Up to $1M/Day in California

June 16, 2015 By in Consular Work, Foreign Service, Functional Bureaus, Security, Shutdown, Social Media, State Department, Technology and Work, Trends, Visas Tags: , , , , , , , ,

Posted: 4:46 pm  EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

On June 12, we posted about the technical problems with the State Department’s overseas passport and visa systems.  Passport applications accepted overseas on or after May 26, 2015 are affected but emergency passports are available.  A hardware failure on June 9 halted the flow of biometric clearance requests from posts to the State Department’s Consular Consolidated Database (CCD). Individuals who submitted online applications or were interviewed for visas on or after June 9 are affected and are asked to reschedule appointments . No emergency visas available. See State Dept’s Overseas Passport and Visa Systems Hit By Glitch Again, Suspends Issuance.

On June 15, the Bureau of Consular Affairs posted the following update on its Facebook page but not on its travel.state.gov news page:

We continue to experience technical problems with our visa systems. This issue is not specific to any particular country or visa category. We apologize for the inconvenience and we are working around the clock to correct the problem. Currently, we are unable to print most immigrant and nonimmigrant visas approved after June 8, 2015. In addition, U.S. embassies and consulates are unable to process new applications submitted on or after June 9, 2015. If you have a visa interview appointment scheduled for June 14-20, 2015, and you submitted your DS-160 online application **after June 9, 2015,** you should reschedule your appointment. If you submitted your DS-160 online application prior to June 9, 2015, you should plan to attend your scheduled visa interview appointment. Our embassies and consulates will be posting location-specific information on their websites, so please check the website of the location where you applied for your visa for more information. 

The technical issues also affected the Department of State’s ability to adjudicate applications for U.S. passports accepted overseas between May 26 and June 14, 2015. If you applied for a U.S. passport overseas during this time frame and have travel plans within the next 10 business days, please consider requesting an emergency passport at the U.S. embassy or consulate at which you originally applied. Information about how to apply for an emergency passport is available on the embassy/consulate website.

The previous time the CCD crashed big time was last summer (see State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience).  It could just be a coincidence (or not!) but the crash has now happened twice during the peak travel season. During the meltdown last summer, CA said that CCD was going to have an upgrade at the end of 2014. It also  said at that time that the upgrade plan included two redundant systems. If this glitch started on May 26th, we’re approaching the three week-mark. And so far, those redundant systems are missing in action.

The Bureau of Consular Affairs on its FAQ states that “This is not the same issue as last year.” But we learned from an unofficial source that “All line officers know that last summer’s CCD glitch was never completely fixed.” 

So, which is it?

On June 15, the Wall Street Journal reported that the CCD glitch has left agricultural workers stranded at the border just as the summer harvest gets under way. Jason Resnick, the general counsel for the Western Growers Association, which represents farmers in California, Arizona and Colorado calls this glitch, “a crisis.” Apparently, more than 1,000 workers who expected H-2A agricultural visas are stuck on the Mexican side of the border, where motels are overflowing.

“The workers are overdue to start harvesting berries and other crops on U.S. farms. Mr. Resnick estimated that California agriculture, already stressed by drought, is losing $500,000 to $1 million for each day of delay.”

.

The State Department’s consular operation is an enormous one with many parts and affects a large number of travelers.  The State Department issued 9,932,480 nonimmigrant/temporary visas in FY2014. It issued 467,370 immigrant/permanent visas in FY2014. During the same period, domestic and overseas passport offices issued 14,087,341 U.S. passports (including 1,463,191 passport cards).

A small fraction of those frustrated travelers have taken to Facebook to connect with Consular Affairs.

One who is stuck in Canada and could not return to her 14th month old baby writes:

Hi, do you have any timeline to fix the issues? I live in Boston, US & visited Vancouver, BC consulate on June 10th for my visa interview. visa officer told me that your visa is approved and you will get your passport back in 3 days. However, since June 10th, there is no update on my visa. I also inquired with Vancouver Consulate and they are ready to give me passport back without visa. As a matter of fact, I can’t enter into US legally until and unless I got printed Visa. My 14th month baby is waiting in Boston,MA and I got stuck here. Can’t do anything.

Here’s one waiting to be reunited with a loved one after a long wait:

Waiting is excruciating my husband was approved on june 10th and my mother has passed away. We need him home please get this fixed our application has been in since 2013.

Somebody who is pregnant, stuck in Mexico writes:

I am currently 8 months pregnant and have been waiting for my TD visa renewal since mid May. Since I will soon have travel restrictions to fly back home, does this qualify as an urgent humanitarian situation where I should contact the embassy in Mexico directly?

One who missed grandma’s funeral makes a plea:”

Can you tell us a estimated time??? My case was expedited and I miss my Grandmother funeral. Me and my wife were supposed to travel yesterday. Please get this fixed.please

A family stuck in Mexico:

Do you have an ETA in order for ys to plan accordingly? I had my appointment on friday june 12th and I am stuck in Mexico (H1B renewal) without passport and without the ability to get back to work in Boston. Flight fees, hotel fees and a family of 4 that needs to get back to Boston.

Some people missing a convention:

My wife and I have a flight to ny tom. Our visa were approved on 9th. We are part of 100+ group attending a convention. Do we expect to get our passports with visas today?

Somebody stuck in Guatemala, fears loss of a job:

All my documents were in order and approved June 1st. my husband and I are in a dire situation stuck in Guatemala. I’m at risk of losing my job if I don’t return to USA.

A frustrated somebody who calls out other technical issues:

There’s always something wrong. There are already technical issues with payment of IV fees, DS260 and DS261. This system needs to be revamped. What’s the government doing about this?!

A Romanian group who worked and saved to attend the Genius Olympiad:

We are desperate. We have a plane ticker for tomorrow and we were supposed to go to an international competition (Genius Olympiad) in Oswego, NYC. Apart from the part that we lost thousands of dollars, our hopes got crashed because we worked for a year at our projects and invested a lot of time and monney… For… Nothing?!?!?! How come you have no plan B for solving this issue? We tried making an appointment more than one month ago and they said on the 9th on june will be our interview, we said it s too late for us but they said that the visa will be delivered within 2 days maximum. And here we are 5 days later with no visas, with crashed hopes, tons of lost monney, wasted time… Should i go on??

#

Related posts:

 

State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience

July 30, 2014 By in Bugs, Consular Work, Contractors, Federal Agencies, Foreign Service, Govt Documents, Huh? News, Legacy, Security, Spectacular, Staffing the FS, State Department, Technology and Work, Visas Tags: , , , , , , , , , 1 Comment

— Domani Spero
[twitter-follow screen_name=’Diplopundit’ ]

 

The first announcement about the troubled Consular Consolidated Database (CCD) went out on Wednesday, July 23:

The Department of State Bureau of Consular Affairs is currently experiencing technical problems with our passport/visa system.  This issue is worldwide and is not specific to any particular country, citizenship document, or visa category.  We apologize to applicants who are experiencing delays or are unable to obtain a passport, Consular Report of Birth Abroad, or visa at this time. We are working urgently to correct the problem and expect our system to be fully operational again soon.

The AP reported on July 23 that unspecified glitches have resulted in performance issues since Saturday, which would be July 19.

On July 25, CA announced:” Our visa and passport processing systems are now operational, however they are working at limited capacity. We are still working to correct the problem and expect to be fully operational soon.”

A State Department official speaking on background told us the same day that this issue was not/not caused by  hackers. We were told that the CCD crashed shortly after maintenance was performed and that the root cause of the problem is not yet known.

On July 27, CA released an update:

As of July 27, the Department of State has made continued progress on restoring our system to full functionality. As we restore our ability to print visas, we are prioritizing immigrant cases, including adoptions visas. System engineers are performing maintenance to address the problems we encountered. As system performance improves, we will continue to process visas at U.S. Embassies and Consulates worldwide. We are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.

On July 29, CA posted this on FB:

The Department of State Bureau of Consular Affairs continues to make progress restoring our nonimmigrant visa system to full functionality. Over the weekend, the Department of State implemented system changes aimed at optimizing performance and addressing the challenges we have faced. We are now testing our system capacity to ensure stability. Processing of immigrant visas cases, including adoptions, remains a high priority. Some Embassies and Consulates may temporarily limit or reschedule nonimmigrant visa interview appointments until more system resources become available to process these new applications. We sincerely regret the inconvenience to travelers, and are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.

 

The CA Bureau’s Facebook page has been inundated with comments. There were complaints that at one post the visas were printing fine and then they were not. There were complains from people waiting for visas for adopted kids, for fiancees, for family members, for family waiting at the border, for students anxious to get to their schools, people worried about time running out for diversity visas, applicants with flights already booked, and many more.  One FB commenter writes, “I feel that the problem most people have is not that the system broke, but the lack of clear, meaningful information so people can make appropriate plans.”

Other than what the CA Bureau chose to tell us, we cannot pry any substantial detail from official sources.  We, however, understand from sources familiar with the system but not authorized to speak for the bureau that the CCD has been having problems for sometime but it got worse in the last couple weeks.   If you’re familiar with the highs and lows of visa operation, this will not be altogether surprising.  Whatever problems already existed in the system prior to this “glitch” could have easily been exacerbated in July, which is the middle of the peak travel season worldwide. A source working in one of our consular posts confirmed to us that the system is back running, but not at the normal level and that the backlogs are building up. Another source told us that Beijing already had a 15k NIV backlog over the weekend.  We haven’t yet heard what are the backlogs like in mega visa-issuing posts like Brazil, Mexico and India.

We understand that everyone is currently doing all they can to get the process moving, but that some cases are getting through the system, while some are not. No one seems to know why this is happening. These machine readable visas are tied to the system and there are no manual back-ups for processing these cases (more of that below).

 

So who owns CCD?

The Consular Systems and Technology (CA/CST) manages the CCD.  We have previously blogged about its troubled past:

CST is currently headed by a new Director, Greg D Ambrose who reports to the CA Bureau’s Assistant Secretary.  It looks like despite the 2011 OIG recommendation, the CST deputy position remains vacant. We should also note that the  Asst Secretary for Consular Affairs Janice Jacobs retired this past April.  No replacement has been nominated to-date and Michele T. Bond has been Acting Assistant Secretary since Ms. Jacobs’ departure.

Last September, Mr. Ambrose was with FedScoopTV and talked about Consular One, the future of consular IT.

 

CST Just Got a New Data Engineering Contract

In Many 2014, ActioNet, Inc., headquartered in Vienna, Virginia,announced a 5-year task order for data engineering, supporting CST.

ActioNet, Inc. announced today the award of a five (5)-year task order entitled Data Engineering (DE) in support of Department of State (DOS). This task order will provide data engineering and database infrastructure support services necessary for planning, analysis, design, and implementation services for the Bureau of Consular Affairs.  These service also include contract and program management support to ensure that innovation, efficiency, and cost control practices are built into the program. […] The Office of Consular Systems and Technology (CST) within the Bureau develops, deploys and maintains the unclassified and classified IT infrastructures that help execute these missions. The Bureau currently manages over 800 servers worldwide, in order to comply with the fast paced changes inherent to data processing and telecommunications, CST requires that contractor services provide for rapid provisioning of highly experienced and trained individuals with the IT (information technology) backgrounds and the security clearances required of CA’s environment of workstation-based local and wide-area network infrastructures.

Due to limited information available, we don’t know if the new Consular One and/or the new DE contract are related to ongoing issues or if there are hardware issues, given the multiple legacy systems, but we do know that CST has both an impressive and troubled history. Let’s take a look.


Records Growing by the Day

The 2010 Consular Consolidated Database (CCD) Privacy Impact Assessment (PIA) describes (pdf) the CCD as “one of the largest Oracle based data warehouses in the world that holds current and archived data from the Consular Affairs (CA) domestic and post databases around the world.”  According to the PIA, in December 2009, the CCD contained over 100 million visa cases and 75 million photographs, utilizing billions of rows of data, and has a current growth rate of approximately 35 thousand visa cases every day. The 2011 OIG report says that in 2010, the CCD contained over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day.

That was almost four years ago.


A Critical Operational and National Security Database with No Back-Up System?

According to publicly available information, the CCD’s chief functions are 1) to support data delivery to approved applications via industry-standard Web Service queries, 2) provide users with easy-to-use data entry interfaces to CCD, and 3) allow emergency recovery of post databases.  The CCD also serves as a gateway to IDENT and IAFIS fingerprint checking databases, the Department of State Facial Recognition system, and the NameCheck system. It  provides access to passport data in Travel Document Issuance System (TDIS), Passport Lookout Tracking System (PLOTS), and Passport Information Electronic Records System (PIERS).  The OIG says that the CCD serves 11,000 users in the Department and more than 19,000 users in other agencies, primarily the Department of Homeland Security (DHS) and various law enforcement elements, and is accessed more than 120 million times every month.

Given that the CCD is considered “a critical operational and national security database,” there is surprisingly no redundancies or any back-up system.


Resurrect the Standard Register protectograph aka: `Burroughs visas’?

No one is actually suggesting that but when the CCD system is down, there is no manual way to issue a visa. No post can  handprint visas  because security measures prevent consular officers from printing a visa unless it is approved through the database system. Here is a quick history of the handprinted ‘Burroughs visas’ and the machine readable visas via the GPO:

November 18, 1988, mandated the development of a machine-readable travel and identity document to improve border entry and departure control using an automated data-capture system. As a result, the Department developed the Machine Readable Visa, a durable, long-lasting adhesive foil made out of Teslin.

Before MRVs, nonimmigrant visas were issued using a device called a Standard Register protectograph, otherwise known as a Burroughs certifier machine. It produced what was colloquially known as a “Burroughs visa,” an indelible ink impression mechanically stamped directly onto a page in the alien’s passport. Over time, Burroughs machines were gradually replaced by MRV technology, which is now used exclusively by all nonimmigrant visa issuing posts throughout the world.

Burroughs visas contained a space in which a consular employee was required to write the name of the alien to whom the visa was being issued. An alien’s passport might also include family members, such as a spouse, or children, who also had to be listed on the visa. In March 1983, in order to expedite the issuance of nonimmigrant visas and to improve operational efficiency, the Department authorized the use of a “bearer(s)” stamp for certain countries so that consular officers would not have to spend time writing in the applicant’s name (and those of accompanying family members). MRVs, however, must be issued individually to qualified aliens. Consequently, the “bearer”annotation has become obsolete.

The problem with the old Burroughs machine, besides the obvious, was maybe — you run out of ink, the plates are ruined/broken or you need it oiled. We could not remember those breaking down. With the MRV technology, all posts are connected to a central database, and the new machines by themselves cannot issue visas.  Which brings us to the security of that system.

 

Management Alert on Information System Security Program

The State Department PIA says that “To appropriately safeguard the information, numerous management, operational, and technical security controls are in place in accordance with the Federal Information Security Management Act (FISMA) of 2002 and information assurance standards published by the National Institute of Standards and Technology (NIST).” Must be why in November 2013, the Office of the Inspector General issued a Management Alert  for significant and recurring weaknesses found in the State Department’s Information System Security Program over the past three fiscal years (FY 2011-2013).

In 2011, State/OIG also issued a report on CA’s CST division and has, what appears to be a lengthy discussion of the CCD, but almost all of it but a paragraph had been redacted:

Screen Shot 2014-07-30 at 8.40.37 AM

That OIG report also includes a discussion of the Systems Development Life Cycle Process and notes that decision control gates within CST’s SDLC process are weak. It cites a couple of examples where this manifested: 1) the development of the Consular report of Birth Abroad (CRBA) system. “The ownership of development and deployment shifted throughout the process, and the business unit’s requirements were not clearly communicated to the development team. As a result, CST designed and tested the CRBA for a printer that did not match the printer model identified and procured by the business unit;” 2)  the Crisis Task Force application, for which CST was tasked to enhance its Web-facing interaction. “The deployment of this application has been challenged by the lack of project ownership and decision controls, as well as by the incomplete requirements definition. The use of incorrect scripts that were provided by the CM group has further delayed the Crisis Task Force application’s deployment.”

 

If there’s somethin’ strange in your CCD, who ya gonna call? (Glitchbusters!)

The Consular Consolidated Database (CCD) is central to all consular operations. It is run by CST where according to the OIG, “the smooth functioning of every part of the office depends on its contractors.” And because it runs such an important element of U.S. national security systems, if all CST’s contractors, all 850 of them quit, this critical consular data delivery to the State Department and other Federal agencies would screech to a a halt.

To carry out its mandate, CST must provide uninterrupted support to 233 overseas posts, 21 passport agencies, 2 passport processing centers, and other domestic facilities, for a total of 30,000 end users across 16 Federal agencies and in nearly every country. CST faces 24/7/365 service requirements, as any disruption in automated support brings operations to an immediate halt, with very serious implications for travelers and the U.S. image.
[…]
CST is led by a director and is staffed by 68 full-time equivalent (FTE) employees (62 Civil Service and 6 Foreign Service). There are 12 positions (3 Foreign Service and 9 Civil Service) currently vacant. CA recently authorized CST 19 additional FTE positions. There are also more than 850 contractors operating under nearly 30 different contracts. In FY 2010, CST’s annual operating budget was approximately $266 million.

If CCD is compromised for a lengthy period such as the last couple of weeks, what is the back up plan to keep the operation going?  Obviously, none. It’s either down or running under limited or full capacity.  No one we know remember CCD problems persist this long.  Right now, we know from a reliable source that the system is not down, and some cases and going through but — what if the CCD is completely down for two weeks … four weeks … wouldn’t international travel come to a slow stop?

What if CCD goes down indefinitely whether by hardware or software glitch or through malicious penetration by foreign hackers, what happens then?

Currently, it appears nothing can be done but for folks to be patient and wait until the fixes are in.  We know they’re working hard at it but there’s got to be a better way.   Perhaps we can also agree that this has very serious national security implications on top of disgruntled travelers and a grave impact on the U.S. image overseas.

 

 Related items:

May 2011 |  Inspection of The Bureau of Consular Affairs, Office of Consular Systems and Technology (CST) Report Number ISP-I-11-51

-11/30/13   Audit of Department of State Information Security Program (FISMA) (AUD-IT-14-03)  [3610 Kb]  Posted January 29th, 2014

-01/13/14   Management Alert on OIG Findings of Significant, Recurring Weaknesses in Dept of State Info System Security Program (MA-A-0001)  [6298 Kb]  Posted on January 16, 2014

 

 Related articles