Commissioned Internal Review Finds @StateDept’s Consular Consolidated Database With Security Gaps

Posted: 3:52 am ET
[twitter-follow screen_name=’Diplopundit’ ]

 

According to the Privacy Impact Assessment (PIA) of December 2009, the Consular Consolidated Database (CCD)  contained over 100 million visa cases and 75 million photographs, utilizing billions of rows of data, and has a current growth rate of approximately 35 thousand visa cases every day.  The 2010 Consular Consolidated Database (CCD) Privacy Impact Assessment (PIA) describes (pdf) the CCD as “one of the largest Oracle based data warehouses in the world that holds current and archived data from the Consular Affairs (CA) domestic and post databases around the world.”  The 2011 OIG report says that in 2010, the CCD contained over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day.

Related posts:

 

#

 

Advertisements

CCD: Report Says Visa Processing Systems Pose Significant Challenges; Also Face User-Friendlessness?

Posted: 12:02 am EDT
[twitter-follow screen_name=’Diplopundit’ ]

Via GAO

According to Commerce, international travelers contributed $220.6 billion to the economy and supported 1.1 million jobs in 2014. Processing visas for such travelers as efficiently and effectively as possible without compromising our national security is critical to maintaining a competitive and secure travel and tourism industry in the United States. Although State has historically struggled with the task of maintaining reasonable wait times for NIV interviews, it has undertaken a number of efforts in recent years that have yielded substantial progress in reducing such waits.

Significant projected increases in NIV demand coupled with consular hiring constraints and other challenges could hinder State’s ability to sustain this progress in the future—especially in countries where the demand for visas is expected to rise the highest. These challenges heighten the importance of systematically evaluating the cost and impact of the multiple measures State has taken to reduce interview wait times in recent years and leveraging that knowledge in future decision making. Without this, State’s ability to direct resources to those activities that offer the greatest likelihood of success is limited. Moreover, State’s future capacity to cope with rising NIV demand will be challenged by inefficiencies in its visa processing technology; consular officers and management officials at the posts we visited pointed to cumbersome user procedures and frequent system failures as enormous obstacles to efficient NIV processing. State’s Bureau of Consular Affairs recognizes these problems and plans a number of system enhancements; however, the bureau does not systematically collect input from consular officers to help guide and prioritize these planned upgrades. Without a systematic effort to gain the input of those who employ these systems on a daily basis, State cannot be assured that it is investing its resources in a way that will optimize the performance of these systems for current and future users.

giphy_daleks

Consular officers and managers at posts we visited identified current information technology (IT) systems as one of the most significant challenges to the efficient processing of NIVs. Consular officers in all 11 focus groups we conducted across the four posts we visited stated that problems with the Consular Consolidated Database (CCD) and the NIV system create significant obstacles for consular officers in the processing of NIVs.26 Specifically, consular officers and managers at posts stated that frequent NIV system outages and failures (where the system stops working) at individual posts, worldwide system outages of CCD, and IT systems that are not user friendly, negatively affected their ability to process NIVs.

— NIV system outages and failures at posts: Consular officers we spoke with in Beijing, Mexico City, New Delhi, and São Paulo explained that the NIV system regularly stops working. This results in a reduced number of adjudications (whether being performed at the interview window or, for an IWP applicant, at an officer’s desk) in a day. Notably, consular officers in 4 of the 11 focus groups reported having to stop work or re-adjudicate NIV applications as a result of these NIV system failures. In fact, during our visit to the U.S. Embassy in New Delhi in March 2015, a local NIV outage occurred, affecting consular officers’ ability to conduct adjudications. In January 2015, officers in Bogotá, Guadalajara, Monterrey, and Moscow—among the top 15 posts with the highest NIV applicant volume in 2014— experienced severe NIV performance issues—specifically an inability to perform background check queries against databases.

— Worldwide outages and operational issues of CCD: Since July 2014, two worldwide outages of CCD have impaired the ability of posts to process NIV applications. On June 9, 2015, an outage affected the ability of posts to run checks of biometric data, thus halting most visa printing along with other services offered at posts.27 According to State officials, the outage affected every post worldwide for 10 days. The system was gradually repaired, but it was not fully restored at all posts until June 29, 2015, exacerbating already increased NIV interview wait times at some posts during the summer high demand season.According to State notices, another significant outage of CCD occurred on July 20, 2014, slowing NIV processing worldwide until September 5, 2014, when CCD returned to full operational capacity.28 State estimated that from the start of operational issues on July 20 through late July, State issued approximately 220,000 NIVs globally— about half of the NIVs State anticipated issuing during that period. According to officials in State’s Bureau of Consular Affairs, Office of Consular Systems and Technology (CST), who are responsible for operating and maintaining CCD and the NIV system, consular officers were still able to collect NIV applicant information during that period; however, processing of applications was significantly delayed with an almost 2-week backlog of NIVs. In the U.S. Consulate in São Paulo, a consular management official reported that due to this outage, the post had a backlog of about 30,000 NIV applications, or approximately 9 days’ worth of NIV interviews during peak season. Consular officers in 8 out of the 11 focus groups we conducted identified a lengthy CCD outage as a challenge to the efficient processing of NIVs.

— IT systems are not user friendly: In 9 out of 11 focus groups, consular officers described the IT systems for NIV processing as not user friendly. Officers in our focus groups explained that some aspects of the system hinder their ability to quickly and efficiently process NIVs. These aspects include a lack of integration among the databases needed for NIV adjudications, the need for manual scanning of documentation provided by an applicant, and an absence of standard keyboard shortcuts29 across all IT applications that would allow users to quickly copy information when processing NIV applications for related applicants, to avoid having to enter data multiple times. Some consular officers in our focus groups stated that they could adjudicate more NIVs in a day if the IT systems were less cumbersome and more user friendly.

— Consular officers in Beijing and Mexico City and consular management at one post indicated that the NIV system appeared to be designed without consideration for the needs of a high volume post, which include efficiently processing a large number of applications per adjudicator each day. According to consular officers, the system is poor at handling today’s high levels of demand because it was originally designed in the mid- 1990s. Consular officers in São Paulo stated that under current IT systems and programs, the post may not be able to process larger volumes that State projects it will have in the future.

— State, recognizing the limits of its current consular IT systems, initiated the development of a new IT platform. State is developing a new system referred to as “ConsularOne,” to modernize 92 applications that include systems such as CCD and the NIV system. According to State, ConsularOne will be implemented in six phases, starting with passport renewal systems and, in phase five, capabilities associated with adjudicating and issuing visas (referred to as non-citizen services). However, CST officials have yet to formally commit to when the capabilities associated with non-citizen services are to be implemented. According to a preliminary CST schedule, the enhanced capabilities associated with processing NIVs are not scheduled for completion until October 2019. Given this timeline, according to State officials, enhancements to existing IT systems are necessary and are being planned.

State Does Not Systematically Obtain End User Input to Prioritize Improvement Efforts for Current IT Systems

Although consular officers and managers we spoke with identified CCD and the NIV system as one of the most significant challenges to the efficient processing of NIVs, State does not systematically measure end user (i.e., consular officers) satisfaction. We have previously reported that in order for IT organizations to be successful, they should measure the satisfaction of their users and take steps to improve it.30 The Software Engineering Institute’s IDEALSM model is a recognized approach for managing efforts to make system improvements.31 According to this model, user satisfaction should be collected and used to help guide improvement efforts through a written plan. With such an approach, IT improvement resources can be invested in a manner that provides optimal results.

Although State is in the process of upgrading and enhancing CCD and the NIV system, State officials told us that they do not systematically measure user satisfaction with their IT systems and do not have a written plan for improving satisfaction. According to CST officials, consular officers may voluntarily submit requests to CST for proposed IT system enhancements. Additionally, State officials noted that an IT stakeholder group comprising officials in State’s Bureau of Consular Affairs regularly meets to identify and prioritize IT resources and can convey end user concerns for the system.32 However, State has not collected comprehensive data regarding end user satisfaction and developed a plan to help guide its current improvement efforts. Furthermore, consular officers continued to express concerns with the functionality of the IT systems, and some officers noted that enhancements to date have not been sufficient to address the largest problems they encounter with the systems.

Given consular officers’ reliance on IT services provided by CST, as well as the feedback we received from focus groups, it is critical that State identify and implement feedback from end users in a disciplined and structured fashion for current and any future IT upgrades. Without a systematic approach to measure end user satisfaction, CST may not be able to adequately ensure that it is investing its resources on improvement efforts that will improve performance of its current and future IT systems for end users.

#

State Dept Authorization Bill Mandates Security Breach Reporting, NSA Consultations –Can PenTest Be Far Behind?

Posted: 12:27 am EDT
Updated: 11:23 am PDT
[twitter-follow screen_name=’Diplopundit’ ]

 

Update: A source on the Hill alerted us that the State Authorization bill was offered as an amendment when the NDAA was debated in the Senate last month but it was not voted on and the NDAA passed on June 18 (That would be H.R. 1735 which passed 215 (71-25)  We understand that both chambers are now starting the process to bring the bill to conference in order to resolve differences.  The State Authorization bill, we are told, will not be part of those discussions.  In order for this to move forward, it will either need to be brought to the floor as a stand alone vote or Corker/Cardin could try again to attach it to another piece of legislation. Given that this is the first authorization bill passed by the SFRC in 5 years, and made it through the committee with bi-partisan support, we suspect that the senators will not just easily forget about this. — DS

On June 9, 2015, U.S. Senators Bob Corker (R-Tenn.) and Ben Cardin (D-Md.), the chairman and ranking member of the Senate Foreign Relations Committee, applauded the unanimous committee passage of the Fiscal Year 2016 Department of State Operations Authorization and Embassy Security Act. The SFRC statement says that it has been five years since the Senate Foreign Relations Committee passed a State Department Authorization bill and 13 years since one was enacted into law.  This State Department Authorization bill has been offered as an amendment to the National Defense Authorization Act, which currently is on the Senate floor. It is quite lengthy so we’re doing this in installments.

Below is the section on information technology system security that mandates security breach reporting, as well as making State Dept systems and networks available to the Director of the National Security Agency (NSA) and any other such departments or agencies to carry out necessary tests and procedures.

The State Department’s Consular Consolidated Database (CCD) as of 2011 contains over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day. If the CCD is compromised, it would be a jackpot for hackers that would make the OPM hack severely pales in comparison.

If this bill passes, will the penetration test by NSA on one of the world’s largest data warehouses finally happen?

Via govtrack:

Section 206.Information technology system security

(a)In general

The Secretary shall regularly consult with the Director of the National Security Agency and any other departments or agencies the Secretary determines to be appropriate regarding the security of United States Government and nongovernment information technology systems and networks owned, operated, managed, or utilized by the Department, including any such systems or networks facilitating the use of sensitive or classified information.

(b)Consultation

In performing the consultations required under subsection (a), the Secretary shall make all such systems and networks available to the Director of the National Security Agency and any other such departments or agencies to carry out such tests and procedures as are necessary to ensure adequate policies and protections are in place to prevent penetrations or compromises of such systems and networks, including by malicious intrusions by any unauthorized individual or state actor or other entity.

(c)Security breach reporting

Not later than 180 days after the date of the enactment of this Act, and every 180 days thereafter, the Secretary, in consultation with the Director of the National Security Agency and any other departments or agencies the Secretary determines to be appropriate, shall submit a report to the appropriate congressional committees that describes in detail—

(1)all known or suspected penetrations or compromises of the systems or networks described in subsection (a) facilitating the use of classified information; and

(2)all known or suspected significant penetrations or compromises of any other such systems and networks that occurred since the submission of the prior report.

(d)Content

Each report submitted under subsection (c) shall include—

(1)a description of the relevant information technology system or network penetrated or compromised;

(2)an assessment of the date and time such penetration or compromise occurred;

(3)an assessment of the duration for which such system or network was penetrated or compromised, including whether such penetration or compromise is ongoing;

(4)an assessment of the amount and sensitivity of information accessed and available to have been accessed by such penetration or compromise, including any such information contained on systems and networks owned, operated, managed, or utilized by any other department or agency of the United States Government;

(5)an assessment of whether such system or network was penetrated by a malicious intrusion, including an assessment of—

(A)the known or suspected perpetrators, including state actors; and

(B)the methods used to conduct such penetration or compromise; and

(6)a description of the actions the Department has taken, or plans to take, to prevent future, similar penetrations or compromises of such systems and networks.

#

Related Post:
S.1635: DOS Operations Authorization and Embassy Security Act, Fiscal Year 2016 – Security Clearance

State Department’s Visa Systems Now Operational at 165 of 220 Posts Worldwide

Posted: 1:56 am  EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

The State Department’s Consular Consolidated Database problems that affected travelers globally is is now back online at 165 of 220 visa issuance posts worldwide.  The latest update does not explain in details the cause of the glitch except to cite the hardware issue.  It also says that service was restored “using a redundant, secondary backup system and other sources.”  It does not explain what “other sources” mean but if it took at least 9 days to get that redundant, secondary back-up system to kick in, that’s not a very good system.

The Consular Affairs-issued FAQ asks how many people were affected by this outage? The answer it provides to this question is neither here nor there.  Folks, if you can’t answer your own question, please don’t include it.

According to travel.state.gov, the average visa applications processed every day worldwide is 50,000 x 9 days (June 9-19)=450,000 + 25,000 (half the average daily applications) x 4 days (June 22-25) = 100,000. Total number potentially affected 550,000.  Is that close enough?

The June 25 update says that if systems had been operating normally, posts would have issued approximately 540,000 visas since the outage started. Whoa! Help us out here. What kind of refusal/approval rates are we looking at here? That 540,000 figure is a little hinky because not all applicants who apply are issued visas. If it would have issued 540,000  visas, what would have been the total number of applicants?  Note that all of them must pay the visa fees. We estimate that the USG loss from this latest glitch is between $72 to $84 million (average daily applications globally x no. of days x $160 visa fee). Is that too low?

Meanwhile, StarrFMonline.com reported that the US Embassy in Accra, has “dismissed reports that it is ripping Ghanaians off by accepting visa fees in spite of the visa issuance imbroglio that has hit US embassies across the world.” The consular section chief  had to explain that “if anybody was refused a visa, that was because of the case and has nothing to do with our technical issues.”

*

On June 24, the Bureau of Consular Affairs reports that 50 posts, representing nearly 73 percent of its  nonimmigrant visa demand worldwide, are back online and issuing visas.  It also says that “posts overseas have issued more than 150,000 non-immigrant visas since June 9.” And that for context, if systems had been operating normally, posts would have issued approximately 450,000 visas during the June 9-23 timeframe.

On June 25, the Bureau of Consular Affairs reports that 165 posts, representing more than 85 percent of nonimmigrant visa demand worldwide, are now online and issuing visas.  The update says that if systems had been operating normally, posts would have issued approximately 540,000 visas since the outage started.

Via travel.state.gov, June 25 update:

Visa Systems Issues

  • The Bureau of Consular Affairs reports that 165 posts, representing more than 85 percent of our nonimmigrant visa demand worldwide, are now online and issuing visas. 

  • Posts overseas issued more than 82,000 visas on June 24. 

  • Posts overseas have issued more than 238,000 non-immigrant visas this week. For context, if systems had been operating normally, posts would have issued approximately 540,000 visas since the outage started. 

  • We will continue to bring additional posts online until connectivity with all posts is restored. All posts worldwide are now scheduling interviews with applicants, including with those who applied after the systems problems began on June 9.

  • We deeply regret the inconvenience to travelers who are waiting for visas, as well as their families and U.S. businesses that have been affected.

  • We continue to post updates to our website, travel.state.gov.

 

Q: Reports indicate that your backlog is 700,000 visas. Is this accurate?

No. While there is a large backlog of cases to clear, it never approached that level, and we have already made good progress issuing those visas. Many posts are working overtime this week and during the upcoming weekend, and we expect to eliminate the backlog in a week or less.


Q: How old is this equipment? And does the age of the equipment and the need to have so many repairs to the hardware mean that this equipment should have been replaced? Is this a funding issue at the base of it?

The hardware that impacted the biometrics system is several years old. The Department was working to move the biometrics system off of this hardware.

The operational requirements to keep this database running for domestic and overseas passport and visa issuances caused delays in upgrading the database according to our planned maintenance schedule.

We have been working to upgrade our systems over the past year.

We will move ahead with planned migration and systems upgrades as soon as we fully restore service.

Q: How did you restore service?

We restored service using a redundant, secondary backup system and other sources. That data allowed us to begin to re-connect posts to the affected portion of the system and synchronize biometric data. This system is running on newer hardware, and has a synchronized standby system in a different Department data center.

In parallel, we are continuing to restore data from backups and overseas post databases. This process is ongoing.

Q: Do you know whether this is equipment that was acquired directly by the State Department, or was this acquired through a third-party contractor?

The equipment was acquired by the Department of State.

Q: How many people were affected by this outage?

During the past two weeks, consular sections have continued to interview travelers who applied June 8 or earlier. Those posts reconnected to our system are now issuing visas for those applicants.

Q: How are cases being prioritized?

We continue to facilitate urgent cases for those individuals who need to travel imminently, and will continue to do so until the systems are normal.

We apologize to travelers and recognize that this has caused hardship to some individuals waiting for visas as well as families and employers.

Q: What about the foreign agricultural workers (H2A visa holders?)

More than 2,500 temporary or seasonal workers have been issued new visas in Mexico since last week.

We will continue to prioritize H-2 applicants as our systems return to normal, and issue as many approved cases as possible. However, we will not be able to process these as quickly as we typically do until our systems are functioning normally. We continue to ask that any employers with urgent needs contact the post which is processing their applicants and we will do everything we can to facilitate the cases.

We are no longer asking CBP to provide Port of Entry waivers, as we have now begun issuing visas at border posts.

Visa applicants, including agricultural workers, who have not received a visa should not report to the border. Please contact the nearest embassy or consulate.

Read more here.

CCD Visa Update: 22 Posts Accounting For About Half of the Global Visa Volume Now Reconnected

Posted: 1:23 am  EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

On June 19, the Capital Press covering agriculture news in the western states reported that after a nine-day delay, the State Department began issuing visas again for Mexican farmworkers stranded at the U.S.-Mexican border wanting to head north for jobs. Visas were reportedly issued June 17 for a majority of some 200 people headed to Washington state to work in cherry harvest and other tree fruit work, according to Dan Fazio, director of the Washington Farm Labor Association in Olympia. The WAFLA posted its request for waiver of the visa requirement online here (pdf).

The update on the CCD glitch late last week said that 750 seasonal workers were issued visas as well as 1500 individuals with humanitarian cases. We asked CA about this since even us found this a tad confusing.  The earlier announcement said that the hardware glitch made it impossible to issue visas at this time. We requested confirmation that the seasonal workers and humanitarian cases who were “issued visas” were actually issued visa waivers. And if that was not the case, how was it possible to issue visas to these applicants if the hardware issues had yet to be fixed?

The Bureau of Consular Affairs has posted a June 22 update on the Consular Consolidated Database problems.  The update below does not specifically answer our questions but it indicates some development.  Note that the 22 posts are not identified in the update (which appears frustrating for consular clients venting on Facebook).  Visa applications affected are those submitted  on or after June 9.  With average global applications at 50,000 a day, this latest CCD glitch potentially affected about 450,000 visa applicants worldwide from June 9 – June 19. Whoa! Is that like $72 million in the last two weeks alone?

Here are the top 10 NIV issuance posts from FY2014. So these ten posts presumably have already been reconnected. We’re missing the next 12 posts that are also back online; hopefully, all 172 posts will be back online before too long.

Top Nonimmigrant (Temporary) Visa Issuance Posts | FY2014

Top Nonimmigrant (Temporary) Visa Issuance Posts | FY2014 (click for larger view)

Update below via travel.state.gov:

Visa Systems Issues

  • The Bureau of Consular Affairs is in the process of resolving technical problems with our visa systems. Though some progress has been made, biometric data processing has not been fully restored.
  • Our team continues to work on this 24/7 to restore the systems to full functionality.
  • As of noon today, 22 posts have been reconnected, representing about half of the global nonimmigrant visa volume.
  • We will continue to bring additional posts online until connectivity with all posts is restored.
  • Last week, nearly 1,250 temporary or seasonal workers who had been issued visas in the past were issued new visas in Mexico, and we have issued more than 3,000 visas globally for urgent and humanitarian travel.
  • We deeply regret the inconvenience to travelers and recognize the hardship to those waiting for visas, and in some cases, their family members or employers in the United States.
  • We continue to post updates to our website, travel.state.gov.

Q: How long before you restore full system functionality?

  • Public and private sector experts are making progress in correcting the visa problem, and we are striving to have the system fully reconnected sometime this week.
  • As of noon today, 22 posts have been reconnected, representing about half of the global nonimmigrant visa volume. All our servers appear to be stable. There is a large backlog of cases to clear, but we have already made good progress. We will continue to bring additional posts online until connectivity with all posts is restored.
  • Overseas and domestic passports are being issued.

Q: How many travelers are affected by this outage?

  • Most posts were able to handle visa interviews and some visa printing as usual through the end of last week. This week, many posts have rescheduled visa appointments.
  • We handle an average of 50,000 applications daily worldwide. Many applicants do not have immediate travel plans, and will receive visas in time for planned trips. We are prioritizing urgent medical and other humanitarian cases.

Q: Once operational, how will cases be prioritized?

  • We are already prioritizing urgent humanitarian cases and temporary agricultural workers. Once the systems are fully operational, we will work as quickly as possible to clear the backlog of pending visa cases.
  • We apologize to travelers and recognize that this has caused hardship to some individuals waiting for visas.

Q: What about domestic passports?

  • Domestic passport operations are functioning, with some processing delays. These technical problems have affected the intake of some mailed applications and same-day service at our passport agencies; however, we continue to issue routine and expedited passports to U.S. citizens for all overseas travel needs.

Q: What about overseas passports?

  • Overseas passport applications are being processed. There have been delays in some cases, but posts overseas are able to issue emergency passports in urgent cases.

Q: What about the foreign agricultural workers (H2A visa holders?)

  • We issued nearly 1,250 H-2 visas for agricultural and temporary workers last week. These are applicants whose biometric data was captured before the systems went down.
  • Additionally over the weekend we worked with DHS/CBP to facilitate the entry of more than 250 H-2 workers to travel to their places of employment in the United States.
  • CBP has agreed to give cases processed through the U.S. Department of State due consideration. However, no visa applicant, including agricultural workers, should report to the border without first having applied for a visa and having been processed through an embassy or consulate.
  • Based on this weekend’s systems progress, we are currently rescheduling more than 1,500 H-2 visa applicants who were unable to be interviewed last week because of systems problems.

Related posts:

State Dept’s Overseas Passport and Visa Systems Hit By Glitch Again, Suspends Issuance.
State Dept’s Consular Database Problems Dash Plans, Dreams … Also Cost Up to $1M/Day in California

State Dept’s Consular Database Problems Dash Plans, Dreams … Also Cost Up to $1M/Day in California

Posted: 4:46 pm  EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

On June 12, we posted about the technical problems with the State Department’s overseas passport and visa systems.  Passport applications accepted overseas on or after May 26, 2015 are affected but emergency passports are available.  A hardware failure on June 9 halted the flow of biometric clearance requests from posts to the State Department’s Consular Consolidated Database (CCD). Individuals who submitted online applications or were interviewed for visas on or after June 9 are affected and are asked to reschedule appointments . No emergency visas available. See State Dept’s Overseas Passport and Visa Systems Hit By Glitch Again, Suspends Issuance.

On June 15, the Bureau of Consular Affairs posted the following update on its Facebook page but not on its travel.state.gov news page:

We continue to experience technical problems with our visa systems. This issue is not specific to any particular country or visa category. We apologize for the inconvenience and we are working around the clock to correct the problem. Currently, we are unable to print most immigrant and nonimmigrant visas approved after June 8, 2015. In addition, U.S. embassies and consulates are unable to process new applications submitted on or after June 9, 2015. If you have a visa interview appointment scheduled for June 14-20, 2015, and you submitted your DS-160 online application **after June 9, 2015,** you should reschedule your appointment. If you submitted your DS-160 online application prior to June 9, 2015, you should plan to attend your scheduled visa interview appointment. Our embassies and consulates will be posting location-specific information on their websites, so please check the website of the location where you applied for your visa for more information. 

The technical issues also affected the Department of State’s ability to adjudicate applications for U.S. passports accepted overseas between May 26 and June 14, 2015. If you applied for a U.S. passport overseas during this time frame and have travel plans within the next 10 business days, please consider requesting an emergency passport at the U.S. embassy or consulate at which you originally applied. Information about how to apply for an emergency passport is available on the embassy/consulate website.

The previous time the CCD crashed big time was last summer (see State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience).  It could just be a coincidence (or not!) but the crash has now happened twice during the peak travel season. During the meltdown last summer, CA said that CCD was going to have an upgrade at the end of 2014. It also  said at that time that the upgrade plan included two redundant systems. If this glitch started on May 26th, we’re approaching the three week-mark. And so far, those redundant systems are missing in action.

The Bureau of Consular Affairs on its FAQ states that “This is not the same issue as last year.” But we learned from an unofficial source that “All line officers know that last summer’s CCD glitch was never completely fixed.” 

So, which is it?

On June 15, the Wall Street Journal reported that the CCD glitch has left agricultural workers stranded at the border just as the summer harvest gets under way. Jason Resnick, the general counsel for the Western Growers Association, which represents farmers in California, Arizona and Colorado calls this glitch, “a crisis.” Apparently, more than 1,000 workers who expected H-2A agricultural visas are stuck on the Mexican side of the border, where motels are overflowing.

“The workers are overdue to start harvesting berries and other crops on U.S. farms. Mr. Resnick estimated that California agriculture, already stressed by drought, is losing $500,000 to $1 million for each day of delay.”

.

The State Department’s consular operation is an enormous one with many parts and affects a large number of travelers.  The State Department issued 9,932,480 nonimmigrant/temporary visas in FY2014. It issued 467,370 immigrant/permanent visas in FY2014. During the same period, domestic and overseas passport offices issued 14,087,341 U.S. passports (including 1,463,191 passport cards).

A small fraction of those frustrated travelers have taken to Facebook to connect with Consular Affairs.

One who is stuck in Canada and could not return to her 14th month old baby writes:

Hi, do you have any timeline to fix the issues? I live in Boston, US & visited Vancouver, BC consulate on June 10th for my visa interview. visa officer told me that your visa is approved and you will get your passport back in 3 days. However, since June 10th, there is no update on my visa. I also inquired with Vancouver Consulate and they are ready to give me passport back without visa. As a matter of fact, I can’t enter into US legally until and unless I got printed Visa. My 14th month baby is waiting in Boston,MA and I got stuck here. Can’t do anything.

Here’s one waiting to be reunited with a loved one after a long wait:

Waiting is excruciating my husband was approved on june 10th and my mother has passed away. We need him home please get this fixed our application has been in since 2013.

Somebody who is pregnant, stuck in Mexico writes:

I am currently 8 months pregnant and have been waiting for my TD visa renewal since mid May. Since I will soon have travel restrictions to fly back home, does this qualify as an urgent humanitarian situation where I should contact the embassy in Mexico directly?

One who missed grandma’s funeral makes a plea:”

Can you tell us a estimated time??? My case was expedited and I miss my Grandmother funeral. Me and my wife were supposed to travel yesterday. Please get this fixed.please

A family stuck in Mexico:

Do you have an ETA in order for ys to plan accordingly? I had my appointment on friday june 12th and I am stuck in Mexico (H1B renewal) without passport and without the ability to get back to work in Boston. Flight fees, hotel fees and a family of 4 that needs to get back to Boston.

Some people missing a convention:

My wife and I have a flight to ny tom. Our visa were approved on 9th. We are part of 100+ group attending a convention. Do we expect to get our passports with visas today?

Somebody stuck in Guatemala, fears loss of a job:

All my documents were in order and approved June 1st. my husband and I are in a dire situation stuck in Guatemala. I’m at risk of losing my job if I don’t return to USA.

A frustrated somebody who calls out other technical issues:

There’s always something wrong. There are already technical issues with payment of IV fees, DS260 and DS261. This system needs to be revamped. What’s the government doing about this?!

A Romanian group who worked and saved to attend the Genius Olympiad:

We are desperate. We have a plane ticker for tomorrow and we were supposed to go to an international competition (Genius Olympiad) in Oswego, NYC. Apart from the part that we lost thousands of dollars, our hopes got crashed because we worked for a year at our projects and invested a lot of time and monney… For… Nothing?!?!?! How come you have no plan B for solving this issue? We tried making an appointment more than one month ago and they said on the 9th on june will be our interview, we said it s too late for us but they said that the visa will be delivered within 2 days maximum. And here we are 5 days later with no visas, with crashed hopes, tons of lost monney, wasted time… Should i go on??

#

State Dept’s Overseas Passport and Visa Systems Hit By Glitch Again, Suspends Issuance

Posted: 11:09 pm EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

Somebody sent us a note on June 11 asking, “Do you think the Chinese hackers could fix the Consular Consolidated Database?” Fix, how, we asked the white, empty space of the burn bag email.

Today, this pops up:

.

Here is the information provided by the State Department to the public:

Passport/Visa Systems Errors

  • The Bureau of Consular Affairs is currently experiencing technical problems with our overseas passport and visa systems.
  • This issue is not specific to any particular country, citizenship document, or visa category.
  • The Consular Consolidated Database (CCD) problems we are experiencing are not the same challenges we overcame last summer. We are working urgently to correct the problems and restore our system to full operational status as soon as possible.
  • We apologize to applicants who are experiencing delays or are unable to obtain a passport overseas, Consular Report of Birth Abroad, or U.S. visa at this time. Domestic passport issuances are not affected at this time. We are able to issue emergency passports to U.S. citizens overseas for urgent travel.
  • We are seeking to assist non-immigrant visa applicants with urgent humanitarian travel. Travelers with an urgent humanitarian need for travel should contact their nearest U.S. embassy or consulate.
  • We are aware of pending overseas adoption cases, including in China. We are prioritizing these cases and seek to issue these visas with few delays.
  • We regret the inconvenience to travelers, and remain committed to facilitating legitimate travel while protecting our borders. We are working urgently to correct the problem and expect our system to be fully operational again soon.
  • We will post updates to Travel.State.Gov as more information becomes available.

How is this affecting consular operations?

      Passports

  • Passport applications accepted overseas on or after May 26, 2015 are affected. If you applied for a U.S. passport during this time frame and have travel plans within the next 10 business days, please consider requesting an emergency passport at the U.S. embassy or consulate at which you originally applied. Information about how to apply for an emergency passport is available on the website of the nearest U.S. embassy or consulate.

      Visas

  • A hardware failure on June 9 halted the flow of biometric clearance requests from posts to the Consular Consolidated Database (CCD). Individuals who submitted online applications or were interviewed for visas on or after June 9 may experience a delay in the processing.
  • The systems in place to perform required national security checks before we issue visas are experiencing technical difficulties. As a result, we are unable to print visas, regular passports overseas, and other travel documents.
  • We cannot bypass the legal requirements necessary to screen visa applicants before we issue visas for travel.
  • As a result, there is a backlog of visas waiting to be processed. We are working as quickly as possible to resolve the issue and to clear the backlog.
  • The technical issues we are experiencing have disrupted or prevented some of the Department’s primary data-share partners from accessing visa records.

The public notice notes that visas cannot be printed without using the CCD system as security measures prevent consular officers from printing a passport, report of birth abroad, or visa until the case completes the required national security checks.visas

On the CA Q&A whether this was a malicious action or hack, the public response only says that the State Department is “working urgently to correct the problem and expect the system to be fully operational again soon.”  There is currently no available timeline on when full system functionality may be restored.

Read the full notice here.

We should note that the person in charge of the CA Bureau’s response the last time the CCD had a meltdown was  Greg Ambrose, a career IT official who was the chief of consular systems and technology (State/CA/CST). FCW previously reported this:

He has been working on a modernization project at State that involves taking the Consular Consolidated Database, a massive system of 12 databases used to process passport and visa applications, from Windows 2003 to Linux. He is also moving the data warehouse to the more powerful Oracle 11g platform. The goal is to give the stovepiped legacy systems a single look and feel.

Not this time around.

.

.

Citing internal State Department email, FCW says that Mr. Ambrose is scheduled to leave his CST job on June 11.  As of today, Kenneth Reynolds, Ambrose’s deputy reportedly replaced him on an acting capacity.

#

Related posts:

 

 

State Dept Re-attached to the Internet, and About Those “Unrelated” Embassy Outages

— Domani Spero
[twitter-follow screen_name=’Diplopundit’ ]

 

A few hours ago, we posted this: State Dept Spox on outages at embassies: “separate”, “unconnected”, “unrelated” — wowie zowie!

It looks like the State Department was re-attached to the Internet sometime this morning. Although as of this writing, go.state.gov is still down for “temporary maintenance.”

Screen Shot 2014-11

Screen capture of http://go.state.gov, still current as of 11/19/2014

Here’s what we’ve learned about the embassy outages:

The Consular Consolidated Database is apparently unaffected, as are visa and passport services.

EXCEPT that Consular Sections were unable to accept credit card payments because those are connected to the Internet, which was unavailable from the State Department’s OpenNet.

Here’s how OpenNet is described in the FAM:

OpenNet is a physical and logical Internet Protocol (IP)-based global network that links the Department of State’s Local Area Networks (LANs) domestically and abroad. The physical aspect of the network uses DTS circuits for posts abroad, FTS-2001-provided circuits, leased lines, and dial-up public switch networks. This includes interconnected hubs, routers, bridges, switches, and cables. The logical aspect of the network uses Integrated Enterprise Management System (NMS) and TCP/IP software, and other operational network applications. OpenNet is a Sensitive But Unclassified (SBU) network, which supports e-mail and data applications.

We understand that the American Citizen Services (ACS) Units, in particular, were not able to process payments by credit cards. Since the Internet connection issue had been reportedly resolved earlier today, we hope that this has resolved itself, too.

As to visa services, those are connected to the Global Support Strategy (GSS) contract, and 99% of fees would have been collected through the GSS contractor, not at post.

EXCEPT that most GSS contractors do scheduling via their own 3rd party websites, which would not be able to be accessed from OpenNet. If visa scheduling had delays, that would be because posts had to find a non-OpenNet Internet connection to update scheduling slots, as necessary.

A note on the GSS:  The GSS contracts provide support services for nonimmigrant and immigrant visa operations at United States consulates and embassies abroad, including but not limited to public inquiry services, appointment services, fee collection services, biometric enrollment services, document delivery services and data collection services.

So when the State Department spox said that these outages were not connected and were unrelated, well —

Congratulations! You sound nice at the podium but what the heck were you talking about?

* * *

Oops! What’s this? Updated at 1552 PST Nov 19:

Screen Shot 2014-11-19 at 3.44.20 PM

* * *

State Department’s “Technical Difficulties” Continue Worldwide, So What About the CCD?

— Domani Spero
[twitter-follow screen_name=’Diplopundit’ ]

 

The “technical difficulties” at the State Department continue today.  State Department spokesman Jeff Rathke told Yahoo News that  the State Department is still investigating who — or what — launched the attack saying, “I don’t have anything to share at this point on the origins of the intrusion.”

Rathke said the attack only hit unclassified email systems at the State Department — and not business databases that contain information about Americans or, for example, foreign visa applicants. Although the temporary shutdown was previously scheduled, “in this case, the response to this specific incident needed to be more comprehensive than our regular updates.

Congress is apparently interested on what’s going on.

Meanwhile, the Department’s mobile site go.state.gov remains down, and the “technical difficulties” now include, according to tweets from overseas posts, not just inability to use email  but also inability to accept credit card payment for visa and passport services, and unusable contact forms for visa and passport inquiries.


US Embassy Albania


US Embassy London

 

 

U.S. Embassy Manila

U.S. Embassy Beirut

 

US Embassy Turkey

U.S. Embassy Moscow

 

U.S. Embassy Madrid

* * *

Below is the template of the notice used today:

U.S. embassies and consulates are currently experiencing technical difficulties that may result in delays in visa processing and receiving and sending communications. Additionally, applicants who have interviews for student and exchange visitor (F/M/J) visas scheduled for this week should bring proof of payment of the SEVIS fee. U.S. citizens may also experience delays in sending and receiving communications. U.S. citizens requiring emergency assistance should contact the Embassy [INSERT contact info].

 

We doubt if the State Department would have acknowledged this intrusion had the Associated Press not reported it on Sunday. On a related matter, we understand that Consular Affairs’ Consular Consolidated Database has been having problems “lately.”

Can somebody please ask CA if these ongoing problems are related to the technical difficulties from this past summer, or if this is related to the just known intrusion that brought down the email system and the GO site? We’re not terribly technical but curious — if a cyber intruder starts deleting data from the CCD, would anyone notice what’s missing?

* * *

 

 

 

 

 

 

 

 

Burn Bag: Consular Leadership Tenet #5: Something Seriously Wrong With the CCD. Communicate?

Via Burn Bag:

“Why is the CCD [Consular Consolidated Database] such a piece of trash lately and when is Senior CA [Consular Affairs] Management going to communicate honestly with the Field what the problem is?  Anyone with a brain can tell there is something seriously wrong with the system.”

giphy_daleks

by rhetthammersmithhorror.tumblr.com via giphy.com

 * * *