Tag Archives: CCD crash

CCD: Report Says Visa Processing Systems Pose Significant Challenges; Also Face User-Friendlessness?

November 4, 2015 By in Consular Work, State Department, Technology and Work, Visas Tags: , , , , , , , , ,

Posted: 12:02 am EDT
[twitter-follow screen_name=’Diplopundit’ ]

Via GAO

According to Commerce, international travelers contributed $220.6 billion to the economy and supported 1.1 million jobs in 2014. Processing visas for such travelers as efficiently and effectively as possible without compromising our national security is critical to maintaining a competitive and secure travel and tourism industry in the United States. Although State has historically struggled with the task of maintaining reasonable wait times for NIV interviews, it has undertaken a number of efforts in recent years that have yielded substantial progress in reducing such waits.

Significant projected increases in NIV demand coupled with consular hiring constraints and other challenges could hinder State’s ability to sustain this progress in the future—especially in countries where the demand for visas is expected to rise the highest. These challenges heighten the importance of systematically evaluating the cost and impact of the multiple measures State has taken to reduce interview wait times in recent years and leveraging that knowledge in future decision making. Without this, State’s ability to direct resources to those activities that offer the greatest likelihood of success is limited. Moreover, State’s future capacity to cope with rising NIV demand will be challenged by inefficiencies in its visa processing technology; consular officers and management officials at the posts we visited pointed to cumbersome user procedures and frequent system failures as enormous obstacles to efficient NIV processing. State’s Bureau of Consular Affairs recognizes these problems and plans a number of system enhancements; however, the bureau does not systematically collect input from consular officers to help guide and prioritize these planned upgrades. Without a systematic effort to gain the input of those who employ these systems on a daily basis, State cannot be assured that it is investing its resources in a way that will optimize the performance of these systems for current and future users.

giphy_daleks

Consular officers and managers at posts we visited identified current information technology (IT) systems as one of the most significant challenges to the efficient processing of NIVs. Consular officers in all 11 focus groups we conducted across the four posts we visited stated that problems with the Consular Consolidated Database (CCD) and the NIV system create significant obstacles for consular officers in the processing of NIVs.26 Specifically, consular officers and managers at posts stated that frequent NIV system outages and failures (where the system stops working) at individual posts, worldwide system outages of CCD, and IT systems that are not user friendly, negatively affected their ability to process NIVs.

— NIV system outages and failures at posts: Consular officers we spoke with in Beijing, Mexico City, New Delhi, and São Paulo explained that the NIV system regularly stops working. This results in a reduced number of adjudications (whether being performed at the interview window or, for an IWP applicant, at an officer’s desk) in a day. Notably, consular officers in 4 of the 11 focus groups reported having to stop work or re-adjudicate NIV applications as a result of these NIV system failures. In fact, during our visit to the U.S. Embassy in New Delhi in March 2015, a local NIV outage occurred, affecting consular officers’ ability to conduct adjudications. In January 2015, officers in Bogotá, Guadalajara, Monterrey, and Moscow—among the top 15 posts with the highest NIV applicant volume in 2014— experienced severe NIV performance issues—specifically an inability to perform background check queries against databases.

— Worldwide outages and operational issues of CCD: Since July 2014, two worldwide outages of CCD have impaired the ability of posts to process NIV applications. On June 9, 2015, an outage affected the ability of posts to run checks of biometric data, thus halting most visa printing along with other services offered at posts.27 According to State officials, the outage affected every post worldwide for 10 days. The system was gradually repaired, but it was not fully restored at all posts until June 29, 2015, exacerbating already increased NIV interview wait times at some posts during the summer high demand season.According to State notices, another significant outage of CCD occurred on July 20, 2014, slowing NIV processing worldwide until September 5, 2014, when CCD returned to full operational capacity.28 State estimated that from the start of operational issues on July 20 through late July, State issued approximately 220,000 NIVs globally— about half of the NIVs State anticipated issuing during that period. According to officials in State’s Bureau of Consular Affairs, Office of Consular Systems and Technology (CST), who are responsible for operating and maintaining CCD and the NIV system, consular officers were still able to collect NIV applicant information during that period; however, processing of applications was significantly delayed with an almost 2-week backlog of NIVs. In the U.S. Consulate in São Paulo, a consular management official reported that due to this outage, the post had a backlog of about 30,000 NIV applications, or approximately 9 days’ worth of NIV interviews during peak season. Consular officers in 8 out of the 11 focus groups we conducted identified a lengthy CCD outage as a challenge to the efficient processing of NIVs.

— IT systems are not user friendly: In 9 out of 11 focus groups, consular officers described the IT systems for NIV processing as not user friendly. Officers in our focus groups explained that some aspects of the system hinder their ability to quickly and efficiently process NIVs. These aspects include a lack of integration among the databases needed for NIV adjudications, the need for manual scanning of documentation provided by an applicant, and an absence of standard keyboard shortcuts29 across all IT applications that would allow users to quickly copy information when processing NIV applications for related applicants, to avoid having to enter data multiple times. Some consular officers in our focus groups stated that they could adjudicate more NIVs in a day if the IT systems were less cumbersome and more user friendly.

— Consular officers in Beijing and Mexico City and consular management at one post indicated that the NIV system appeared to be designed without consideration for the needs of a high volume post, which include efficiently processing a large number of applications per adjudicator each day. According to consular officers, the system is poor at handling today’s high levels of demand because it was originally designed in the mid- 1990s. Consular officers in São Paulo stated that under current IT systems and programs, the post may not be able to process larger volumes that State projects it will have in the future.

— State, recognizing the limits of its current consular IT systems, initiated the development of a new IT platform. State is developing a new system referred to as “ConsularOne,” to modernize 92 applications that include systems such as CCD and the NIV system. According to State, ConsularOne will be implemented in six phases, starting with passport renewal systems and, in phase five, capabilities associated with adjudicating and issuing visas (referred to as non-citizen services). However, CST officials have yet to formally commit to when the capabilities associated with non-citizen services are to be implemented. According to a preliminary CST schedule, the enhanced capabilities associated with processing NIVs are not scheduled for completion until October 2019. Given this timeline, according to State officials, enhancements to existing IT systems are necessary and are being planned.

State Does Not Systematically Obtain End User Input to Prioritize Improvement Efforts for Current IT Systems

Although consular officers and managers we spoke with identified CCD and the NIV system as one of the most significant challenges to the efficient processing of NIVs, State does not systematically measure end user (i.e., consular officers) satisfaction. We have previously reported that in order for IT organizations to be successful, they should measure the satisfaction of their users and take steps to improve it.30 The Software Engineering Institute’s IDEALSM model is a recognized approach for managing efforts to make system improvements.31 According to this model, user satisfaction should be collected and used to help guide improvement efforts through a written plan. With such an approach, IT improvement resources can be invested in a manner that provides optimal results.

Although State is in the process of upgrading and enhancing CCD and the NIV system, State officials told us that they do not systematically measure user satisfaction with their IT systems and do not have a written plan for improving satisfaction. According to CST officials, consular officers may voluntarily submit requests to CST for proposed IT system enhancements. Additionally, State officials noted that an IT stakeholder group comprising officials in State’s Bureau of Consular Affairs regularly meets to identify and prioritize IT resources and can convey end user concerns for the system.32 However, State has not collected comprehensive data regarding end user satisfaction and developed a plan to help guide its current improvement efforts. Furthermore, consular officers continued to express concerns with the functionality of the IT systems, and some officers noted that enhancements to date have not been sufficient to address the largest problems they encounter with the systems.

Given consular officers’ reliance on IT services provided by CST, as well as the feedback we received from focus groups, it is critical that State identify and implement feedback from end users in a disciplined and structured fashion for current and any future IT upgrades. Without a systematic approach to measure end user satisfaction, CST may not be able to adequately ensure that it is investing its resources on improvement efforts that will improve performance of its current and future IT systems for end users.

#

State Department’s Visa Systems Now Operational at 165 of 220 Posts Worldwide

June 26, 2015 By in Consular Work, Foreign Service, Functional Bureaus, Legacy, Lessons, Security, State Department, Technology and Work, U.S. Missions, Visas Tags: , , , , , , , ,

Posted: 1:56 am  EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

The State Department’s Consular Consolidated Database problems that affected travelers globally is is now back online at 165 of 220 visa issuance posts worldwide.  The latest update does not explain in details the cause of the glitch except to cite the hardware issue.  It also says that service was restored “using a redundant, secondary backup system and other sources.”  It does not explain what “other sources” mean but if it took at least 9 days to get that redundant, secondary back-up system to kick in, that’s not a very good system.

The Consular Affairs-issued FAQ asks how many people were affected by this outage? The answer it provides to this question is neither here nor there.  Folks, if you can’t answer your own question, please don’t include it.

According to travel.state.gov, the average visa applications processed every day worldwide is 50,000 x 9 days (June 9-19)=450,000 + 25,000 (half the average daily applications) x 4 days (June 22-25) = 100,000. Total number potentially affected 550,000.  Is that close enough?

The June 25 update says that if systems had been operating normally, posts would have issued approximately 540,000 visas since the outage started. Whoa! Help us out here. What kind of refusal/approval rates are we looking at here? That 540,000 figure is a little hinky because not all applicants who apply are issued visas. If it would have issued 540,000  visas, what would have been the total number of applicants?  Note that all of them must pay the visa fees. We estimate that the USG loss from this latest glitch is between $72 to $84 million (average daily applications globally x no. of days x $160 visa fee). Is that too low?

Meanwhile, StarrFMonline.com reported that the US Embassy in Accra, has “dismissed reports that it is ripping Ghanaians off by accepting visa fees in spite of the visa issuance imbroglio that has hit US embassies across the world.” The consular section chief  had to explain that “if anybody was refused a visa, that was because of the case and has nothing to do with our technical issues.”

*

On June 24, the Bureau of Consular Affairs reports that 50 posts, representing nearly 73 percent of its  nonimmigrant visa demand worldwide, are back online and issuing visas.  It also says that “posts overseas have issued more than 150,000 non-immigrant visas since June 9.” And that for context, if systems had been operating normally, posts would have issued approximately 450,000 visas during the June 9-23 timeframe.

On June 25, the Bureau of Consular Affairs reports that 165 posts, representing more than 85 percent of nonimmigrant visa demand worldwide, are now online and issuing visas.  The update says that if systems had been operating normally, posts would have issued approximately 540,000 visas since the outage started.

Via travel.state.gov, June 25 update:

Visa Systems Issues

  • The Bureau of Consular Affairs reports that 165 posts, representing more than 85 percent of our nonimmigrant visa demand worldwide, are now online and issuing visas. 

  • Posts overseas issued more than 82,000 visas on June 24. 

  • Posts overseas have issued more than 238,000 non-immigrant visas this week. For context, if systems had been operating normally, posts would have issued approximately 540,000 visas since the outage started. 

  • We will continue to bring additional posts online until connectivity with all posts is restored. All posts worldwide are now scheduling interviews with applicants, including with those who applied after the systems problems began on June 9.

  • We deeply regret the inconvenience to travelers who are waiting for visas, as well as their families and U.S. businesses that have been affected.

  • We continue to post updates to our website, travel.state.gov.

 

Q: Reports indicate that your backlog is 700,000 visas. Is this accurate?

No. While there is a large backlog of cases to clear, it never approached that level, and we have already made good progress issuing those visas. Many posts are working overtime this week and during the upcoming weekend, and we expect to eliminate the backlog in a week or less.


Q: How old is this equipment? And does the age of the equipment and the need to have so many repairs to the hardware mean that this equipment should have been replaced? Is this a funding issue at the base of it?

The hardware that impacted the biometrics system is several years old. The Department was working to move the biometrics system off of this hardware.

The operational requirements to keep this database running for domestic and overseas passport and visa issuances caused delays in upgrading the database according to our planned maintenance schedule.

We have been working to upgrade our systems over the past year.

We will move ahead with planned migration and systems upgrades as soon as we fully restore service.

Q: How did you restore service?

We restored service using a redundant, secondary backup system and other sources. That data allowed us to begin to re-connect posts to the affected portion of the system and synchronize biometric data. This system is running on newer hardware, and has a synchronized standby system in a different Department data center.

In parallel, we are continuing to restore data from backups and overseas post databases. This process is ongoing.

Q: Do you know whether this is equipment that was acquired directly by the State Department, or was this acquired through a third-party contractor?

The equipment was acquired by the Department of State.

Q: How many people were affected by this outage?

During the past two weeks, consular sections have continued to interview travelers who applied June 8 or earlier. Those posts reconnected to our system are now issuing visas for those applicants.

Q: How are cases being prioritized?

We continue to facilitate urgent cases for those individuals who need to travel imminently, and will continue to do so until the systems are normal.

We apologize to travelers and recognize that this has caused hardship to some individuals waiting for visas as well as families and employers.

Q: What about the foreign agricultural workers (H2A visa holders?)

More than 2,500 temporary or seasonal workers have been issued new visas in Mexico since last week.

We will continue to prioritize H-2 applicants as our systems return to normal, and issue as many approved cases as possible. However, we will not be able to process these as quickly as we typically do until our systems are functioning normally. We continue to ask that any employers with urgent needs contact the post which is processing their applicants and we will do everything we can to facilitate the cases.

We are no longer asking CBP to provide Port of Entry waivers, as we have now begun issuing visas at border posts.

Visa applicants, including agricultural workers, who have not received a visa should not report to the border. Please contact the nearest embassy or consulate.

Read more here.

CCD Visa Update: 22 Posts Accounting For About Half of the Global Visa Volume Now Reconnected

June 23, 2015 By in Bugs, Consular Work, Foreign Service, Functional Bureaus, Govt Documents, Security, State Department, Technology and Work, U.S. Missions, Visas Tags: , , , ,

Posted: 1:23 am  EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

On June 19, the Capital Press covering agriculture news in the western states reported that after a nine-day delay, the State Department began issuing visas again for Mexican farmworkers stranded at the U.S.-Mexican border wanting to head north for jobs. Visas were reportedly issued June 17 for a majority of some 200 people headed to Washington state to work in cherry harvest and other tree fruit work, according to Dan Fazio, director of the Washington Farm Labor Association in Olympia. The WAFLA posted its request for waiver of the visa requirement online here (pdf).

The update on the CCD glitch late last week said that 750 seasonal workers were issued visas as well as 1500 individuals with humanitarian cases. We asked CA about this since even us found this a tad confusing.  The earlier announcement said that the hardware glitch made it impossible to issue visas at this time. We requested confirmation that the seasonal workers and humanitarian cases who were “issued visas” were actually issued visa waivers. And if that was not the case, how was it possible to issue visas to these applicants if the hardware issues had yet to be fixed?

The Bureau of Consular Affairs has posted a June 22 update on the Consular Consolidated Database problems.  The update below does not specifically answer our questions but it indicates some development.  Note that the 22 posts are not identified in the update (which appears frustrating for consular clients venting on Facebook).  Visa applications affected are those submitted  on or after June 9.  With average global applications at 50,000 a day, this latest CCD glitch potentially affected about 450,000 visa applicants worldwide from June 9 – June 19. Whoa! Is that like $72 million in the last two weeks alone?

Here are the top 10 NIV issuance posts from FY2014. So these ten posts presumably have already been reconnected. We’re missing the next 12 posts that are also back online; hopefully, all 172 posts will be back online before too long.

Top Nonimmigrant (Temporary) Visa Issuance Posts | FY2014

Top Nonimmigrant (Temporary) Visa Issuance Posts | FY2014 (click for larger view)

Update below via travel.state.gov:

Visa Systems Issues

  • The Bureau of Consular Affairs is in the process of resolving technical problems with our visa systems. Though some progress has been made, biometric data processing has not been fully restored.
  • Our team continues to work on this 24/7 to restore the systems to full functionality.
  • As of noon today, 22 posts have been reconnected, representing about half of the global nonimmigrant visa volume.
  • We will continue to bring additional posts online until connectivity with all posts is restored.
  • Last week, nearly 1,250 temporary or seasonal workers who had been issued visas in the past were issued new visas in Mexico, and we have issued more than 3,000 visas globally for urgent and humanitarian travel.
  • We deeply regret the inconvenience to travelers and recognize the hardship to those waiting for visas, and in some cases, their family members or employers in the United States.
  • We continue to post updates to our website, travel.state.gov.

Q: How long before you restore full system functionality?

  • Public and private sector experts are making progress in correcting the visa problem, and we are striving to have the system fully reconnected sometime this week.
  • As of noon today, 22 posts have been reconnected, representing about half of the global nonimmigrant visa volume. All our servers appear to be stable. There is a large backlog of cases to clear, but we have already made good progress. We will continue to bring additional posts online until connectivity with all posts is restored.
  • Overseas and domestic passports are being issued.

Q: How many travelers are affected by this outage?

  • Most posts were able to handle visa interviews and some visa printing as usual through the end of last week. This week, many posts have rescheduled visa appointments.
  • We handle an average of 50,000 applications daily worldwide. Many applicants do not have immediate travel plans, and will receive visas in time for planned trips. We are prioritizing urgent medical and other humanitarian cases.

Q: Once operational, how will cases be prioritized?

  • We are already prioritizing urgent humanitarian cases and temporary agricultural workers. Once the systems are fully operational, we will work as quickly as possible to clear the backlog of pending visa cases.
  • We apologize to travelers and recognize that this has caused hardship to some individuals waiting for visas.

Q: What about domestic passports?

  • Domestic passport operations are functioning, with some processing delays. These technical problems have affected the intake of some mailed applications and same-day service at our passport agencies; however, we continue to issue routine and expedited passports to U.S. citizens for all overseas travel needs.

Q: What about overseas passports?

  • Overseas passport applications are being processed. There have been delays in some cases, but posts overseas are able to issue emergency passports in urgent cases.

Q: What about the foreign agricultural workers (H2A visa holders?)

  • We issued nearly 1,250 H-2 visas for agricultural and temporary workers last week. These are applicants whose biometric data was captured before the systems went down.
  • Additionally over the weekend we worked with DHS/CBP to facilitate the entry of more than 250 H-2 workers to travel to their places of employment in the United States.
  • CBP has agreed to give cases processed through the U.S. Department of State due consideration. However, no visa applicant, including agricultural workers, should report to the border without first having applied for a visa and having been processed through an embassy or consulate.
  • Based on this weekend’s systems progress, we are currently rescheduling more than 1,500 H-2 visa applicants who were unable to be interviewed last week because of systems problems.

Related posts:

State Dept’s Overseas Passport and Visa Systems Hit By Glitch Again, Suspends Issuance.
State Dept’s Consular Database Problems Dash Plans, Dreams … Also Cost Up to $1M/Day in California

State Dept’s Consular Database Problems Dash Plans, Dreams … Also Cost Up to $1M/Day in California

June 16, 2015 By in Consular Work, Foreign Service, Functional Bureaus, Security, Shutdown, Social Media, State Department, Technology and Work, Trends, Visas Tags: , , , , , , , ,

Posted: 4:46 pm  EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

On June 12, we posted about the technical problems with the State Department’s overseas passport and visa systems.  Passport applications accepted overseas on or after May 26, 2015 are affected but emergency passports are available.  A hardware failure on June 9 halted the flow of biometric clearance requests from posts to the State Department’s Consular Consolidated Database (CCD). Individuals who submitted online applications or were interviewed for visas on or after June 9 are affected and are asked to reschedule appointments . No emergency visas available. See State Dept’s Overseas Passport and Visa Systems Hit By Glitch Again, Suspends Issuance.

On June 15, the Bureau of Consular Affairs posted the following update on its Facebook page but not on its travel.state.gov news page:

We continue to experience technical problems with our visa systems. This issue is not specific to any particular country or visa category. We apologize for the inconvenience and we are working around the clock to correct the problem. Currently, we are unable to print most immigrant and nonimmigrant visas approved after June 8, 2015. In addition, U.S. embassies and consulates are unable to process new applications submitted on or after June 9, 2015. If you have a visa interview appointment scheduled for June 14-20, 2015, and you submitted your DS-160 online application **after June 9, 2015,** you should reschedule your appointment. If you submitted your DS-160 online application prior to June 9, 2015, you should plan to attend your scheduled visa interview appointment. Our embassies and consulates will be posting location-specific information on their websites, so please check the website of the location where you applied for your visa for more information. 

The technical issues also affected the Department of State’s ability to adjudicate applications for U.S. passports accepted overseas between May 26 and June 14, 2015. If you applied for a U.S. passport overseas during this time frame and have travel plans within the next 10 business days, please consider requesting an emergency passport at the U.S. embassy or consulate at which you originally applied. Information about how to apply for an emergency passport is available on the embassy/consulate website.

The previous time the CCD crashed big time was last summer (see State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience).  It could just be a coincidence (or not!) but the crash has now happened twice during the peak travel season. During the meltdown last summer, CA said that CCD was going to have an upgrade at the end of 2014. It also  said at that time that the upgrade plan included two redundant systems. If this glitch started on May 26th, we’re approaching the three week-mark. And so far, those redundant systems are missing in action.

The Bureau of Consular Affairs on its FAQ states that “This is not the same issue as last year.” But we learned from an unofficial source that “All line officers know that last summer’s CCD glitch was never completely fixed.” 

So, which is it?

On June 15, the Wall Street Journal reported that the CCD glitch has left agricultural workers stranded at the border just as the summer harvest gets under way. Jason Resnick, the general counsel for the Western Growers Association, which represents farmers in California, Arizona and Colorado calls this glitch, “a crisis.” Apparently, more than 1,000 workers who expected H-2A agricultural visas are stuck on the Mexican side of the border, where motels are overflowing.

“The workers are overdue to start harvesting berries and other crops on U.S. farms. Mr. Resnick estimated that California agriculture, already stressed by drought, is losing $500,000 to $1 million for each day of delay.”

.

The State Department’s consular operation is an enormous one with many parts and affects a large number of travelers.  The State Department issued 9,932,480 nonimmigrant/temporary visas in FY2014. It issued 467,370 immigrant/permanent visas in FY2014. During the same period, domestic and overseas passport offices issued 14,087,341 U.S. passports (including 1,463,191 passport cards).

A small fraction of those frustrated travelers have taken to Facebook to connect with Consular Affairs.

One who is stuck in Canada and could not return to her 14th month old baby writes:

Hi, do you have any timeline to fix the issues? I live in Boston, US & visited Vancouver, BC consulate on June 10th for my visa interview. visa officer told me that your visa is approved and you will get your passport back in 3 days. However, since June 10th, there is no update on my visa. I also inquired with Vancouver Consulate and they are ready to give me passport back without visa. As a matter of fact, I can’t enter into US legally until and unless I got printed Visa. My 14th month baby is waiting in Boston,MA and I got stuck here. Can’t do anything.

Here’s one waiting to be reunited with a loved one after a long wait:

Waiting is excruciating my husband was approved on june 10th and my mother has passed away. We need him home please get this fixed our application has been in since 2013.

Somebody who is pregnant, stuck in Mexico writes:

I am currently 8 months pregnant and have been waiting for my TD visa renewal since mid May. Since I will soon have travel restrictions to fly back home, does this qualify as an urgent humanitarian situation where I should contact the embassy in Mexico directly?

One who missed grandma’s funeral makes a plea:”

Can you tell us a estimated time??? My case was expedited and I miss my Grandmother funeral. Me and my wife were supposed to travel yesterday. Please get this fixed.please

A family stuck in Mexico:

Do you have an ETA in order for ys to plan accordingly? I had my appointment on friday june 12th and I am stuck in Mexico (H1B renewal) without passport and without the ability to get back to work in Boston. Flight fees, hotel fees and a family of 4 that needs to get back to Boston.

Some people missing a convention:

My wife and I have a flight to ny tom. Our visa were approved on 9th. We are part of 100+ group attending a convention. Do we expect to get our passports with visas today?

Somebody stuck in Guatemala, fears loss of a job:

All my documents were in order and approved June 1st. my husband and I are in a dire situation stuck in Guatemala. I’m at risk of losing my job if I don’t return to USA.

A frustrated somebody who calls out other technical issues:

There’s always something wrong. There are already technical issues with payment of IV fees, DS260 and DS261. This system needs to be revamped. What’s the government doing about this?!

A Romanian group who worked and saved to attend the Genius Olympiad:

We are desperate. We have a plane ticker for tomorrow and we were supposed to go to an international competition (Genius Olympiad) in Oswego, NYC. Apart from the part that we lost thousands of dollars, our hopes got crashed because we worked for a year at our projects and invested a lot of time and monney… For… Nothing?!?!?! How come you have no plan B for solving this issue? We tried making an appointment more than one month ago and they said on the 9th on june will be our interview, we said it s too late for us but they said that the visa will be delivered within 2 days maximum. And here we are 5 days later with no visas, with crashed hopes, tons of lost monney, wasted time… Should i go on??

#

Related posts:

 

State Dept’s Overseas Passport and Visa Systems Hit By Glitch Again, Suspends Issuance

June 12, 2015 By in Consular Work, Security, State Department, Technology and Work, U.S. Missions, Visas Tags: , , , , , , , , 1 Comment

Posted: 11:09 pm EDT
[twitter-follow screen_name=’Diplopundit’ ]

 

Somebody sent us a note on June 11 asking, “Do you think the Chinese hackers could fix the Consular Consolidated Database?” Fix, how, we asked the white, empty space of the burn bag email.

Today, this pops up:

.

Here is the information provided by the State Department to the public:

Passport/Visa Systems Errors

  • The Bureau of Consular Affairs is currently experiencing technical problems with our overseas passport and visa systems.
  • This issue is not specific to any particular country, citizenship document, or visa category.
  • The Consular Consolidated Database (CCD) problems we are experiencing are not the same challenges we overcame last summer. We are working urgently to correct the problems and restore our system to full operational status as soon as possible.
  • We apologize to applicants who are experiencing delays or are unable to obtain a passport overseas, Consular Report of Birth Abroad, or U.S. visa at this time. Domestic passport issuances are not affected at this time. We are able to issue emergency passports to U.S. citizens overseas for urgent travel.
  • We are seeking to assist non-immigrant visa applicants with urgent humanitarian travel. Travelers with an urgent humanitarian need for travel should contact their nearest U.S. embassy or consulate.
  • We are aware of pending overseas adoption cases, including in China. We are prioritizing these cases and seek to issue these visas with few delays.
  • We regret the inconvenience to travelers, and remain committed to facilitating legitimate travel while protecting our borders. We are working urgently to correct the problem and expect our system to be fully operational again soon.
  • We will post updates to Travel.State.Gov as more information becomes available.

How is this affecting consular operations?

      Passports

  • Passport applications accepted overseas on or after May 26, 2015 are affected. If you applied for a U.S. passport during this time frame and have travel plans within the next 10 business days, please consider requesting an emergency passport at the U.S. embassy or consulate at which you originally applied. Information about how to apply for an emergency passport is available on the website of the nearest U.S. embassy or consulate.

      Visas

  • A hardware failure on June 9 halted the flow of biometric clearance requests from posts to the Consular Consolidated Database (CCD). Individuals who submitted online applications or were interviewed for visas on or after June 9 may experience a delay in the processing.
  • The systems in place to perform required national security checks before we issue visas are experiencing technical difficulties. As a result, we are unable to print visas, regular passports overseas, and other travel documents.
  • We cannot bypass the legal requirements necessary to screen visa applicants before we issue visas for travel.
  • As a result, there is a backlog of visas waiting to be processed. We are working as quickly as possible to resolve the issue and to clear the backlog.
  • The technical issues we are experiencing have disrupted or prevented some of the Department’s primary data-share partners from accessing visa records.

The public notice notes that visas cannot be printed without using the CCD system as security measures prevent consular officers from printing a passport, report of birth abroad, or visa until the case completes the required national security checks.visas

On the CA Q&A whether this was a malicious action or hack, the public response only says that the State Department is “working urgently to correct the problem and expect the system to be fully operational again soon.”  There is currently no available timeline on when full system functionality may be restored.

Read the full notice here.

We should note that the person in charge of the CA Bureau’s response the last time the CCD had a meltdown was  Greg Ambrose, a career IT official who was the chief of consular systems and technology (State/CA/CST). FCW previously reported this:

He has been working on a modernization project at State that involves taking the Consular Consolidated Database, a massive system of 12 databases used to process passport and visa applications, from Windows 2003 to Linux. He is also moving the data warehouse to the more powerful Oracle 11g platform. The goal is to give the stovepiped legacy systems a single look and feel.

Not this time around.

.

.

Citing internal State Department email, FCW says that Mr. Ambrose is scheduled to leave his CST job on June 11.  As of today, Kenneth Reynolds, Ambrose’s deputy reportedly replaced him on an acting capacity.

#

Related posts:

 

 

State Dept/CBP Reportedly Announced Fix for Certain Applicants Ensnared By Visa Glitch

August 12, 2014 By in Consular Work, Federal Agencies, Functional Bureaus, State Department, Technology and Work, Visas Tags: , , , , , , ,

— Domani Spero
[twitter-follow screen_name=’Diplopundit’ ]

 

The State Department’s Consular Consolidated Database has been having performance issues since July 19th. We have written about it in this blog (see State Dept Answers FAQ on Ongoing Visa and Passport Database Performance Issues and  State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience).

Last week, Greenberg Traurig posted on The National Law Review that the State Department and the U.S. Customs and Border Protection (CBP) have reportedly announced a fix for certain visa applicants affected by the technical glitch.

“DOS and CBP will, on a case-by-case basis, waive nonimmigrant visa (H-1B, L-1, O-1, etc.) requirements for admission into the United States. In particular, applicants whose U.S. travel involves an “emergency” (i.e., humanitarian travel and life-and-death situations) or impacts U.S. national interests may request consideration for special travel permission.”

The post further states that if “emergency” travel is approved, the embassy or consulate will issue a transportation letter for presentation to common carriers to allow boarding of international U.S.-bound flights. (See DOS and CBP Announce Fix for Certain Visa Applicants Who Are Experiencing Consular Delays Due to Recent Technical Challenges).

This information is nowhere to be found on the State Department’s website or on the Visa Section of travel.state.gov nor the FB page of the Bureau of Consular Affairs. No such announcement is made available from the CBP website.

An  August 10 update from U.S.-China Visa Law Blog includes the following details:

A nonimmigrant visa applicant whose U.S. travel is urgent because it either involves an “emergency” or impacts U.S. national interests, may request consideration for special travel permission to the United States if their visa issuance is delayed as a result CCD systems problems. “Emergencies” in this instance include urgent humanitarian travel and life-and-death situations. Upcoming business engagements and U.S. employment needs are “not typically considered humanitarian emergencies and likely will not be considered as such in most cases.”

If approved jointly by the State Department and U.S. Customs and Border Protection (CBP), the consular post that accepted the visa application will release the traveler’s passport and will issue a transportation letter, which can be presented to the airlines to allow boarding of international U.S.-bound flights. Upon arrival to a U.S. port of entry and presentation of the transportation letter, CBP will waive the nonimmigrant visa requirement for admission.

Read more:  An Computer Crash Hobbles U.S. Visa, Passport Operations in China (Aug. 10 Update).

It is, of course, just a coincidence that the two sources noting the transportation letter fix are both law firms working on immigration, right? 😉  CA bureau’s FB page does not have an August 8 or August 10 update that includes this information. If there was an announcement, are we to understand that it was done on limited distribution with the State/CBP telling lawyers about this but not releasing this guidance to the general public?

We must confess that we’ve made a mistake of asking for clarification about this from the press office of the Bureau of Consular Affairs.  It turns out that some  folks there are unable to answer “yes” or “no” questions and are only able to provide cut and paste “on background” information for recycled details already publicly available.

Don’t get us wrong. It certainly is impressive cut and paste skills, but we won’t help them recycle the canned info and add to the glut.

* * *

 

 

Related articles

 

 

 

 

 

 

 

 

 

State Dept Answers FAQ on Ongoing Visa and Passport Database Performance Issues

July 31, 2014 By in Consular Work, Contractors, Foreign Service, Functional Bureaus, Social Media, Spectacular, State Department, Technology and Work, U.S. Missions, Visas Tags: , , , , , , , 2 Comments

— Domani Spero
[twitter-follow screen_name=’Diplopundit’ ]

 

Yesterday, we posted about the troubled Consular Consolidated Database (CCD) (see State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience).  During the Daily Press Briefing, yesterday, the State Department officially stated that it believed the root cause of the problem was “a combination of software optimization and hardware compatibility issues.” According to the deputy spokesperson, the servers are getting back online but that they are coming back in a queue and that fixes are not being done on a country-by-country basis. And by the way, it’s not just the peak summer travel season, there’s also the Africa Summit in D.C. next week.

“Obviously, there’s actually a huge crush right now because of the Africa Leaders Summit, so obviously that’s a huge priority for us to make sure everybody gets their visas for the Africa Leaders Summit. We do believe that a vast majority of the travelers who have applied for visas for the summit have been issued.”

CA’s FB folks have been regularly answering questions from angry complaints posted on its Facebook page and have announced that they will continue to monitor and respond to consular clients at 9:00 EDT tomorrow, Thursday, July 31.

Late yesterday, the Bureau of Consular Affairs also posted a new Frequently Asked Questions on Facebook and on its website (not easily accessible from the main visa page) concerning the CCD performance issues and the steps taken to address those issues. Perhaps the most surprising is that its back-up capability and redundancy built into the CCD were both affected killed by the upgrade that hobbled the system.  Something to look forward to by end of calendar year — CA is upgrading the CCD to a newer version of the Oracle commercial database software and that plan includes establishing two fully redundant systems. We are republishing the FAQ in full below.

Screen Shot 2014-07-31 at 12.07.05 AM

Information Regarding Ongoing Consular Consolidated Database (CCD) Performance Issues and Steps Taken| JULY 30, 2014

The Department of State continues to work to restore our visa system to full functionality.

We anticipate it will take weeks to resume full visa processing capacity.

We continue to prioritize immigrant visas, including adoption cases.  So far, we have been able to issue most cases with few delays.

Nearly all passports are currently being issued within our customer service standards, despite the system problems.

We are able to issue passports for emergency travel.

Frequently Asked Questions

Q:  What caused the system performance issues?  Hardware, software, or both?  Details?

On July 20, to improve overall system performance and address previous intermittent performance issues, we updated software as recommended. Our database began experiencing significant performance issues shortly after this maintenance was performed.

A root cause has not been identified at this time.  Current efforts are focused on bringing the system back to normal operations.  Once that has been accomplished, resources will be applied to determine the root cause.

Q:  What steps did we take to mitigate the performance issues?

Since July 20, our team has worked to restore operations to full capacity.  On July 23, the Consular Consolidated Database (CCD) was brought back online with limited capacity.

The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues.

We are incrementally increasing the number of processed cases as our systems will allow.

Q:  Has the Bureau of Consular Affairs experienced these types of outages in the past?

CA has experienced minor outages in the past, but never of this magnitude.  We have a plan in place to mitigate these occurrences in the future.

Q:  Is the software to blame?  Are contractors at fault?  Why was this allowed to happen?

We have been working to improve our services through upgrades while maintaining existing operations worldwide.  However, we are limited by outdated software and hardware.

Q:  Why did those steps not work?  What’s the next step?

We have not determined why the problems occurred.  We are working with our contractor and the software vendor to address the problems.

We are bringing additional servers online to increase capacity and response time.

Q: Why wasn’t there a back-up server?

There was back-up capability and redundancy built into the system.  However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system.

Q: What steps are being taken prevent this from happening again?

CA has a plan in place to upgrade the CCD to a newer version of the Oracle commercial database software by the end of the calendar year.  We are working to ensure the existing system will remain fully functional until the new database is up and running and thoroughly tested.  The plan includes establishing two fully redundant systems.

Q:  If CA is fee funded, why can’t it build a robust database that doesn’t fail?

The database has grown dramatically, in both quantity of data and functionality, and vastly improved border security.  In addition to checking names against databases, we review fingerprints and perform facial recognition.

We are working towards modernization of our software, hardware, and infrastructure.  Demand for our services outpaced our modernization efforts.

Consular Affairs has, and has had, a redundant system.  However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system.  This is one of the issues we are urgently addressing now.

Q:  What do I need to know if I’m a passport applicant?

Almost all passports are currently being issued within our customer service standards, despite the system problems.

We are able to issue passports for emergency travel.

Q:  What do I need to know if I’m a visa applicant?

Visa applicants they can expect delays as we process pending cases.  We remain able to quickly process emergency cases to completion.

We are working urgently to correct the problem to avoid further inconveniencing travelers.

We are posting updates to the visa page of travel.state.gov, and our embassies and consulates overseas are communicating with visa applicants.

In addition to communicating through our websites, e-mail, and letters, we are also reaching out to applicants via Facebook and other social media sites, such as Weibo, to relay the latest information.

Q:  Why hasn’t the Department been more forthcoming until now?

We have experienced CCD outages in the past, but they have never disrupted our ability to perform consular tasks at this magnitude.

We informed the public as soon as it was apparent there was not a quick fix to bring the CCD back to normal operating capacity, and are briefing Congressional staffers regularly.

Q:  What is the outlook for Non-immigrant visas?  When do we estimate the backlog will be processed?

That will depend on a number of factors.  Current efforts are focused on bringing the system back to normal operations.

We must also continue processing new requests.  We are committed to reducing the number of pending visa cases as quickly as possible, but we want applicants to know that we will continue to be operating at less than optimal efficiency until the system is restored to full functionality.

Q:  Is the Department going to reimburse applicants who missed flights/canceled weddings/missed funerals?

We sincerely regret any delays, inconvenience, or expense that applicants have may have incurred due to the CCD performance issues.

While it might be of little solace to those who have experienced hardship, we are always very careful to tell travelers NOT to make travel plans until they have a visa in hand.  Even when the CCD is operating normally, there may be delays in printing visas.

The Department does not have the authority to reimburse applicants for personal travel, nor do we include these costs when calculating our fees.  The Department cannot refund visa fees except in the specific circumstances set out in our regulations.

Q:  What impact will this have on SIVs?

We have the highest respect for the men and women who take enormous risks in supporting our military and civilian personnel.  We are committed to helping those who have helped us.  While issuances of Special Immigrant Visas (SIVs) to Afghans and Iraqis have been impacted, as have visa issuances around the world, SIV processing continues and remains a high priority.

Q: How is this impacting student visas?  They are scheduled to start the fall semester soon.

We are committed to issuing visas to all qualified students and exchange visitors.  Issuance of student and exchange visitor visas has been impacted in the past few days, but visa processing continues.

We understand the importance to international students and exchange visitors, their families, and their U.S. host institutions of timely visa issuance in order to facilitate travel and to ensure all students and exchange visitors may begin their programs on time.

Q: What about situations where the student won’t arrive to school on time?

Students should contact their educational institution’s Designated School Official (F and M visas) or designated U.S. sponsor’s Responsible Officer (J visas) and discuss with them what arrangements they can provide for you to begin your program after the start date on your Form I-20 (F and M visas) or Form DS 2019 (J visas), should such a circumstance become necessary.

Q:  Will this have any impact on the Diversity Visa program in September?

While issuances of all immigrant visas, including diversity visas, have been impacted in the past few days, IV processing continues and remains a high priority.  The Department expects to have used all numbers for DV-2014 when the program year ends on September 30, 2014.

Q:  What impact do we anticipate this will have on the U.S. economy?

Tourism and students have a major impact on our economy.  Last year, it was estimated that international visitors spent $180.7 billion and supported 1.3 million American jobs.  International students contribute $24.7 billion to the U.S. economy through their expenditures on tuition and living expenses, according to the Department of Commerce.

We recognize the significant impact that international travel and tourism has on the U.S. economy, and are taking all possible steps to ensure that the economic impact is minimal.

People traveling under the Visa Waiver Program are not affected at all; nor are those whose previously-issued visas remain valid.

We routinely advise applicants needing new visas to make appointments well in advance of their planned travel, and not to book their travel until they have their printed visas in hand.

The original post is available here.  If CA is reading this, it would be helpful if a link to the FAQ is posted on the main visa page of travel.state.gov and in the News section.  We were only able to find the FAQ from a link provided in Facebook and not from browsing around the travel.state.gov website.

 

 

 

Related articles