– Domani Spero
Yesterday, we posted about the troubled Consular Consolidated Database (CCD) (see State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience). During the Daily Press Briefing, yesterday, the State Department officially stated that it believed the root cause of the problem was “a combination of software optimization and hardware compatibility issues.” According to the deputy spokesperson, the servers are getting back online but that they are coming back in a queue and that fixes are not being done on a country-by-country basis. And by the way, it’s not just the peak summer travel season, there’s also the Africa Summit in D.C. next week.
“Obviously, there’s actually a huge crush right now because of the Africa Leaders Summit, so obviously that’s a huge priority for us to make sure everybody gets their visas for the Africa Leaders Summit. We do believe that a vast majority of the travelers who have applied for visas for the summit have been issued.”
CA’s FB folks have been regularly answering questions from angry complaints posted on its Facebook page and have announced that they will continue to monitor and respond to consular clients at 9:00 EDT tomorrow, Thursday, July 31.
Late yesterday, the Bureau of Consular Affairs also posted a new Frequently Asked Questions on Facebook and on its website (not easily accessible from the main visa page) concerning the CCD performance issues and the steps taken to address those issues. Perhaps the most surprising is that its back-up capability and redundancy built into the CCD were both
affected killed by the upgrade that hobbled the system. Something to look forward to by end of calendar year — CA is upgrading the CCD to a newer version of the Oracle commercial database software and that plan includes establishing two fully redundant systems. We are republishing the FAQ in full below.
Information Regarding Ongoing Consular Consolidated Database (CCD) Performance Issues and Steps Taken| JULY 30, 2014
The Department of State continues to work to restore our visa system to full functionality.
We anticipate it will take weeks to resume full visa processing capacity.
We continue to prioritize immigrant visas, including adoption cases. So far, we have been able to issue most cases with few delays.
Nearly all passports are currently being issued within our customer service standards, despite the system problems.
We are able to issue passports for emergency travel.
Frequently Asked Questions
Q: What caused the system performance issues? Hardware, software, or both? Details?
On July 20, to improve overall system performance and address previous intermittent performance issues, we updated software as recommended. Our database began experiencing significant performance issues shortly after this maintenance was performed.
A root cause has not been identified at this time. Current efforts are focused on bringing the system back to normal operations. Once that has been accomplished, resources will be applied to determine the root cause.
Q: What steps did we take to mitigate the performance issues?
Since July 20, our team has worked to restore operations to full capacity. On July 23, the Consular Consolidated Database (CCD) was brought back online with limited capacity.
The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues.
We are incrementally increasing the number of processed cases as our systems will allow.
Q: Has the Bureau of Consular Affairs experienced these types of outages in the past?
CA has experienced minor outages in the past, but never of this magnitude. We have a plan in place to mitigate these occurrences in the future.
Q: Is the software to blame? Are contractors at fault? Why was this allowed to happen?
We have been working to improve our services through upgrades while maintaining existing operations worldwide. However, we are limited by outdated software and hardware.
Q: Why did those steps not work? What’s the next step?
We have not determined why the problems occurred. We are working with our contractor and the software vendor to address the problems.
We are bringing additional servers online to increase capacity and response time.
Q: Why wasn’t there a back-up server?
There was back-up capability and redundancy built into the system. However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system.
Q: What steps are being taken prevent this from happening again?
CA has a plan in place to upgrade the CCD to a newer version of the Oracle commercial database software by the end of the calendar year. We are working to ensure the existing system will remain fully functional until the new database is up and running and thoroughly tested. The plan includes establishing two fully redundant systems.
Q: If CA is fee funded, why can’t it build a robust database that doesn’t fail?
The database has grown dramatically, in both quantity of data and functionality, and vastly improved border security. In addition to checking names against databases, we review fingerprints and perform facial recognition.
We are working towards modernization of our software, hardware, and infrastructure. Demand for our services outpaced our modernization efforts.
Consular Affairs has, and has had, a redundant system. However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system. This is one of the issues we are urgently addressing now.
Q: What do I need to know if I’m a passport applicant?
Almost all passports are currently being issued within our customer service standards, despite the system problems.
We are able to issue passports for emergency travel.
Q: What do I need to know if I’m a visa applicant?
Visa applicants they can expect delays as we process pending cases. We remain able to quickly process emergency cases to completion.
We are working urgently to correct the problem to avoid further inconveniencing travelers.
We are posting updates to the visa page of travel.state.gov, and our embassies and consulates overseas are communicating with visa applicants.
In addition to communicating through our websites, e-mail, and letters, we are also reaching out to applicants via Facebook and other social media sites, such as Weibo, to relay the latest information.
Q: Why hasn’t the Department been more forthcoming until now?
We have experienced CCD outages in the past, but they have never disrupted our ability to perform consular tasks at this magnitude.
We informed the public as soon as it was apparent there was not a quick fix to bring the CCD back to normal operating capacity, and are briefing Congressional staffers regularly.
Q: What is the outlook for Non-immigrant visas? When do we estimate the backlog will be processed?
That will depend on a number of factors. Current efforts are focused on bringing the system back to normal operations.
We must also continue processing new requests. We are committed to reducing the number of pending visa cases as quickly as possible, but we want applicants to know that we will continue to be operating at less than optimal efficiency until the system is restored to full functionality.
Q: Is the Department going to reimburse applicants who missed flights/canceled weddings/missed funerals?
We sincerely regret any delays, inconvenience, or expense that applicants have may have incurred due to the CCD performance issues.
While it might be of little solace to those who have experienced hardship, we are always very careful to tell travelers NOT to make travel plans until they have a visa in hand. Even when the CCD is operating normally, there may be delays in printing visas.
The Department does not have the authority to reimburse applicants for personal travel, nor do we include these costs when calculating our fees. The Department cannot refund visa fees except in the specific circumstances set out in our regulations.
Q: What impact will this have on SIVs?
We have the highest respect for the men and women who take enormous risks in supporting our military and civilian personnel. We are committed to helping those who have helped us. While issuances of Special Immigrant Visas (SIVs) to Afghans and Iraqis have been impacted, as have visa issuances around the world, SIV processing continues and remains a high priority.
Q: How is this impacting student visas? They are scheduled to start the fall semester soon.
We are committed to issuing visas to all qualified students and exchange visitors. Issuance of student and exchange visitor visas has been impacted in the past few days, but visa processing continues.
We understand the importance to international students and exchange visitors, their families, and their U.S. host institutions of timely visa issuance in order to facilitate travel and to ensure all students and exchange visitors may begin their programs on time.
Q: What about situations where the student won’t arrive to school on time?
Students should contact their educational institution’s Designated School Official (F and M visas) or designated U.S. sponsor’s Responsible Officer (J visas) and discuss with them what arrangements they can provide for you to begin your program after the start date on your Form I-20 (F and M visas) or Form DS 2019 (J visas), should such a circumstance become necessary.
Q: Will this have any impact on the Diversity Visa program in September?
While issuances of all immigrant visas, including diversity visas, have been impacted in the past few days, IV processing continues and remains a high priority. The Department expects to have used all numbers for DV-2014 when the program year ends on September 30, 2014.
Q: What impact do we anticipate this will have on the U.S. economy?
Tourism and students have a major impact on our economy. Last year, it was estimated that international visitors spent $180.7 billion and supported 1.3 million American jobs. International students contribute $24.7 billion to the U.S. economy through their expenditures on tuition and living expenses, according to the Department of Commerce.
We recognize the significant impact that international travel and tourism has on the U.S. economy, and are taking all possible steps to ensure that the economic impact is minimal.
People traveling under the Visa Waiver Program are not affected at all; nor are those whose previously-issued visas remain valid.
We routinely advise applicants needing new visas to make appointments well in advance of their planned travel, and not to book their travel until they have their printed visas in hand.
The original post is available here. If CA is reading this, it would be helpful if a link to the FAQ is posted on the main visa page of travel.state.gov and in the News section. We were only able to find the FAQ from a link provided in Facebook and not from browsing around the travel.state.gov website.
- State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience (diplopundit.net)
- Passport Database Outage: ‘We Regret The Inconvenience’ (informationweek.com)
- US consular database crash: Not a good time to renew your passport (siliconangle.com)
- Oracle-based system for US visas still glitchy after software update (pcworld.com)
- ‘My papers, please?’ US passport database crash leaves travelers stranded (go.theregister.com)