“The best way to treat obstacles is to use them as stepping-stones. Laugh at them, tread on them, and let them lead you to something better.”
― Enid Blyton, Mr Galliano’s Circus
The Tumblr for Foreign Service Problems has been around for many months now. Sometime this past spring it also joined Twitter. Yes, it is hysterical and absolutely spot on. Below are some of our favorite entries to delight your day. Unless, the Foreign Service has also ruined your sense of humor, in which case, we pray you get it back — fast! or that could quickly be a future entry. With permission from @FS_Problems:
When someone mistakes you for being the Ambassador’s personal household help rather than a Foreign Service Officer or Specialist
When you know that you won’t be promoted before you TIS/TIC out and just don’t care any more.
— FS Problems (@FS_Problems) August 22, 2014 (Note: TIS for time-in- service, time in a combination of salary classes, computed from date of entry into the Foreign Service; TIC for time-in-class, time in a single salary class).
In an interview last week with Philippine media, Ambassador Philip Goldberg expressed disappointment over the disposition of the murder case:
In an interview with ANC, Goldberg said nobody “served a day for that brutal crime.” The diplomat is referring to the murder of US Marine Major George Anikow’s killing on November 24, 2012 at a security checkpoint in Bel-Air. The incident was partly captured in a security camera. Charged were Juan Alfonso Abastillas, Osric Cabrera, Galicano Datu III, and Crispin de la Paz.
Goldberg noted only two suspects were convicted of homicide “but were given probation” by the trial court. The two others got scot free from any charges. […] He said it’s been hard explaining to the family as to “why this happened in a case of very brutal murder.”
The Philippine Justice Department had reportedly filed murder charges previously against the four suspects who, according to reports, come from well-to-do families — Juan Alfonso Abastillas, 24; Crispin dela Paz, 28; Osric Cabrera, 27; and Galicano Datu III, 22.
News report from the Philippines indicate that the victim’s sister, Mary Anikow and his 77 year-old mother traveled to Manila to observed the trial in 2013. “The United States is not perfect; everyone knows this. But most people generally don’t get away with murder,” Ms Anikow said.
Ambassador Goldberg in the ANC interview said that the Philippine Department of Justice promised the embassy there could be something done with regard to the probation. “But it has been appealed once, and it was denied. So it looks like it’s the end of the road,” he said.
‘Well-to-do kids accused in murder of American diplomat’s husband get visas to study in the United States’ — please, can we at least make sure we don’t end up with a headline like that?
Yesterday, after three days of chaos, the State Department issued a Travel Alert informing U.S. citizens of the risks of traveling to Burkina Faso following the fall of the government of President Compaore:
The State Department alerts U.S. citizens of the risks of travel to or residing in Burkina Faso and recommends U.S. citizens defer all non-essential travel. This Travel Alert will expire on January 29, 2015.
On October 31, Burkina Faso’s President Compaore resigned. The status of a transitional government remains unclear. There are incidents of looting throughout the capital city of Ouagadougou, Bobo-Dioulasso, and other parts of the country.
The situation is dynamic and closures or openings of border and airports are likely to change and remain unpredictable for some time. Currently, land and air borders have been closed. U.S. citizens should stay informed and abreast of local media reports for land border and airport updates.
U.S. citizens in Burkina Faso may find that at times sheltering in place may be the only and best security option.
U.S. citizens residing in Burkina Faso should remain vigilant and utilize appropriate personal security practices. Avoid large gatherings, protests, or demonstrations; maintain situational awareness and exercise good judgment; be alert and remain aware of your surroundings; and stay abreast of the situation through media outlets.
Yesterday, the State Department expressed concern over the transfer of power in Burkina Faso:
The United States is concerned about the unfolding events in Burkina Faso. We regret the violence and the loss of life in Burkina Faso and call on all parties to avoid further violence. We reiterate our call for all parties to follow the constitutionally mandated process for the transfer of power and holding of democratic elections following the resignation of former President Blaise Compaore. We condemn any attempts by the military or other parties to take advantage of the situation for unconstitutional gain and call on all parties to respect the people’s support for the democratic process.
According to Vice News, Lieutenant Colonel Yacouba Isaac Zida, who assumed power has been a member of the military for more than 20 years, and served as the second in command of the ex-president’s security regiment. This is apparently, the seventh time a military officer has seized power since Burkina Faso won its independence from France more than 50 years ago. If history is any indication, he may still be around in 2022 in the “land of upright people.”
The U.S. Embassy in Ouagadougou issued the following emergency message to U.S. citizens in Burkina Faso. The messages are dated but typically do not carry a timestamp:
On Thursday, October 30, President Compaore announced in a televised address that he will continue dialogue to form a transitional government after which he will transfer power to a democratically elected president. He reiterated the message that the government is dissolved and announced that the state of martial law is lifted in all of Burkina Faso.
However, there is currently a 7:00 pm to 6:00 am curfew nationwide.
The city of Ouagadougou currently appears to be calm, however protesters continue to gather at the Place de la Nation in Ouagadougou, and at the Place Tiefo Amoro (Station Square) in Bobo-Dioulasso. Crowds and spontaneous protests may also form elsewhere.
Embassy staff continues to shelter in place until further notice. We urge U.S. citizens in Ouagadougou to do the same and to make movements for essential purposes only.
At this time we do not know if civilians have access to the Ouagadougou International Airport. We are monitoring the situation but it is unclear whether flights continue to operate.
Meanwhile, today, Burkina Faso said bye-bye Blaise:
Except for the Emergency Message from Embassy Ouagadougou, there is no Travel Warning or Alert issued on Burkina Faso as of this writing. The latest State Department statement is dated October 30, and obviously had been overtaken by events.
The United States welcomes President Compaore’s decision to withdraw a National Assembly bill which would have amended the constitution and allowed him to run for an additional term of office. We also welcome his decision to form a government of national unity to prepare for national elections and to transfer power to a democratically elected successor. We look forward to that transition taking place in 2015. We regret the violence and the loss of life today in Burkina Faso, and call on all parties to avoid further violence. We underscore our commitment to peaceful transitions of power through democratic elections and emphasize neither side should attempt to change the situation through extra-constitutional means.
In September, we blogged that the State Dept Awarded $4.9 Million Contract to Phoenix Air for Air Ambulance Evacuation #Ebola. Apparently, the last couple of days there was a flap over a State Department memo on a plan to bring non-Americans with Ebola to U.S. soil for treatment. The memo labeled Sensitive But Unclassified – Predesicional is available to read hereand notes USG obligation to non-U.S. citizen employees and contractors of U.S. agencies (USAID, CDC, etc.) and programs as well as NGOs and private firms based in the United States.
The Washington Times identified the memo’s author as Robert Sorenson, deputy director of the Office of International Health and Biodefense (OES/IHB). The Office of International Health and Biodefense is the primary State Department policy office responsible for a variety of international health issues. It takes part in U.S. Government policymaking on infectious disease, environmental health, noncommunicable disease issues, global health security, antimicrobial resistance, and counterfeit and substandard medications. A clearance sheet attached to the memo reportedly says it was cleared by offices of the deputy secretary, the deputy secretary for management, the office of Central African affairs and the medical services office.
QUESTION: And then the last one on this is: There was a report last night and again this morning about this memo that was – the State Department memo —
MS. PSAKI: Sure, let me address that.
QUESTION: — about bringing —
MS. PSAKI: Mm-hmm. One, just factually, the document referenced was drafted by a midlevel official but not cleared by senior leaders. It never came to senior officials for approval. And any assertion that the memo was cleared by decision-makers is inaccurate. There are no plans to medevac non-Americans who become ill with Ebola to the United States. We have discussed allowing other countries to use our medevac capabilities to evacuate their own citizens to their home countries or third countries subject to reimbursement and availability. But we’re not contemplating bringing them back to the United States for treatment.
QUESTION: So the – but essentially, what you’re saying is that one guy somewhere in this building came up with this idea and put it on paper, but it never went anywhere? Is that what you’re saying?
MS. PSAKI: Correct. It’s also weeks old and the memo isn’t current because European – our European partners —
QUESTION: All right. Okay.
MS. PSAKI: — have addressed this matter by providing their own guarantees, but go ahead.
QUESTION: One problem that – I mean, that I see is that a week ago, the Pentagon and the White House was insisting that, no, no, no, there is no overall quarantine order and it’s just this one commander, or these guys who are in Italy. And now all of a sudden, today we have Secretary Hagel saying no, it’s going to be – it’s Pentagon-wide and it’s going to go to all of the troops that are there. What is there to prevent this memo from coming back to life, as it were —
MS. PSAKI: Well, I think with this —
QUESTION: — and becoming policy? Has it been flat out rejected or is it just kind of sitting on a shelf someplace and maybe could be implemented at some point?
MS. PSAKI: It’s sitting on a shelf or on a computer – since we use computers nowadays – by the individual who wrote it, I suppose. I think the important point here is that our European partners, since several weeks ago when that was written, have addressed this by providing a guarantee to international health workers that they would either be flown to Europe or receive high-quality treatment on the spot. So it’s not applicable at this point.
QUESTION: Okay. Well, in general, why was this never approved? I mean, it seems – I mean, you could make the argument that the U.S. has great healthcare facilities, that no one who has contracted the disease in the United States has actually died. So I think there might be some who could make the argument that why not bring people?
MS. PSAKI: Sure, but many countries have decided to make that decision to deal with it themselves, and we’ve certainly been discussing with them how to do that.
QUESTION: So this has been discarded as unnecessary rather that rejected —
MS. PSAKI: It was never discussed at any levels, in any serious level with decision-makers. So I don’t – wouldn’t say it was discarded, but —
QUESTION: Along the lines of what Matt was saying, on page 5 of the memo, it says that it was approved by Nancy Powell, the head of the Ebola Coordination Unit. Doesn’t that suggest it was fairly further along in the process?
MS. PSAKI: I’m happy to look at the approval memo. As I understand, and just so you know, sometimes there are people listed. It doesn’t mean they cleared it. It just means there are people who need to clear a memo. So I will check and see if there was anybody who actually cleared it.
“One guy somewhere in this building came up with this idea and put it on paper, but it never went anywhere?” And the official spokesperson, without blinking said, “correct.”
Don’t you just hate it when they say things like that and throw some midlevel official under the medevac plane?
In fact, the justification for the air ambulance evacuation contract awarded to Phoenix Air on August 18, 2014 appears clear enough as to why this was necessary:
The USG is left with only two options in supporting a CDC scientist that has a high risk exposure to an EVD patient — use the PAG capability to ﬂy the person back to the US for observation and optimum care should disease develop, or leave the person in place where no care is available if the disease develops. The question, then, is not how many EVD patients will be moved, but rather how many contacts and EVD patients will be moved across the entire international response population (as many as three per month). Finally, from a pragmatic stand point, given the limited options for movement of even asymptomatic contacts, it has become clear that an international response to this crisis will not proceed if a reliable mechanism for patient movement cannot be established and centrally managed.
That leaked memo is not saying we’re moving Liberia’s entire infected population for treatment in U.S. hospitals, is it? An argument can be made that the USG has an obligation to assist in the treatment of those infected in the course of their work fighting the ebola outbreak on behalf of the international community. The State Department is not/not making that argument, of course. The only official argument it is making is that — that memo, that never went anywhere beyond the midlevel officer’s desk.
Nothing to do with an election coming up? Sure, okay.
Mr. Kerry is vocal and forceful in internal debates, officials said, but he frequently gets out of sync with the White House in his public statements. White House officials joke that he is like the astronaut played by Sandra Bullock in the movie “Gravity,” somersaulting through space, untethered to the White House.
Aides to Mr. Kerry reject that portrait, saying he dials into White House meetings from the road and is heavily involved in the policy process.
Yup, you’re careening wildly all over the place, and undermining very publicly this administration’s Secretary of State, and you think there will be no consequences because it’s just a joke? Didn’t you stop to consider that foreign ministries around the world may start questioning just how “untethered” the Secretary is to the White House and that it could impact how they deal with him or his agency?
For Baracksakes, adult supervisors need to please step on the brakes here!
* * *
Oops, what did the WH say?
.@PressSec, per WH pool: #SecKerry, like Sandra Bullock’s astronaut in Gravity, “has circled the globe many times to advance U.S. interests”
First the good news! The British diplomats in D.C. enjoy a smokin’ good barbecue. Apparently, they had a a ‘White House BBQ’ to mark the 200th anniversary of a ‘rather unfortunate event in UK/US relations.’ The embassy even had a huge White House cake to commemorate the burning. Then tweeted about it:
In related news, the embassy’s social media ninjas in D.C. shortly thereafter had to apologize for their tweet. It looks like some in the Twitterverse did not appreciate the joke even if it came 200 years later with cake. Some complained that this was bad form, some blamed David Cameron. No, no one has thought of blaming the Queen yet.
We hope they’re not going to fire the intern. Anyone from Foggy Bottom attended the barbecue?
Apologies for earlier Tweet. We meant to mark an event in history & celebrate our strong friendship today http://t.co/gs3heJDMzt
Yesterday, we posted about the troubled Consular Consolidated Database (CCD) (see State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience). During the Daily Press Briefing, yesterday, the State Department officially stated that it believed the root cause of the problem was “a combination of software optimization and hardware compatibility issues.” According to the deputy spokesperson, the servers are getting back online but that they are coming back in a queue and that fixes are not being done on a country-by-country basis. And by the way, it’s not just the peak summer travel season, there’s also the Africa Summit in D.C. next week.
“Obviously, there’s actually a huge crush right now because of the Africa Leaders Summit, so obviously that’s a huge priority for us to make sure everybody gets their visas for the Africa Leaders Summit. We do believe that a vast majority of the travelers who have applied for visas for the summit have been issued.”
CA’s FB folks have been regularly answering questions from angry complaints posted on its Facebook page and have announced that they will continue to monitor and respond to consular clients at 9:00 EDT tomorrow, Thursday, July 31.
Late yesterday, the Bureau of Consular Affairs also posted a new Frequently Asked Questions on Facebook and on its website (not easily accessible from the main visa page) concerning the CCD performance issues and the steps taken to address those issues. Perhaps the most surprising is that its back-up capability and redundancy built into the CCD were both affected killed by the upgrade that hobbled the system. Something to look forward to by end of calendar year — CA is upgrading the CCD to a newer version of the Oracle commercial database software and that plan includes establishing two fully redundant systems. We are republishing the FAQ in full below.
Information Regarding Ongoing Consular Consolidated Database (CCD) Performance Issues and Steps Taken| JULY 30, 2014
The Department of State continues to work to restore our visa system to full functionality.
We anticipate it will take weeks to resume full visa processing capacity.
We continue to prioritize immigrant visas, including adoption cases. So far, we have been able to issue most cases with few delays.
Nearly all passports are currently being issued within our customer service standards, despite the system problems.
We are able to issue passports for emergency travel.
Frequently Asked Questions
Q: What caused the system performance issues? Hardware, software, or both? Details?
On July 20, to improve overall system performance and address previous intermittent performance issues, we updated software as recommended. Our database began experiencing significant performance issues shortly after this maintenance was performed.
A root cause has not been identified at this time. Current efforts are focused on bringing the system back to normal operations. Once that has been accomplished, resources will be applied to determine the root cause.
Q: What steps did we take to mitigate the performance issues?
Since July 20, our team has worked to restore operations to full capacity. On July 23, the Consular Consolidated Database (CCD) was brought back online with limited capacity.
The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues.
We are incrementally increasing the number of processed cases as our systems will allow.
Q: Has the Bureau of Consular Affairs experienced these types of outages in the past?
CA has experienced minor outages in the past, but never of this magnitude. We have a plan in place to mitigate these occurrences in the future.
Q: Is the software to blame? Are contractors at fault? Why was this allowed to happen?
We have been working to improve our services through upgrades while maintaining existing operations worldwide. However, we are limited by outdated software and hardware.
Q: Why did those steps not work? What’s the next step?
We have not determined why the problems occurred. We are working with our contractor and the software vendor to address the problems.
We are bringing additional servers online to increase capacity and response time.
Q: Why wasn’t there a back-up server?
There was back-up capability and redundancy built into the system. However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system.
Q: What steps are being taken prevent this from happening again?
CA has a plan in place to upgrade the CCD to a newer version of the Oracle commercial database software by the end of the calendar year. We are working to ensure the existing system will remain fully functional until the new database is up and running and thoroughly tested. The plan includes establishing two fully redundant systems.
Q: If CA is fee funded, why can’t it build a robust database that doesn’t fail?
The database has grown dramatically, in both quantity of data and functionality, and vastly improved border security. In addition to checking names against databases, we review fingerprints and perform facial recognition.
We are working towards modernization of our software, hardware, and infrastructure. Demand for our services outpaced our modernization efforts.
Consular Affairs has, and has had, a redundant system. However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system. This is one of the issues we are urgently addressing now.
Q: What do I need to know if I’m a passport applicant?
Almost all passports are currently being issued within our customer service standards, despite the system problems.
We are able to issue passports for emergency travel.
Q: What do I need to know if I’m a visa applicant?
Visa applicants they can expect delays as we process pending cases. We remain able to quickly process emergency cases to completion.
We are working urgently to correct the problem to avoid further inconveniencing travelers.
We are posting updates to the visa page of travel.state.gov, and our embassies and consulates overseas are communicating with visa applicants.
In addition to communicating through our websites, e-mail, and letters, we are also reaching out to applicants via Facebook and other social media sites, such as Weibo, to relay the latest information.
Q: Why hasn’t the Department been more forthcoming until now?
We have experienced CCD outages in the past, but they have never disrupted our ability to perform consular tasks at this magnitude.
We informed the public as soon as it was apparent there was not a quick fix to bring the CCD back to normal operating capacity, and are briefing Congressional staffers regularly.
Q: What is the outlook for Non-immigrant visas? When do we estimate the backlog will be processed?
That will depend on a number of factors. Current efforts are focused on bringing the system back to normal operations.
We must also continue processing new requests. We are committed to reducing the number of pending visa cases as quickly as possible, but we want applicants to know that we will continue to be operating at less than optimal efficiency until the system is restored to full functionality.
Q: Is the Department going to reimburse applicants who missed flights/canceled weddings/missed funerals?
We sincerely regret any delays, inconvenience, or expense that applicants have may have incurred due to the CCD performance issues.
While it might be of little solace to those who have experienced hardship, we are always very careful to tell travelers NOT to make travel plans until they have a visa in hand. Even when the CCD is operating normally, there may be delays in printing visas.
The Department does not have the authority to reimburse applicants for personal travel, nor do we include these costs when calculating our fees. The Department cannot refund visa fees except in the specific circumstances set out in our regulations.
Q: What impact will this have on SIVs?
We have the highest respect for the men and women who take enormous risks in supporting our military and civilian personnel. We are committed to helping those who have helped us. While issuances of Special Immigrant Visas (SIVs) to Afghans and Iraqis have been impacted, as have visa issuances around the world, SIV processing continues and remains a high priority.
Q: How is this impacting student visas? They are scheduled to start the fall semester soon.
We are committed to issuing visas to all qualified students and exchange visitors. Issuance of student and exchange visitor visas has been impacted in the past few days, but visa processing continues.
We understand the importance to international students and exchange visitors, their families, and their U.S. host institutions of timely visa issuance in order to facilitate travel and to ensure all students and exchange visitors may begin their programs on time.
Q: What about situations where the student won’t arrive to school on time?
Students should contact their educational institution’s Designated School Official (F and M visas) or designated U.S. sponsor’s Responsible Officer (J visas) and discuss with them what arrangements they can provide for you to begin your program after the start date on your Form I-20 (F and M visas) or Form DS 2019 (J visas), should such a circumstance become necessary.
Q: Will this have any impact on the Diversity Visa program in September?
While issuances of all immigrant visas, including diversity visas, have been impacted in the past few days, IV processing continues and remains a high priority. The Department expects to have used all numbers for DV-2014 when the program year ends on September 30, 2014.
Q: What impact do we anticipate this will have on the U.S. economy?
Tourism and students have a major impact on our economy. Last year, it was estimated that international visitors spent $180.7 billion and supported 1.3 million American jobs. International students contribute $24.7 billion to the U.S. economy through their expenditures on tuition and living expenses, according to the Department of Commerce.
We recognize the significant impact that international travel and tourism has on the U.S. economy, and are taking all possible steps to ensure that the economic impact is minimal.
People traveling under the Visa Waiver Program are not affected at all; nor are those whose previously-issued visas remain valid.
We routinely advise applicants needing new visas to make appointments well in advance of their planned travel, and not to book their travel until they have their printed visas in hand.
The original post is available here. If CA is reading this, it would be helpful if a link to the FAQ is posted on the main visa page of travel.state.gov and in the News section. We were only able to find the FAQ from a link provided in Facebook and not from browsing around the travel.state.gov website.
The first announcement about the troubled Consular Consolidated Database (CCD) went out on Wednesday, July 23:
The Department of State Bureau of Consular Affairs is currently experiencing technical problems with our passport/visa system. This issue is worldwide and is not specific to any particular country, citizenship document, or visa category. We apologize to applicants who are experiencing delays or are unable to obtain a passport, Consular Report of Birth Abroad, or visa at this time. We are working urgently to correct the problem and expect our system to be fully operational again soon.
The AP reported on July 23 that unspecified glitches have resulted in performance issues since Saturday, which would be July 19.
On July 25, CA announced:” Our visa and passport processing systems are now operational, however they are working at limited capacity. We are still working to correct the problem and expect to be fully operational soon.”
A State Department official speaking on background told us the same day that this issue was not/not caused by hackers. We were told that the CCD crashed shortly after maintenance was performed and that the root cause of the problem is not yet known.
As of July 27, the Department of State has made continued progress on restoring our system to full functionality. As we restore our ability to print visas, we are prioritizing immigrant cases, including adoptions visas. System engineers are performing maintenance to address the problems we encountered. As system performance improves, we will continue to process visas at U.S. Embassies and Consulates worldwide. We are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.
The Department of State Bureau of Consular Affairs continues to make progress restoring our nonimmigrant visa system to full functionality. Over the weekend, the Department of State implemented system changes aimed at optimizing performance and addressing the challenges we have faced. We are now testing our system capacity to ensure stability. Processing of immigrant visas cases, including adoptions, remains a high priority. Some Embassies and Consulates may temporarily limit or reschedule nonimmigrant visa interview appointments until more system resources become available to process these new applications. We sincerely regret the inconvenience to travelers, and are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.
The CA Bureau’s Facebook page has been inundated with comments. There were complaints that at one post the visas were printing fine and then they were not. There were complains from people waiting for visas for adopted kids, for fiancees, for family members, for family waiting at the border, for students anxious to get to their schools, people worried about time running out for diversity visas, applicants with flights already booked, and many more. One FB commenter writes, “I feel that the problem most people have is not that the system broke, but the lack of clear, meaningful information so people can make appropriate plans.”
Other than what the CA Bureau chose to tell us, we cannot pry any substantial detail from official sources. We, however, understand from sources familiar with the system but not authorized to speak for the bureau that the CCD has been having problems for sometime but it got worse in the last couple weeks. If you’re familiar with the highs and lows of visa operation, this will not be altogether surprising. Whatever problems already existed in the system prior to this “glitch” could have easily been exacerbated in July, which is the middle of the peak travel season worldwide. A source working in one of our consular posts confirmed to us that the system is back running, but not at the normal level and that the backlogs are building up. Another source told us that Beijing already had a 15k NIV backlog over the weekend. We haven’t yet heard what are the backlogs like in mega visa-issuing posts like Brazil, Mexico and India.
We understand that everyone is currently doing all they can to get the process moving, but that some cases are getting through the system, while some are not. No one seems to know why this is happening. These machine readable visas are tied to the system and there are no manual back-ups for processing these cases (more of that below).
So who owns CCD?
The Consular Systems and Technology (CA/CST) manages the CCD. We have previously blogged about its troubled past:
CST is currently headed by a new Director, Greg D Ambrose who reports to the CA Bureau’s Assistant Secretary. It looks like despite the 2011 OIG recommendation, the CST deputy position remains vacant. We should also note that the Asst Secretary for Consular Affairs Janice Jacobs retired this past April. No replacement has been nominated to-date and Michele T. Bond has been Acting Assistant Secretary since Ms. Jacobs’ departure.
Last September, Mr. Ambrose was with FedScoopTV and talked about Consular One, the future of consular IT.
CST Just Got a New Data Engineering Contract
In Many 2014, ActioNet, Inc., headquartered in Vienna, Virginia,announced a 5-year task order for data engineering, supporting CST.
ActioNet, Inc. announced today the award of a five (5)-year task order entitled Data Engineering (DE) in support of Department of State (DOS). This task order will provide data engineering and database infrastructure support services necessary for planning, analysis, design, and implementation services for the Bureau of Consular Affairs. These service also include contract and program management support to ensure that innovation, efficiency, and cost control practices are built into the program. […] The Office of Consular Systems and Technology (CST) within the Bureau develops, deploys and maintains the unclassified and classified IT infrastructures that help execute these missions. The Bureau currently manages over 800 servers worldwide, in order to comply with the fast paced changes inherent to data processing and telecommunications, CST requires that contractor services provide for rapid provisioning of highly experienced and trained individuals with the IT (information technology) backgrounds and the security clearances required of CA’s environment of workstation-based local and wide-area network infrastructures.
Due to limited information available, we don’t know if the new Consular One and/or the new DE contract are related to ongoing issues or if there are hardware issues, given the multiple legacy systems, but we do know that CST has both an impressive and troubled history. Let’s take a look.
Records Growing by the Day
The 2010 Consular Consolidated Database (CCD) Privacy Impact Assessment (PIA) describes (pdf) the CCD as “one of the largest Oracle based data warehouses in the world that holds current and archived data from the Consular Affairs (CA) domestic and post databases around the world.” According to the PIA, in December 2009, the CCD contained over 100 million visa cases and 75 million photographs, utilizing billions of rows of data, and has a current growth rate of approximately 35 thousand visa cases every day. The 2011 OIG report says that in 2010, the CCD contained over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day.
That was almost four years ago.
A Critical Operational and National Security Database with No Back-Up System?
According to publicly available information, the CCD’s chief functions are 1) to support data delivery to approved applications via industry-standard Web Service queries, 2) provide users with easy-to-use data entry interfaces to CCD, and 3) allow emergency recovery of post databases. The CCD also serves as a gateway to IDENT and IAFIS fingerprint checking databases, the Department of State Facial Recognition system, and the NameCheck system. It provides access to passport data in Travel Document Issuance System (TDIS), Passport Lookout Tracking System (PLOTS), and Passport Information Electronic Records System (PIERS). The OIG says that the CCD serves 11,000 users in the Department and more than 19,000 users in other agencies, primarily the Department of Homeland Security (DHS) and various law enforcement elements, and is accessed more than 120 million times every month.
Given that the CCD is considered “a critical operational and national security database,” there is surprisingly no redundancies or any back-up system.
Resurrect the Standard Register protectograph aka: `Burroughs visas’?
No one is actually suggesting that but when the CCD system is down, there is no manual way to issue a visa. No post can handprint visas because security measures prevent consular officers from printing a visa unless it is approved through the database system. Here is a quick history of the handprinted ‘Burroughs visas’ and the machine readable visas via the GPO:
November 18, 1988, mandated the development of a machine-readable travel and identity document to improve border entry and departure control using an automated data-capture system. As a result, the Department developed the Machine Readable Visa, a durable, long-lasting adhesive foil made out of Teslin.
Before MRVs, nonimmigrant visas were issued using a device called a Standard Register protectograph, otherwise known as a Burroughs certifier machine. It produced what was colloquially known as a “Burroughs visa,” an indelible ink impression mechanically stamped directly onto a page in the alien’s passport. Over time, Burroughs machines were gradually replaced by MRV technology, which is now used exclusively by all nonimmigrant visa issuing posts throughout the world.
Burroughs visas contained a space in which a consular employee was required to write the name of the alien to whom the visa was being issued. An alien’s passport might also include family members, such as a spouse, or children, who also had to be listed on the visa. In March 1983, in order to expedite the issuance of nonimmigrant visas and to improve operational efficiency, the Department authorized the use of a “bearer(s)” stamp for certain countries so that consular officers would not have to spend time writing in the applicant’s name (and those of accompanying family members). MRVs, however, must be issued individually to qualified aliens. Consequently, the “bearer”annotation has become obsolete.
The problem with the old Burroughs machine, besides the obvious, was maybe — you run out of ink, the plates are ruined/broken or you need it oiled. We could not remember those breaking down. With the MRV technology, all posts are connected to a central database, and the new machines by themselves cannot issue visas. Which brings us to the security of that system.
Management Alert on Information System Security Program
The State Department PIA says that “To appropriately safeguard the information, numerous management, operational, and technical security controls are in place in accordance with the Federal Information Security Management Act (FISMA) of 2002 and information assurance standards published by the National Institute of Standards and Technology (NIST).” Must be why in November 2013, the Office of the Inspector General issued a Management Alert for significant and recurring weaknesses found in the State Department’s Information System Security Program over the past three fiscal years (FY 2011-2013).
In 2011, State/OIG also issued a report on CA’s CST division and has, what appears to be a lengthy discussion of the CCD, but almost all of it but a paragraph had been redacted:
That OIG report also includes a discussion of the Systems Development Life Cycle Process and notes that decision control gates within CST’s SDLC process are weak. It cites a couple of examples where this manifested: 1) the development of the Consular report of Birth Abroad (CRBA) system. “The ownership of development and deployment shifted throughout the process, and the business unit’s requirements were not clearly communicated to the development team. As a result, CST designed and tested the CRBA for a printer that did not match the printer model identified and procured by the business unit;” 2) the Crisis Task Force application, for which CST was tasked to enhance its Web-facing interaction. “The deployment of this application has been challenged by the lack of project ownership and decision controls, as well as by the incomplete requirements definition. The use of incorrect scripts that were provided by the CM group has further delayed the Crisis Task Force application’s deployment.”
If there’s somethin’ strange in your CCD, who ya gonna call? (Glitchbusters!)
The Consular Consolidated Database (CCD) is central to all consular operations. It is run by CST where according to the OIG, “the smooth functioning of every part of the office depends on its contractors.” And because it runs such an important element of U.S. national security systems, if all CST’s contractors, all 850 of them quit, this critical consular data delivery to the State Department and other Federal agencies would screech to a a halt.
To carry out its mandate, CST must provide uninterrupted support to 233 overseas posts, 21 passport agencies, 2 passport processing centers, and other domestic facilities, for a total of 30,000 end users across 16 Federal agencies and in nearly every country. CST faces 24/7/365 service requirements, as any disruption in automated support brings operations to an immediate halt, with very serious implications for travelers and the U.S. image. […] CST is led by a director and is staffed by 68 full-time equivalent (FTE) employees (62 Civil Service and 6 Foreign Service). There are 12 positions (3 Foreign Service and 9 Civil Service) currently vacant. CA recently authorized CST 19 additional FTE positions. There are also more than 850 contractors operating under nearly 30 different contracts. In FY 2010, CST’s annual operating budget was approximately $266 million.
If CCD is compromised for a lengthy period such as the last couple of weeks, what is the back up plan to keep the operation going? Obviously, none. It’s either down or running under limited or full capacity. No one we know remember CCD problems persist this long. Right now, we know from a reliable source that the system is not down, and some cases and going through but — what if the CCD is completely down for two weeks … four weeks … wouldn’t international travel come to a slow stop?
What if CCD goes down indefinitely whether by hardware or software glitch or through malicious penetration by foreign hackers, what happens then?
Currently, it appears nothing can be done but for folks to be patient and wait until the fixes are in. We know they’re working hard at it but there’s got to be a better way. Perhaps we can also agree that this has very serious national security implications on top of disgruntled travelers and a grave impact on the U.S. image overseas.