The first announcement about the troubled Consular Consolidated Database (CCD) went out on Wednesday, July 23:
The Department of State Bureau of Consular Affairs is currently experiencing technical problems with our passport/visa system. This issue is worldwide and is not specific to any particular country, citizenship document, or visa category. We apologize to applicants who are experiencing delays or are unable to obtain a passport, Consular Report of Birth Abroad, or visa at this time. We are working urgently to correct the problem and expect our system to be fully operational again soon.
The AP reported on July 23 that unspecified glitches have resulted in performance issues since Saturday, which would be July 19.
On July 25, CA announced:” Our visa and passport processing systems are now operational, however they are working at limited capacity. We are still working to correct the problem and expect to be fully operational soon.”
A State Department official speaking on background told us the same day that this issue was not/not caused by hackers. We were told that the CCD crashed shortly after maintenance was performed and that the root cause of the problem is not yet known.
As of July 27, the Department of State has made continued progress on restoring our system to full functionality. As we restore our ability to print visas, we are prioritizing immigrant cases, including adoptions visas. System engineers are performing maintenance to address the problems we encountered. As system performance improves, we will continue to process visas at U.S. Embassies and Consulates worldwide. We are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.
The Department of State Bureau of Consular Affairs continues to make progress restoring our nonimmigrant visa system to full functionality. Over the weekend, the Department of State implemented system changes aimed at optimizing performance and addressing the challenges we have faced. We are now testing our system capacity to ensure stability. Processing of immigrant visas cases, including adoptions, remains a high priority. Some Embassies and Consulates may temporarily limit or reschedule nonimmigrant visa interview appointments until more system resources become available to process these new applications. We sincerely regret the inconvenience to travelers, and are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.
The CA Bureau’s Facebook page has been inundated with comments. There were complaints that at one post the visas were printing fine and then they were not. There were complains from people waiting for visas for adopted kids, for fiancees, for family members, for family waiting at the border, for students anxious to get to their schools, people worried about time running out for diversity visas, applicants with flights already booked, and many more. One FB commenter writes, “I feel that the problem most people have is not that the system broke, but the lack of clear, meaningful information so people can make appropriate plans.”
Other than what the CA Bureau chose to tell us, we cannot pry any substantial detail from official sources. We, however, understand from sources familiar with the system but not authorized to speak for the bureau that the CCD has been having problems for sometime but it got worse in the last couple weeks. If you’re familiar with the highs and lows of visa operation, this will not be altogether surprising. Whatever problems already existed in the system prior to this “glitch” could have easily been exacerbated in July, which is the middle of the peak travel season worldwide. A source working in one of our consular posts confirmed to us that the system is back running, but not at the normal level and that the backlogs are building up. Another source told us that Beijing already had a 15k NIV backlog over the weekend. We haven’t yet heard what are the backlogs like in mega visa-issuing posts like Brazil, Mexico and India.
We understand that everyone is currently doing all they can to get the process moving, but that some cases are getting through the system, while some are not. No one seems to know why this is happening. These machine readable visas are tied to the system and there are no manual back-ups for processing these cases (more of that below).
So who owns CCD?
The Consular Systems and Technology (CA/CST) manages the CCD. We have previously blogged about its troubled past:
CST is currently headed by a new Director, Greg D Ambrose who reports to the CA Bureau’s Assistant Secretary. It looks like despite the 2011 OIG recommendation, the CST deputy position remains vacant. We should also note that the Asst Secretary for Consular Affairs Janice Jacobs retired this past April. No replacement has been nominated to-date and Michele T. Bond has been Acting Assistant Secretary since Ms. Jacobs’ departure.
Last September, Mr. Ambrose was with FedScoopTV and talked about Consular One, the future of consular IT.
CST Just Got a New Data Engineering Contract
In Many 2014, ActioNet, Inc., headquartered in Vienna, Virginia,announced a 5-year task order for data engineering, supporting CST.
ActioNet, Inc. announced today the award of a five (5)-year task order entitled Data Engineering (DE) in support of Department of State (DOS). This task order will provide data engineering and database infrastructure support services necessary for planning, analysis, design, and implementation services for the Bureau of Consular Affairs. These service also include contract and program management support to ensure that innovation, efficiency, and cost control practices are built into the program. […] The Office of Consular Systems and Technology (CST) within the Bureau develops, deploys and maintains the unclassified and classified IT infrastructures that help execute these missions. The Bureau currently manages over 800 servers worldwide, in order to comply with the fast paced changes inherent to data processing and telecommunications, CST requires that contractor services provide for rapid provisioning of highly experienced and trained individuals with the IT (information technology) backgrounds and the security clearances required of CA’s environment of workstation-based local and wide-area network infrastructures.
Due to limited information available, we don’t know if the new Consular One and/or the new DE contract are related to ongoing issues or if there are hardware issues, given the multiple legacy systems, but we do know that CST has both an impressive and troubled history. Let’s take a look.
Records Growing by the Day
The 2010 Consular Consolidated Database (CCD) Privacy Impact Assessment (PIA) describes (pdf) the CCD as “one of the largest Oracle based data warehouses in the world that holds current and archived data from the Consular Affairs (CA) domestic and post databases around the world.” According to the PIA, in December 2009, the CCD contained over 100 million visa cases and 75 million photographs, utilizing billions of rows of data, and has a current growth rate of approximately 35 thousand visa cases every day. The 2011 OIG report says that in 2010, the CCD contained over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day.
That was almost four years ago.
A Critical Operational and National Security Database with No Back-Up System?
According to publicly available information, the CCD’s chief functions are 1) to support data delivery to approved applications via industry-standard Web Service queries, 2) provide users with easy-to-use data entry interfaces to CCD, and 3) allow emergency recovery of post databases. The CCD also serves as a gateway to IDENT and IAFIS fingerprint checking databases, the Department of State Facial Recognition system, and the NameCheck system. It provides access to passport data in Travel Document Issuance System (TDIS), Passport Lookout Tracking System (PLOTS), and Passport Information Electronic Records System (PIERS). The OIG says that the CCD serves 11,000 users in the Department and more than 19,000 users in other agencies, primarily the Department of Homeland Security (DHS) and various law enforcement elements, and is accessed more than 120 million times every month.
Given that the CCD is considered “a critical operational and national security database,” there is surprisingly no redundancies or any back-up system.
Resurrect the Standard Register protectograph aka: `Burroughs visas’?
No one is actually suggesting that but when the CCD system is down, there is no manual way to issue a visa. No post can handprint visas because security measures prevent consular officers from printing a visa unless it is approved through the database system. Here is a quick history of the handprinted ‘Burroughs visas’ and the machine readable visas via the GPO:
November 18, 1988, mandated the development of a machine-readable travel and identity document to improve border entry and departure control using an automated data-capture system. As a result, the Department developed the Machine Readable Visa, a durable, long-lasting adhesive foil made out of Teslin.
Before MRVs, nonimmigrant visas were issued using a device called a Standard Register protectograph, otherwise known as a Burroughs certifier machine. It produced what was colloquially known as a “Burroughs visa,” an indelible ink impression mechanically stamped directly onto a page in the alien’s passport. Over time, Burroughs machines were gradually replaced by MRV technology, which is now used exclusively by all nonimmigrant visa issuing posts throughout the world.
Burroughs visas contained a space in which a consular employee was required to write the name of the alien to whom the visa was being issued. An alien’s passport might also include family members, such as a spouse, or children, who also had to be listed on the visa. In March 1983, in order to expedite the issuance of nonimmigrant visas and to improve operational efficiency, the Department authorized the use of a “bearer(s)” stamp for certain countries so that consular officers would not have to spend time writing in the applicant’s name (and those of accompanying family members). MRVs, however, must be issued individually to qualified aliens. Consequently, the “bearer”annotation has become obsolete.
The problem with the old Burroughs machine, besides the obvious, was maybe — you run out of ink, the plates are ruined/broken or you need it oiled. We could not remember those breaking down. With the MRV technology, all posts are connected to a central database, and the new machines by themselves cannot issue visas. Which brings us to the security of that system.
Management Alert on Information System Security Program
The State Department PIA says that “To appropriately safeguard the information, numerous management, operational, and technical security controls are in place in accordance with the Federal Information Security Management Act (FISMA) of 2002 and information assurance standards published by the National Institute of Standards and Technology (NIST).” Must be why in November 2013, the Office of the Inspector General issued a Management Alert for significant and recurring weaknesses found in the State Department’s Information System Security Program over the past three fiscal years (FY 2011-2013).
In 2011, State/OIG also issued a report on CA’s CST division and has, what appears to be a lengthy discussion of the CCD, but almost all of it but a paragraph had been redacted:
That OIG report also includes a discussion of the Systems Development Life Cycle Process and notes that decision control gates within CST’s SDLC process are weak. It cites a couple of examples where this manifested: 1) the development of the Consular report of Birth Abroad (CRBA) system. “The ownership of development and deployment shifted throughout the process, and the business unit’s requirements were not clearly communicated to the development team. As a result, CST designed and tested the CRBA for a printer that did not match the printer model identified and procured by the business unit;” 2) the Crisis Task Force application, for which CST was tasked to enhance its Web-facing interaction. “The deployment of this application has been challenged by the lack of project ownership and decision controls, as well as by the incomplete requirements definition. The use of incorrect scripts that were provided by the CM group has further delayed the Crisis Task Force application’s deployment.”
If there’s somethin’ strange in your CCD, who ya gonna call? (Glitchbusters!)
The Consular Consolidated Database (CCD) is central to all consular operations. It is run by CST where according to the OIG, “the smooth functioning of every part of the office depends on its contractors.” And because it runs such an important element of U.S. national security systems, if all CST’s contractors, all 850 of them quit, this critical consular data delivery to the State Department and other Federal agencies would screech to a a halt.
To carry out its mandate, CST must provide uninterrupted support to 233 overseas posts, 21 passport agencies, 2 passport processing centers, and other domestic facilities, for a total of 30,000 end users across 16 Federal agencies and in nearly every country. CST faces 24/7/365 service requirements, as any disruption in automated support brings operations to an immediate halt, with very serious implications for travelers and the U.S. image. […] CST is led by a director and is staffed by 68 full-time equivalent (FTE) employees (62 Civil Service and 6 Foreign Service). There are 12 positions (3 Foreign Service and 9 Civil Service) currently vacant. CA recently authorized CST 19 additional FTE positions. There are also more than 850 contractors operating under nearly 30 different contracts. In FY 2010, CST’s annual operating budget was approximately $266 million.
If CCD is compromised for a lengthy period such as the last couple of weeks, what is the back up plan to keep the operation going? Obviously, none. It’s either down or running under limited or full capacity. No one we know remember CCD problems persist this long. Right now, we know from a reliable source that the system is not down, and some cases and going through but — what if the CCD is completely down for two weeks … four weeks … wouldn’t international travel come to a slow stop?
What if CCD goes down indefinitely whether by hardware or software glitch or through malicious penetration by foreign hackers, what happens then?
Currently, it appears nothing can be done but for folks to be patient and wait until the fixes are in. We know they’re working hard at it but there’s got to be a better way. Perhaps we can also agree that this has very serious national security implications on top of disgruntled travelers and a grave impact on the U.S. image overseas.
“Please Human Resources, we beg of you, control your colleagues in HR/CDA and stop the madness. These untenured walking talking EEO violations responsible for the “career development” of other officers are a contradiction to what officers expect from HR. Hazing? Bullying? Or just plain incompetence? Where are the HR professionals at State?”
The State Department’s Assistant Secretary of State for European Affairs, Victoria Nuland was caught on tape with Geoffrey Pyatt, the U.S. Ambassador to Ukraine working the diplomatic sausage over the Ukraine crisis. The private conversation was recorded and uploaded to YouTube by an anonymous user/s who made an effort to include a photo collage of the individuals referred to in the conversation. The leaked recording is available hereand has been viewed 485,122.
The State Department spokeswoman was asked if this call is “an authentic recording of an authentic conversation between Assistant Secretary Nuland and Ambassador Pyatt?” As can be expected, Ms. Psaki replied:
“Well, I’m not going to confirm or outline details. I understand there are a lot of reports out there and there’s a recording out there, but I’m not going to confirm private diplomatic conversations.”
QUESTION: As related to Assistant Secretary Nuland’s comments about the European Union, do – are the United States and the EU on the same page on what to deal – how to deal with the situation in Ukraine and how best to resolve the crisis? MS. PSAKI: Well, let me first say, obviously, we work incredibly closely with the EU and with representatives of the EU, and Assistant Secretary Nuland certainly does as it relates to Ukraine. And she’s been in close contact with EU High Representative Ashton. Also, let me convey that she has been in contact with her EU counterparts, and of course, has apologized. But — QUESTION: What did she apologize for? MS. PSAKI: For these reported comments, of course. QUESTION: So you’re not confirming that the comments are accurate? She’s — MS. PSAKI: I’m just not going to speak to a private diplomatic conversation, Arshad, but I’m obviously speaking to the content of the reports.
Ukraine’s Security Service has reportedly declined to comment on a leaked recording of this telephone conversation.
The Guardian says that Germany condemns the comments made by Assistant Secretary Nuland:
The German spokeswoman Christiane Wirtz said Merkel appreciated the work of Catherine Ashton, the EU’s foreign policy chief, who had tried to mediate between the Ukrainian president, Viktor Yanukovych, and protesters who have taken to the streets. “The chancellor finds these remarks totally unacceptable and wants to emphasise that Mrs Ashton is doing an outstanding job,” Wirtz said.
Meanwhile, Polish Minister of Foreign Affairs Radosław Sikorski tweeted his support:
In Kiev, Assistant Secretary Nuland also refused to comment “on a private diplomatic conversation” except to say “It was pretty impressive tradecraft. [The] audio quality was very good.” And so here we are:
And as to those who are shocked, just shocked, by the U.S. attempt to manipulate Ukraine’s opposition, this is exactly what diplomats at higher levels try to do: All the foreign ambassadors in Washington worth their pensions have salty, Machiavellian conversations with their superiors and colleagues about how to shape votes of the U.S. Congress. If you’re an American, be glad that pros such as Nuland are on the job, and hope that your other diplomats aren’t sitting around munching cucumber sandwiches in between demarches.
That said, here is one scandal that this intercepted call does point to, however: Were Nuland and Pyatt speaking, as they should have been, on the kind of encrypted phone designed for such discussions? If not, that’s a major diplo-no-no. If they were, and some foreign power still managed to crack the code, then Uncle Sam needs to invest in some new phones ASAP.
This is a congressional hearing just waiting to happen. Where did you learn such language? Who did what, where, when with these phones and how come you did not know that you were bugged? Congress is always curious about those things.
Now, please do us a favor and stop sending us hate mail for Ms. Nuland.
The Cable’s Josh Rogin points out that the government shutdown would mean soldiers stop getting paid:
In the event of a shutdown, all uniformed military personnel would continue to work but would stop receiving paychecks, an official familiar with the government’s planning told The Cable. As April 8 falls in the middle of the Defense Department’s two-week pay period, military personnel would actually receive a paycheck totaling half the normal amount. A large number of Pentagon civilians would be furloughed without pay for the duration of the shutdown. Support structures for military families, such as military schools, would remain open. When the shutdown ends, the soldiers would get their back pay but the civilians might not.
Most personnel at U.S. foreign missions would be retained, the official said, although about two-thirds of the State Department and USAID staff in Washington would be furloughed. Non-emergency passport services for Americans would also likely be suspended. Up to three-quarters of the staff at the Office of the U.S. Trade Representative would be sent home without pay.
U.S. diplomats and military officials would still be able to travel for important meetings, but “it will be a much, much, much tougher standard,” the official said, explaining that travel would be approved only “if it is integral to the foreign relations and safety and security of the country.”
The shutdown would also impact government organizations that help American companies do business abroad. For example, the Export-Import Bank would stop approving new loan guarantees or insurance policies, the official said, which could cost American exporters $2 billion to $4 billion each month in income and jeopardize deals already in progress.
Also from The Cable guy, about that bit on Congressmen still getting paid during the shutdown:
A senior administration official confirmed to The Cable that even if the taps are shut off, all Congressmen will later be reinbursed their entire salaries no matter how long the shutdown lasts. Staffers who are deemed essential enough to keep working through the crisis could also get paid, but most will be sent home, without pay for the forced leave.
On the conference call, the officials confirmed The Cable’s report that uniformed members of the military will not get paid during the shutdown, although they will get the money back later (not with interest). The officials also confirmed that the vast majority of Defense Department, State Department, and USAID civilians would be furloughed, as well as most White House staff.
“We expect that a significant number of DOD employees, unfortunately, would be furloughed during this shutdown,” the official said.
LA Times reported on what happens to law enforcement folks and the military and who gets paid when:
Federal law enforcement agencies would be up and running, and many in the military would still be working. Those employees, however, wouldn’t be paid for their work until a bill is passed.
“They will be paid once we have money again to pay them,” the first senior administration official said.
If a shutdown lasts only a few days, most in the military would receive their full paycheck April 15, officials said. But if a shutdown lasts beyond the mid-April pay period, they would get about half of their check on April 15 and have to wait until the next pay period for the rest.
The burden on military families, at a time when troops are deployed on three fronts, was a pointed reminder of how a 2011 shutdown could be markedly different from its infamous predecessors in 1995 and 1996.
The Senate had passed a stand alone bill that precludes paychecks and retroactive pay to lawmakers and the President in the event of a shutdown but the House of Reps has continued to danced around this paycheck issue and has refused to even consider a stand alone bill.
Voters elect politicians who appears to be uncompromising. But what get things done is when our elected representatives actually do the hard work of hammering out a compromise that is acceptable to most of their constituents, not just to a tiny, loud, fraction of ideologues.
So if politicians are actually conducting negotiation in honest to goodness effort beyond old politics and ideology, then let’s follow the money. If they don’t get paid, we’d know that they won’t be able to pay their bills like regular people working for Uncle Sam. We know that they are doing their darn best otherwise they, too, won’t get paid.
Lets call their congressional paychecks innocent hostages of our times. But they’d have more credibility when they talk about sacrifices and all.
But if our representatives get their paychecks while 800,000 feds and I don’t know how many soldiers suffers the consequences of their juvenile antics, what does that tell us about our elected representatives?
Simply that they can’t do their jobs. And that they are frankly, incompetent at what they were elected to do but most competent at looking after their own self interest.
In any functioning democracies, elected representatives have to learn to compromise. Only dictators get 101% of what they want. Haven’t they learned that in their basic civics class?
And here’s the other thing that is just supremely poor taste –not only are members of Congress exempted from the furloughs and continue to earn their paychecks during a shutdown, they also get to designate their staffers as essential employees.
Politico reported that about 800,000 federal employees will have to stay home if the government shuts down, but Rep. Darrell Issa’s staff won’t be among them.
The California Republican said he’ll use his congressional prerogative to keep his House Oversight and Government Reform Committee staff at work. Congressional offices can declare that their employees are necessary to fulfill constitutional responsibilities — which can cover pretty much anything under the sun — and that’s what Issa (@DarrellIssa) tweeted that he’ll do.
“If gov’t shuts down, we won’t. I believe those who choose to come into work fall under my Constitutional arm. Accountability must continue.”
Okay so — they’ll be holding hearings while their witnesses are in furloughs? Just swell!
“U.S. military officials sent a medical team to a remote outpost in southern Afghanistan this week to take blood samples from members of an Army unit after a soldier in the unit died from an Ebola-like virus.”
The report says that Sgt. Robert David Gordon, 22, from River Falls, Ala., died Sept. 16 from what turned out to be Crimean-Congo hemorrhagic fever after he was bitten by a tick.Read the whole thing here.
According to the CDC, the Crimean-Congo hemorrhagic fever (CCHF) is caused by infection with a tick-borne virus (Nairovirus) in the family Bunyaviridae. The disease was first characterized in the Crimea in 1944 and given the name Crimean hemorrhagic fever. It was then later recognized in 1969 as the cause of illness in the Congo, thus resulting in the current name of the disease.
Symptoms according to the CDC factsheet:
The onset of CCHF is sudden, with initial signs and symptoms including headache, high fever, back pain, joint pain, stomach pain, and vomiting. Red eyes, a flushed face, a red throat, and petechiae (red spots) on the palate are common. Symptoms may also include jaundice, and in severe cases, changes in mood and sensory perception. As the illness progresses, large areas of severe bruising, severe nosebleeds, and uncontrolled bleeding at injection sites can be seen, beginning on about the fourth day of illness and lasting for about two weeks.
The CDC also says:
There is no safe and effective vaccine widely available for human use.
Fatality rates in hospitalized patients have ranged from 9% to as high as 50%.
Insect repellants containing DEET (N, N-diethyl-m-toluamide) are the most effective in warding off ticks.
The WHO info on CCHF indicates that the length of the incubation period for the illness appears to depend on the mode of acquisition of the virus. Following infection via tick bite, the incubation period is usually one to three days, with a maximum of nine days. The incubation period following contact with infected blood or tissues is usually five to six days, with a documented maximum of 13 days.
This is actually not the first time that CCHF was found in Afghanistan. According to the Federation of American Scientists, 41 deathsfrom “a form of hemorrhagic fever”were reported in eastern Afghanistan in 2002. In August 2008, the WHO reported a total of 19 CCHF cases with 5 deaths in the Herat region. Click here for info on previous CCHF outbreaks in the area from the International Society for Infectious Disease.
If you’re heading out that way, don’t forget to pack some DEET insect repellants.
At the DPB yesterday, a reporter inquired about provisions for Foreign Service personnel in the H1N1 outbreak:
QUESTION: In regard to the H1N1 outbreak, what provisions are being made by the State Department on behalf of Foreign Service Officers serving at posts overseas? Are vaccines being made available to those serving overseas?
ANSWER: The Office of Medical Services expects the H1N1 vaccine will be available to most of our overseas missions sometime in December.
Because of the extremely limited amounts of vaccine available to the Department of State through the Centers for Disease Control and Prevention, the Office of Medical Services is purchasing additional H1N1 vaccine doses from the Department of Defense for shipment overseas to try to address our needs.
The Office of Medical Services already has shipped its initial allotments of the vaccine to Baghdad and Kabul where employees live in barrack-like conditions. The next distribution priority is to hardship posts where local medical care is inadequate, and these shipments have begun.
H1N1 remains sensitive to Tamiflu and Relenza, therefore all posts overseas have been stocked with enough of these anti-viral drugs to treat all individuals at post.
* * *
Would you zap me an email if you don’t get it by December? Excerpt below from U.S. Government Pandemic Policy for Americans Abroad, in case you have not seen it:
It is U.S. Government policy for all overseas employees under Chief of Mission authority and their accompanying dependents to plan for the possibility that they will remain abroad during a severe pandemic. Information for both official and private Americans on how to prepare for this possibility is contained in the flyer “ Options During a Pandemic ,” which urges Americans to maintain adequate provisions for a pandemic wave or waves that could last from two to twelve weeks.
Once the World Health Organization (WHO) confirms a severe pandemic, American citizens (including non-emergency government personnel and their dependents, as well as private citizens) who are residing or traveling overseas should consider returning to the United States while commercial travel options are still available. Americans will be permitted to re-enter the United States, although the U.S. Department of Health and Human Services/Centers for Disease Control and Prevention (HHS/CDC) may quarantine or isolate incoming travelers, depending on their health status and whether they are traveling from or through an area affected by pandemic influenza.
In the event of a severe pandemic, non-emergency U.S. Government employees and all dependents in affected areas will be encouraged to return to the United States while commercial transportation is still available. U.S. Government employees who return to the United States will be expected to work there during the pandemic unless they take leave. Private American citizens should make an informed decision: either remain abroad to wait out the pandemic, as noted above, or return to the United States while this option still exists. Any American (whether overseas in a private capacity or a U.S. Government employee or dependent) who chooses not to return to the United States via commercial means might have to remain abroad for the duration of the pandemic if transportation is disrupted or borders close. Americans should be aware that only in cases of a complete breakdown in civil order within a country will the U.S. Government consider a U.S. Government-sponsored evacuation operation.