The Great Firewall of State Bites, State/IRM Now Considers Diplopundit “Suspicious.” Humph!

Posted: 11:43 am EDT

 

The cornerstone of the 21st century statecraft policy agenda is Internet freedom. The policy contains three fundamental elements: the human rights of free speech, press, and assembly in cyberspace; open markets for digital goods and services to foster innovation, investment, and economic opportunity; and the freedom to connect—promoting access to connection technologies around the world. A third of the world’s population, even if they have access, live under governments that block content, censor speech, conduct invasive mass surveillance and curb the potential of the Internet as an engine of free speech and commerce.

— 21st Century Statecraft
U.S. Department of State

 

We’ve made references in this blog about the Great Firewall of State, most recently, when we blogged about the FS promotion stats on race and gender (see 2014 Foreign Service Promotion Results By Gender & Race Still Behind the Great Firewall of State),  What we did not realize is that there is an entire operation at the State Department running the firewall operations from Annex SA-9.  It is run by the Firewall Branch of the Bureau of Information Resource Management, Operations,  Office of Enterprise Network Management, Perimeter Security Division (IRM/OPS/ENM/PSD/FWB).

Sometime this week, some folks apparently were no longer able to access this blog from the State Department’s OpenNet.  OpenNet is the Sensitive but Unclassified (SBU) network in the Department. It provides access to standard desktop applications, such as word processing, e-mail, and Internet browsing, and supports a battery of custom Department software solutions and database management systems.

At this time, we believe that the block is not agency-wide and appears to affect only certain bureaus.  Not sure how that works. We understand that some employees have submitted “unblock requests” to the State Department’s Firewall Operations Branch and were reportedly told that http://www.diplopundit.com/ has been categorized as “Suspicious.”

via giphy.com

Holy moly macaroni!

We don’t know what constitute “suspicious” but apparently, under State’s Internet policy, this gives the agency the right to block State Department readers from connecting to this blog and reading its content.

But … but … this is the blog’s 8th year of operation and State has now just decreed that this blog is “suspicious”? Just for the record, this blog is hosted by WordPress, and supported by the wonderful people of Automattic. Apparently, the State Department’s DipNote also uses WordPress. Well, now that’s a tad awkward, hey?

Unless …

Was it something we wrote? Was it about the journalists who ran out of undies? NSFW? Nah, that couldn’t be it.   Was it about the petty little beaver? Um, seriously? Maybe that nugget about the aerial eradication in Colombia was upsetting? Pardon me, it’s not like we’re asking folks to drink the herbicide. Come again? You have no expectation of privacy when using the OpenNet? Well, can you blink three times when we hit the right note?

What should we call our State Department that’s quick to criticize foreign governments for blocking internet content for their nationals then turns around and blocks internet content for its employees?

Wass that?  The right hand does not know what the left hand is doing? Blink. Blink. Blink.

We sent a couple emails to the IRM shop — cio@state.gov and Dr. Glen H. Johnson, the senior official in charge of IRM ops asking what’s going on.  It seems the emails were chewed to bits, and we haven’t heard anything back.  Looking for Vanguard contractors to blame? Blink.Blink.Blink.  We’ll update if we hear anything more.

#

State Dept Awards $2.8M “High Availability and Disaster Recovery Services” IT Contract to VMware

Posted: 12:53 am EDT

 

On March 31, 2015, the State Department awarded a $2.8 million “High Availability and Disaster Recovery Services” contract to VMware.  The contract awarded on behalf of the Bureau of Information Resource Management, Operations, Systems Integration Office, Enterprise Server Operations Center or IRM/OPS/SIO/ESOC is for 12 months, and appears to be a modification of a prior task order.  The J&A document posted online justifying “other than full competition” indicates “only one source capable” in handwritten notation. “Persistent security concerns,” “changing strategic landscape” and  “heightened vulnerability” all appear in the limited source justification for the award.  VMware is located in Palo Alto, CA and Reston, VA.

click for larger view

click for larger view

 #

 

Dear USAID OIG — That Nonprofit Contractor Mess Really Needs a Fact Sheet

Posted: 1:23  am EDT

 

.

.

.

.

.

We’ve used the USAID OIG website but it does not have a smart nor responsive search function. We wanted to know how many inspections, audits, whatev reports the Office of the Inspector General at USAID did on IRD over the years.  If they were rigorous in their oversight and USAID and  the State Department did not do anything about it, that is an important component to this story.  And if that is true, we wanted to see just how rigorous based on the reports the oversight office put out through the years, because how else can we tell but by the number and quality of their output?

We sent a direct message to USAID OIG via Twitter and we got a response back:

.
For specific inquiries, please contact our office directly http://oig.usaid.gov/content/contact-usaid-oig

.

You click on that link and you’re told that “for media or general information inquiries, contact the OIG’s Immediate Office by mail, telephone, or fax. Whoa!  The Immediate Office, apparently, is not immediate enough.

.

Late last year, following a Washington Post report on a USAID program in Pakistan, USAID OIG released (pdf) a statement with the following:

OIG is committed to providing products and information that are responsive to the needs of external customers and stakeholders. In responding to questions posed by Members of Congress and congressional staff, OIG has always endeavored to provide complete and accurate information based on the documentation and information available to us.

This is USAID’s largest nonprofit contractor.  According to WaPo, USAID suspended IRD this past January from receiving any more federal work. The suspension came in the wake of allegations of misspending highlighted in a Post investigation in May 2014.  USAID told the Post that they are cracking down on contractors who misspend tax money.

Hookay. So let’s start with finding out what type of oversight USAID OIG provided on IRD contracts since 2006. This is one time when those USAID OIG Fact Sheets would really be helpful.

#

 

Related items:

USAID Suspends Big Contractor IRD: What Took So Long? (NonProfit Quarterly)

Doing well by doing good: The high price of working in war zones (WaPo, May 2014)

 

Munns v. Kerry: Court Dismisses Suit Challenging Policies on Private Security Contractors in Iraq

Posted: 12:30 am EDT

 

WaPo covered the ambushed and abduction of  four Americans and an Austrian employed by Crescent Security Group, a small private security firm in Iraq in July 2007.  In March 2008, U.S. authorities were reported to be in possession of five severed fingers, four of which belong to private security contractors.  In May 2008, the FBI identified the remains of the kidnapped contractors. This case was originally filed on March 22, 2010, Munns et al v. Clinton et al; case number 2:2010cv00681.

Via Opinion from the Court of Appeals for the Ninth Circuit, filed on Mar 20, 2015 (pdf):

Summary:

The panel affirmed the district court’s dismissal of the plaintiffs’ equitable claims due to lack of standing and their federal benefits claims due to lack of jurisdiction, and vacated the district court’s dismissal of the due process and takings claims for withheld back pay and insurance proceeds in an action brought against United States government officials by family members and a coworker of three Americans who were kidnapped and killed while providing contract security services during the United States military occupation of Iraq.

Opinion:

This case arises from the kidnappings and brutal killings of three Americans who were providing contract security services during the United States military occupation of Iraq. The plaintiffs, who include family members and a former coworker of these three men, brought suit against United States government officials to challenge policies governing the supervision of private contractors and the response to kidnappings of American citizens in Iraq (“policy claims”). They also claim the government is withholding back pay, life insurance proceeds and government benefits owed to the families of the deceased contractors (“monetary claims”).

The district court dismissed the policy claims for lack of standing and for presenting nonjusticiable political questions. It dismissed the monetary claims for failure to establish a waiver of the government’s sovereign immunity from suits for damages and for failure to state a claim for which relief could be granted. We hold that the plaintiffs have not shown they are likely to be harmed in the future by the challenged policies. They therefore lack standing to seek prospective declaratory and injunctive relief regarding those policies. We further hold that the plaintiffs have failed to allege a governmental waiver of sovereign immunity that would confer jurisdiction in the district court over their monetary claims. Finally, we hold that the United States Court of Federal Claims has jurisdiction over the plaintiffs’ claims for withheld back pay and insurance proceeds, and we direct the district court to transfer those claims under 28 U.S.C. § 1631. We thus affirm in part and vacate in part and remand.

Background:

In November 2006, while working for Crescent, contractors Munns, Young and Cote were assigned to guard a 46-truck convoy traveling from Kuwait to southern Iraq. The plaintiffs allege that on the day of the convoy, Crescent issued the men substandard military equipment and ordered other security team members not to accompany them on the convoy, and that Iraqi security team members slated to join the convoy failed to show up for work, leaving only seven contractors to guard the convoy. When the convoy stopped at an Iraqi police checkpoint, 10 armed men approached and, along with the Iraqi police, took five of the contractors captive, including Munns, Young and Cote. The men were held for over a year, until their kidnappers brutally executed them sometime in 2008.

The plaintiffs trace the contractors’ kidnappings and murders to Crescent’s failure to adequately prepare and supervise its personnel in Iraq. They allege Crescent’s deficient conduct was “officially sanctioned” by the Secretary of State through an unlawful order issued by the Coalition Provisional Authority (CPA) overseeing the U.S. occupation. CPA Order 17 allegedly gave “blanket immunity [to contractors] from all prosecution,” granting them a “license to kill” with impunity and permitting contractors to “circumvent the authority of Congress, the Courts, and the Constitution.”2 Additionally, the plaintiffs say they heard rumors that CPA Order 17, and the consequent lawless behavior of some security contractors, may have been the motivation behind the kidnappings.

Circuit Judge Reinhardt:

The more troubling and painful question is what the role of our government should be if and when terrorist groups like ISIS or Al Queda capture an American citizen and hold him hostage, and whether the government may, or should, impose any limitation on the rights of the citizen’s family or friends to communicate with that group or pay a ransom. It is significant that the government has told this court that currently there are no policies preventing private individuals from making efforts to secure the release of relatives who are held captive abroad. More important however from the standpoint of the legal rules that govern us, the parties bringing the action – relatives of contractors’ employees “brutally killed,” as Judge Fisher puts it, in the Middle East – seek no damages resulting from that policy but simply seek to have the policy declared unlawful. They ask that the government be enjoined from implementing the policy in the future. Again, even assuming that contrary to what the government tells us, such a policy exists, we cannot under well established legal rules render a decision that will be of no immediate benefit to the individuals bringing the lawsuit. Because the plaintiffs have no relatives currently in the Middle East, or currently in greater danger from terrorist groups than any of the rest of us, we again face only a hypothetical question – the kind that courts do not answer

Read in full online here or download the opinion in pdf file here.

 

Related item:

7 FAM 1820 Hostage Taking and Kidnapping (pdf)

#

State Dept Seeks Organizational Shrink to Assist in Foreign Service Selection Procedures

— Domani Spero

 

On September 12, the State Department published a solicitation via FedBiz.gov seeking “a certified industrial and organizational psychologist to provide advice, assistance and support for Foreign Service selection procedures.” 

Extracted from the FedBiz documents:

The Foreign Service Act of 1980 tasks the U.S. Department of State (the Department), and the Board of Examiners (BEX) specifically, with the responsibility for the evaluation and selection of candidates for the Foreign Service. The Bureau of Human Resources, Office of Recruitment, Examination and Employment, Board of Examiners (HR/REE/BEX) oversees these examinations, including the Foreign Service Officer Test (FSOT), Qualifications Evaluation Panel (QEP), Foreign Service Oral Assessment (FSOA), and selection procedures for Foreign Service generalists, specialists and limited non-career appointments. HR/REE/BEX is seeking a certified industrial and organizational psychologist to provide necessary advice and assistance in support of the Foreign Service Selection Process.

The contractor will assist the Department in ensuring that all examinations for Foreign Service generalists, specialists and limited non career appointments have been professionally validated and constitute a reliable means of identifying those applicants who show the greatest possibility of success in the Foreign Service. The successful contractor will provide consultative and analytical services as requested including formulating program alternatives and operational support for successful implementation of any revisions to testing and hiring procedures.

 

According to the solicitation, the organizational shrink, formally known as the contractor here shall perform the following work, as assigned by the Department:

1. Assist in evaluating the extent to which the generalist, specialist and limited non-career appointment hiring programs are effective in meeting the needs of the Foreign Service.

2. Work with the contractor who develops and administers the FSOT to review test components, as directed by HR/REE/BEX, including redesign of sections where requested; review and advise HR/REE/BEX on any revisions to the FSOT prior to their inclusion in the Department’s hiring process.

3. Attend, as the Department’s expert contractor, meetings of the Board of Examiners for the Foreign Service, established pursuant to Section 211 of the Foreign Service Act of 1980, as amended. At the request of HR/REE/BEX, attend meetings with the Director General that involve discussion of Foreign Service selection procedures.

4. Provide advice on the procedures and training involved in the generalist, specialist and limited non-career appointment Qualifications Evaluation Panel (QEP), and assess the validity of QEP results.

5. Provide advice on the content validation of the Foreign Service Oral Assessment process (FSOA) and prepare FSOA validation reports for use by the Bureau of Human Resources.

6. Work with subject matter experts to create, review and revise all Foreign Service Selection Process assessments (QEPs, interviews, cases, competency tests, etc.). Provide programming and administrative support for online competency assessments.

7. Compile, manage, and report on assessment data. Validate assessments by conducting studies to ensure compliance with legal and professional testing guidelines. Analyze assessment data for statistical quality, adverse impact, and other purposes (e.g., answering questions from management).

8. Create feedback reports for assessors and management. Conduct special studies on the assessments (e.g., passing rates, comparing equivalence, faking, etc.) as requested. To include documenting all validation evidence, analyses, and special studies in technical reports.

9. Monitor all aspects of the implementation of the assessments and make continuous improvements.

10. Provide advice on alternate methods of entry to the FSOA (other than the FSOT) and assess the validity of these programs.

11. Evaluate on a recurring basis the Department’s recruiting and testing procedures, and advise HR/REE on how best to meet its hiring objectives and ensure the validity of any changes made to the examination processes.

12. Develop an online practice FSOT that potential candidates can use to assess their chances of passing the FSOT. Provide support to HR/REE for the Department’s recruiting mobile application.

13. Provide advice on the Department’s specialist hiring program, including possible examination alternatives; to include remote testing. Review and revise specialist and limited non-career appointment vacancy announcements and questionnaires used for initial screening of applicants.

14. Provide the Department with professional expertise in litigation should there be legal challenges to the FSOT, Oral Assessment,specialist examinations, or selection processes, including through production of requested documentation and service as an expert witness.

15. Provide professional advice and consultation to other HR offices within the Department as requested by HR/REE.

16. Define the mission-critical competencies required of entry-level Foreign Service Officers. Use this information to update the 2007 Job Analysis of Foreign Service Officer Positions

17. Conduct organizational or workforce surveys. To include a survey of generalists and specialists who have participated in the Oral Assessment; Entry-level Officers; and other candidate groups as designated by BEX.

Additionally the contractor should be an expert in psychometrics, the statistical science of psychological measures that are used to comprise knowledge tests and shall be conversant with:

  • The Uniform Guidelines on Employee Selection Procedures (1978) . These guidelines were established by federal agencies in charge of enforcing employment anti-discrimination laws. Among those agencies are the Equal Employment Opportunity Commission, the Office of Federal Contract Compliance, and the Department of Justice.
  • The Principles for the Validation and Use of Personnel Selection Procedures , published by the Society for Industrial and Organizational Psychology.
  • The Civil Rights Act of 1964 {Public Law 88-352 (78 Stat. 241)} prohibits discrimination based on race, color, religion, sex, or national origin.

 

The State Department expects the following deliverables:

  • Based on its observations, the Contractor shall prepare a comprehensive report on generalist and specialist hiring programs, including the FSOT, Foreign Service Oral Assessment and specialist hiring programs, in addition to test-specific reports. The contractor may be required to brief HR/REE/BEX on the findings contained in the report to the Contract Officer’s Representative (COR).
  • The Contractor shall develop and provide in person (not recorded) an up-to-date Oral Assessment training program for assessors in order to ensure consistency among those conducting the oral assessment. Training shall address at a minimum the following elements: orientation to the concept of assessment centers and their role in pre-hire screening, background on the Foreign Service Oral Assessment process, and any revisions made since the last training session.
  • The Contractor shall provide training to BEX on each of the testing exercises that make up the FSOA (see http://www.careers.state.gov) and shall provide detailed guidance on scoring methodologies and anchors. The contractor shall ensure that the training is consistent with professional and legal standards or guidance.
  • The Contractor shall conduct a job analysis of the five Foreign Service Officer career tracks to determine what knowledge, skills, abilities and other characteristics FSOs need to perform their jobs effectively. Based on this analysis, the contractor will update the current blue prints being used by the Department.
  • The contractor shall compile evidence on the validity of the FSOA, and prepare a report summarizing such evidence, including a complete analysis of the demographics of those participating in the FSOA.
  • The contractor shall develop, monitor, provide, and maintain a comprehensive training program for the panel members involved in the generalists qualifications/evaluation/assessment (QEP).
  • As necessary, assist the Department, including its legal counsel, in legal matters pertaining to the FSOT, QEP and Oral Assessment, or other selection procedures established for the Foreign Service generalists and specialists.
  • The contractor will be required to compile a library of materials created pursuant to the contract on the content validation for all FSOTs administered during the contract period. Title to the library of materials compiled by the Contractor for which the Contractor is entitled to be reimbursed under this contract shall pass to and vest in the Government.

A couple of thoughts — this organizational psychologist has the potential to impact the hiring process of the State Department. Two, we are not sure if this is one of the results of the EEOC class action, but the requirement that this contractor provide the Department “with professional expertise in litigation should there be legal challenges to the FSOT, Oral Assessment,specialist examinations, or selection processes” seems to indicate that an expected challenge/s may be in the works.

* * *

 

 

 

 

 

 

Snapshot: Top 30 State Department Contractors (Based on Highest Dollar Amounts)

— Domani Spero

 

According to State/OIG, after several media reports were written about the use of confidentiality agreements that limit the ability of contractor employees to report fraud, waste, or abuse to Inspectors General or other oversight entities, it sent a letter in August 2014 requesting information from the thirty companies which have the highest dollar amount of contracts with the Department of State. The list does not indicate their rank in any particular order:

Screen Shot 2014-09-21

Screen Shot 2014-09-21

 

 

 

 

 

 

 

State Department’s Embassy “Design Excellence” Initiative: Year in Review (Video)

— Domani Spero

 

The State Department’s Bureau of Overseas Buildings Operations has just released a ‘Year in Review 2013-2014′ video, primarily highlighting the new embassies built under its “design excellence”initiative. You will note that some of the projects in this video have been completed while others like the New London Embassy, and those buildings in artist’s renderings are still undergoing construction or in the early phases of the projects  and won’t be completed for a few more years.

The Bureau of Overseas Buildings Operations (OBO) “sets worldwide priorities for the design, construction, acquisition, maintenance, use, and sale of real properties and the use of sales proceeds” for the State Department. The bureau has recently caught congressional attention with its New London Embassy project and its “design excellence” initiative. See Congress to State Dept: We Want All Your Stuff on New London Embassy Except Paperclips and New Embassy Construction Hearing: Witnesses Not Invited, and What About the Blast-Proof Glass?

We understand that the bureau is still working on providing Congress with the documents requested during the latest congressional hearing. Congress won’t be back in session until September 8, and then, it will only conduct business for a couple of weeks before it runs out again.  Nonetheless, we are hearing that there may be personnel shuffles at the bureau in the offing.  We’ll update when we know more.

 

Related items:

-05/31/11   Compliance Follow-up Review of the Bureau of Overseas Buildings Operations (ISP-C-11-26)  [2452 Kb]  Posted June 8, 2011

-08/30/08   Bureau of Overseas Buildings Operations (ISP-I-08-34) Aug 2008  [1846 Kb]

* * *

 

 

 

 

 

 

 

 

 

 

USAID’s undercover Latin youth — whose brainchild is this, pray tell (video)

— Domani Spero

 

Read more here. Documents about this program is at http://apne.ws/UxJ05x.

Whose brainchild is this, pray tell.

Alan Gross, the  65-year-old American citizen mentioned in this article has been imprisoned in Cuba since 2009. His family has mounted a petition demanding Mr. Gross’ “immediate release” and  that “the Cuban and U.S. governments sit down and resolve Alan’s case.”

This morning, USAID released a statement about what it calls, the AP’s “sensational claims,”excerpt below:

Congress funds democracy programming in Cuba to empower Cubans to access more information and strengthen civil society. USAID makes information about its Cuba programs available publicly at foreignassistance.gov. This work is not secret, it is not covert, nor is it undercover. Instead, it is important to our mission to support universal values, end extreme poverty and promote resilient, democratic societies. Chief among those universal values are the right to speak freely, assemble and associate without fear, and freely elect political leaders. Sadly, the Cuban people and many others in the global community continue to be denied these basic rights.

One paragraph in the article captures the purpose of these and many civil society programs, which is to empower citizens to “tackle a community or social problem, win a ‘small victory’ and ultimately realize that they could be the masters of their own destiny.” But the story then goes on to make sensational claims against aid workers for supporting civil society programs and striving to give voice to these democratic aspirations. This is wrong.

USAID remains committed to balancing the realities of working in closed societies–particularly in places where we do not have a USAID mission and governments are hostile to U.S. assistance–with our commitment to transparency, and we continuously balance our commitment to transparency with the need for discretion in repressive environments. In the end, USAID’s goal is to continue to support democracy, governance and human rights activities in multiple settings, while providing the maximum transparency possible given the specific circumstances.

A couple of items from that USAID statement: 1)   “the Cuban people and many others in the global community,” does that mean this happened in Cuba and elsewhere?; 2) “with our commitment to transparency” — USAID’s Cuba programs data available publicly at foreignassistance.gov only covers FY2013 and 2014 and not the years covered by the AP report. USAID also would not tell the AP how much the Costa Rica-based program cost.

These young “aid workers” from Venezuela, Costa Rica and Peru sent to Cuba could have been arrested and jailed for 10 years for the work they did for USAID, and the agency would have been able to claim that these are not USG employees.  The US has not been able to effect the release of USAID contractor Alan Gross, would it be any more successful intervening for the release of foreign nationals who are not?  Also, the notion that you can run democracy promotion operations like this in certain parts of the world and that it will not have a dangerous blowback against USAID employees advancing development work in other parts of the world, is frankly, lunacy.

Does USAID have a scenario planned for what happens after a ‘Cuban Spring’unfolds in Cuba? Is it publicly available at fomentingchange.gov?

Just a reminder, the nominee for USAID OIG, in case you’re wondering has been waiting for Senate confirmation since July 2013 (see Officially In: Michael G. Carroll – From Deputy IG to USAID/OIG).

* * *

 

 

 

 

 

State Dept Answers FAQ on Ongoing Visa and Passport Database Performance Issues

— Domani Spero

 

Yesterday, we posted about the troubled Consular Consolidated Database (CCD) (see State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience).  During the Daily Press Briefing, yesterday, the State Department officially stated that it believed the root cause of the problem was “a combination of software optimization and hardware compatibility issues.” According to the deputy spokesperson, the servers are getting back online but that they are coming back in a queue and that fixes are not being done on a country-by-country basis. And by the way, it’s not just the peak summer travel season, there’s also the Africa Summit in D.C. next week.

“Obviously, there’s actually a huge crush right now because of the Africa Leaders Summit, so obviously that’s a huge priority for us to make sure everybody gets their visas for the Africa Leaders Summit. We do believe that a vast majority of the travelers who have applied for visas for the summit have been issued.”

CA’s FB folks have been regularly answering questions from angry complaints posted on its Facebook page and have announced that they will continue to monitor and respond to consular clients at 9:00 EDT tomorrow, Thursday, July 31.

Late yesterday, the Bureau of Consular Affairs also posted a new Frequently Asked Questions on Facebook and on its website (not easily accessible from the main visa page) concerning the CCD performance issues and the steps taken to address those issues. Perhaps the most surprising is that its back-up capability and redundancy built into the CCD were both affected killed by the upgrade that hobbled the system.  Something to look forward to by end of calendar year — CA is upgrading the CCD to a newer version of the Oracle commercial database software and that plan includes establishing two fully redundant systems. We are republishing the FAQ in full below.

Screen Shot 2014-07-31 at 12.07.05 AM

Information Regarding Ongoing Consular Consolidated Database (CCD) Performance Issues and Steps Taken| JULY 30, 2014

The Department of State continues to work to restore our visa system to full functionality.

We anticipate it will take weeks to resume full visa processing capacity.

We continue to prioritize immigrant visas, including adoption cases.  So far, we have been able to issue most cases with few delays.

Nearly all passports are currently being issued within our customer service standards, despite the system problems.

We are able to issue passports for emergency travel.

Frequently Asked Questions

Q:  What caused the system performance issues?  Hardware, software, or both?  Details?

On July 20, to improve overall system performance and address previous intermittent performance issues, we updated software as recommended. Our database began experiencing significant performance issues shortly after this maintenance was performed.

A root cause has not been identified at this time.  Current efforts are focused on bringing the system back to normal operations.  Once that has been accomplished, resources will be applied to determine the root cause.

Q:  What steps did we take to mitigate the performance issues?

Since July 20, our team has worked to restore operations to full capacity.  On July 23, the Consular Consolidated Database (CCD) was brought back online with limited capacity.

The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues.

We are incrementally increasing the number of processed cases as our systems will allow.

Q:  Has the Bureau of Consular Affairs experienced these types of outages in the past?

CA has experienced minor outages in the past, but never of this magnitude.  We have a plan in place to mitigate these occurrences in the future.

Q:  Is the software to blame?  Are contractors at fault?  Why was this allowed to happen?

We have been working to improve our services through upgrades while maintaining existing operations worldwide.  However, we are limited by outdated software and hardware.

Q:  Why did those steps not work?  What’s the next step?

We have not determined why the problems occurred.  We are working with our contractor and the software vendor to address the problems.

We are bringing additional servers online to increase capacity and response time.

Q: Why wasn’t there a back-up server?

There was back-up capability and redundancy built into the system.  However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system.

Q: What steps are being taken prevent this from happening again?

CA has a plan in place to upgrade the CCD to a newer version of the Oracle commercial database software by the end of the calendar year.  We are working to ensure the existing system will remain fully functional until the new database is up and running and thoroughly tested.  The plan includes establishing two fully redundant systems.

Q:  If CA is fee funded, why can’t it build a robust database that doesn’t fail?

The database has grown dramatically, in both quantity of data and functionality, and vastly improved border security.  In addition to checking names against databases, we review fingerprints and perform facial recognition.

We are working towards modernization of our software, hardware, and infrastructure.  Demand for our services outpaced our modernization efforts.

Consular Affairs has, and has had, a redundant system.  However, the upgrade affected not only our current processing capability, but also our ability to use our redundant system.  This is one of the issues we are urgently addressing now.

Q:  What do I need to know if I’m a passport applicant?

Almost all passports are currently being issued within our customer service standards, despite the system problems.

We are able to issue passports for emergency travel.

Q:  What do I need to know if I’m a visa applicant?

Visa applicants they can expect delays as we process pending cases.  We remain able to quickly process emergency cases to completion.

We are working urgently to correct the problem to avoid further inconveniencing travelers.

We are posting updates to the visa page of travel.state.gov, and our embassies and consulates overseas are communicating with visa applicants.

In addition to communicating through our websites, e-mail, and letters, we are also reaching out to applicants via Facebook and other social media sites, such as Weibo, to relay the latest information.

Q:  Why hasn’t the Department been more forthcoming until now?

We have experienced CCD outages in the past, but they have never disrupted our ability to perform consular tasks at this magnitude.

We informed the public as soon as it was apparent there was not a quick fix to bring the CCD back to normal operating capacity, and are briefing Congressional staffers regularly.

Q:  What is the outlook for Non-immigrant visas?  When do we estimate the backlog will be processed?

That will depend on a number of factors.  Current efforts are focused on bringing the system back to normal operations.

We must also continue processing new requests.  We are committed to reducing the number of pending visa cases as quickly as possible, but we want applicants to know that we will continue to be operating at less than optimal efficiency until the system is restored to full functionality.

Q:  Is the Department going to reimburse applicants who missed flights/canceled weddings/missed funerals?

We sincerely regret any delays, inconvenience, or expense that applicants have may have incurred due to the CCD performance issues.

While it might be of little solace to those who have experienced hardship, we are always very careful to tell travelers NOT to make travel plans until they have a visa in hand.  Even when the CCD is operating normally, there may be delays in printing visas.

The Department does not have the authority to reimburse applicants for personal travel, nor do we include these costs when calculating our fees.  The Department cannot refund visa fees except in the specific circumstances set out in our regulations.

Q:  What impact will this have on SIVs?

We have the highest respect for the men and women who take enormous risks in supporting our military and civilian personnel.  We are committed to helping those who have helped us.  While issuances of Special Immigrant Visas (SIVs) to Afghans and Iraqis have been impacted, as have visa issuances around the world, SIV processing continues and remains a high priority.

Q: How is this impacting student visas?  They are scheduled to start the fall semester soon.

We are committed to issuing visas to all qualified students and exchange visitors.  Issuance of student and exchange visitor visas has been impacted in the past few days, but visa processing continues.

We understand the importance to international students and exchange visitors, their families, and their U.S. host institutions of timely visa issuance in order to facilitate travel and to ensure all students and exchange visitors may begin their programs on time.

Q: What about situations where the student won’t arrive to school on time?

Students should contact their educational institution’s Designated School Official (F and M visas) or designated U.S. sponsor’s Responsible Officer (J visas) and discuss with them what arrangements they can provide for you to begin your program after the start date on your Form I-20 (F and M visas) or Form DS 2019 (J visas), should such a circumstance become necessary.

Q:  Will this have any impact on the Diversity Visa program in September?

While issuances of all immigrant visas, including diversity visas, have been impacted in the past few days, IV processing continues and remains a high priority.  The Department expects to have used all numbers for DV-2014 when the program year ends on September 30, 2014.

Q:  What impact do we anticipate this will have on the U.S. economy?

Tourism and students have a major impact on our economy.  Last year, it was estimated that international visitors spent $180.7 billion and supported 1.3 million American jobs.  International students contribute $24.7 billion to the U.S. economy through their expenditures on tuition and living expenses, according to the Department of Commerce.

We recognize the significant impact that international travel and tourism has on the U.S. economy, and are taking all possible steps to ensure that the economic impact is minimal.

People traveling under the Visa Waiver Program are not affected at all; nor are those whose previously-issued visas remain valid.

We routinely advise applicants needing new visas to make appointments well in advance of their planned travel, and not to book their travel until they have their printed visas in hand.

The original post is available here.  If CA is reading this, it would be helpful if a link to the FAQ is posted on the main visa page of travel.state.gov and in the News section.  We were only able to find the FAQ from a link provided in Facebook and not from browsing around the travel.state.gov website.

 

 

 

 

 

State Dept’s Critical National Security Database Crashes, Melts Global Travelers’ Patience

— Domani Spero

 

The first announcement about the troubled Consular Consolidated Database (CCD) went out on Wednesday, July 23:

The Department of State Bureau of Consular Affairs is currently experiencing technical problems with our passport/visa system.  This issue is worldwide and is not specific to any particular country, citizenship document, or visa category.  We apologize to applicants who are experiencing delays or are unable to obtain a passport, Consular Report of Birth Abroad, or visa at this time. We are working urgently to correct the problem and expect our system to be fully operational again soon.

The AP reported on July 23 that unspecified glitches have resulted in performance issues since Saturday, which would be July 19.

On July 25, CA announced:” Our visa and passport processing systems are now operational, however they are working at limited capacity. We are still working to correct the problem and expect to be fully operational soon.”

A State Department official speaking on background told us the same day that this issue was not/not caused by  hackers. We were told that the CCD crashed shortly after maintenance was performed and that the root cause of the problem is not yet known.

On July 27, CA released an update:

As of July 27, the Department of State has made continued progress on restoring our system to full functionality. As we restore our ability to print visas, we are prioritizing immigrant cases, including adoptions visas. System engineers are performing maintenance to address the problems we encountered. As system performance improves, we will continue to process visas at U.S. Embassies and Consulates worldwide. We are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.

On July 29, CA posted this on FB:

The Department of State Bureau of Consular Affairs continues to make progress restoring our nonimmigrant visa system to full functionality. Over the weekend, the Department of State implemented system changes aimed at optimizing performance and addressing the challenges we have faced. We are now testing our system capacity to ensure stability. Processing of immigrant visas cases, including adoptions, remains a high priority. Some Embassies and Consulates may temporarily limit or reschedule nonimmigrant visa interview appointments until more system resources become available to process these new applications. We sincerely regret the inconvenience to travelers, and are committed to resolving the problem as soon as possible. Additional updates will be posted to travel.state.gov as more information becomes available.

 

The CA Bureau’s Facebook page has been inundated with comments. There were complaints that at one post the visas were printing fine and then they were not. There were complains from people waiting for visas for adopted kids, for fiancees, for family members, for family waiting at the border, for students anxious to get to their schools, people worried about time running out for diversity visas, applicants with flights already booked, and many more.  One FB commenter writes, “I feel that the problem most people have is not that the system broke, but the lack of clear, meaningful information so people can make appropriate plans.

Other than what the CA Bureau chose to tell us, we cannot pry any substantial detail from official sources.  We, however, understand from sources familiar with the system but not authorized to speak for the bureau that the CCD has been having problems for sometime but it got worse in the last couple weeks.   If you’re familiar with the highs and lows of visa operation, this will not be altogether surprising.  Whatever problems already existed in the system prior to this “glitch” could have easily been exacerbated in July, which is the middle of the peak travel season worldwide. A source working in one of our consular posts confirmed to us that the system is back running, but not at the normal level and that the backlogs are building up. Another source told us that Beijing already had a 15k NIV backlog over the weekend.  We haven’t yet heard what are the backlogs like in mega visa-issuing posts like Brazil, Mexico and India.

We understand that everyone is currently doing all they can to get the process moving, but that some cases are getting through the system, while some are not. No one seems to know why this is happening. These machine readable visas are tied to the system and there are no manual back-ups for processing these cases (more of that below).

 

So who owns CCD?

The Consular Systems and Technology (CA/CST) manages the CCD.  We have previously blogged about its troubled past:

CST is currently headed by a new Director, Greg D Ambrose who reports to the CA Bureau’s Assistant Secretary.  It looks like despite the 2011 OIG recommendation, the CST deputy position remains vacant. We should also note that the  Asst Secretary for Consular Affairs Janice Jacobs retired this past April.  No replacement has been nominated to-date and Michele T. Bond has been Acting Assistant Secretary since Ms. Jacobs’ departure.

Last September, Mr. Ambrose was with FedScoopTV and talked about Consular One, the future of consular IT.

 

CST Just Got a New Data Engineering Contract

In Many 2014, ActioNet, Inc., headquartered in Vienna, Virginia,announced a 5-year task order for data engineering, supporting CST.

ActioNet, Inc. announced today the award of a five (5)-year task order entitled Data Engineering (DE) in support of Department of State (DOS). This task order will provide data engineering and database infrastructure support services necessary for planning, analysis, design, and implementation services for the Bureau of Consular Affairs.  These service also include contract and program management support to ensure that innovation, efficiency, and cost control practices are built into the program. […] The Office of Consular Systems and Technology (CST) within the Bureau develops, deploys and maintains the unclassified and classified IT infrastructures that help execute these missions. The Bureau currently manages over 800 servers worldwide, in order to comply with the fast paced changes inherent to data processing and telecommunications, CST requires that contractor services provide for rapid provisioning of highly experienced and trained individuals with the IT (information technology) backgrounds and the security clearances required of CA’s environment of workstation-based local and wide-area network infrastructures.

Due to limited information available, we don’t know if the new Consular One and/or the new DE contract are related to ongoing issues or if there are hardware issues, given the multiple legacy systems, but we do know that CST has both an impressive and troubled history. Let’s take a look.


Records Growing by the Day

The 2010 Consular Consolidated Database (CCD) Privacy Impact Assessment (PIA) describes (pdf) the CCD as “one of the largest Oracle based data warehouses in the world that holds current and archived data from the Consular Affairs (CA) domestic and post databases around the world.”  According to the PIA, in December 2009, the CCD contained over 100 million visa cases and 75 million photographs, utilizing billions of rows of data, and has a current growth rate of approximately 35 thousand visa cases every day. The 2011 OIG report says that in 2010, the CCD contained over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day.

That was almost four years ago.


A Critical Operational and National Security Database with No Back-Up System?

According to publicly available information, the CCD’s chief functions are 1) to support data delivery to approved applications via industry-standard Web Service queries, 2) provide users with easy-to-use data entry interfaces to CCD, and 3) allow emergency recovery of post databases.  The CCD also serves as a gateway to IDENT and IAFIS fingerprint checking databases, the Department of State Facial Recognition system, and the NameCheck system. It  provides access to passport data in Travel Document Issuance System (TDIS), Passport Lookout Tracking System (PLOTS), and Passport Information Electronic Records System (PIERS).  The OIG says that the CCD serves 11,000 users in the Department and more than 19,000 users in other agencies, primarily the Department of Homeland Security (DHS) and various law enforcement elements, and is accessed more than 120 million times every month.

Given that the CCD is considered “a critical operational and national security database,” there is surprisingly no redundancies or any back-up system.


Resurrect the Standard Register protectograph aka: `Burroughs visas’?

No one is actually suggesting that but when the CCD system is down, there is no manual way to issue a visa. No post can  handprint visas  because security measures prevent consular officers from printing a visa unless it is approved through the database system. Here is a quick history of the handprinted ‘Burroughs visas’ and the machine readable visas via the GPO:

November 18, 1988, mandated the development of a machine-readable travel and identity document to improve border entry and departure control using an automated data-capture system. As a result, the Department developed the Machine Readable Visa, a durable, long-lasting adhesive foil made out of Teslin.

Before MRVs, nonimmigrant visas were issued using a device called a Standard Register protectograph, otherwise known as a Burroughs certifier machine. It produced what was colloquially known as a “Burroughs visa,” an indelible ink impression mechanically stamped directly onto a page in the alien’s passport. Over time, Burroughs machines were gradually replaced by MRV technology, which is now used exclusively by all nonimmigrant visa issuing posts throughout the world.

Burroughs visas contained a space in which a consular employee was required to write the name of the alien to whom the visa was being issued. An alien’s passport might also include family members, such as a spouse, or children, who also had to be listed on the visa. In March 1983, in order to expedite the issuance of nonimmigrant visas and to improve operational efficiency, the Department authorized the use of a “bearer(s)” stamp for certain countries so that consular officers would not have to spend time writing in the applicant’s name (and those of accompanying family members). MRVs, however, must be issued individually to qualified aliens. Consequently, the “bearer”annotation has become obsolete.

The problem with the old Burroughs machine, besides the obvious, was maybe — you run out of ink, the plates are ruined/broken or you need it oiled. We could not remember those breaking down. With the MRV technology, all posts are connected to a central database, and the new machines by themselves cannot issue visas.  Which brings us to the security of that system.

 

Management Alert on Information System Security Program

The State Department PIA says that “To appropriately safeguard the information, numerous management, operational, and technical security controls are in place in accordance with the Federal Information Security Management Act (FISMA) of 2002 and information assurance standards published by the National Institute of Standards and Technology (NIST).” Must be why in November 2013, the Office of the Inspector General issued a Management Alert  for significant and recurring weaknesses found in the State Department’s Information System Security Program over the past three fiscal years (FY 2011-2013).

In 2011, State/OIG also issued a report on CA’s CST division and has, what appears to be a lengthy discussion of the CCD, but almost all of it but a paragraph had been redacted:

Screen Shot 2014-07-30 at 8.40.37 AM

That OIG report also includes a discussion of the Systems Development Life Cycle Process and notes that decision control gates within CST’s SDLC process are weak. It cites a couple of examples where this manifested: 1) the development of the Consular report of Birth Abroad (CRBA) system. “The ownership of development and deployment shifted throughout the process, and the business unit’s requirements were not clearly communicated to the development team. As a result, CST designed and tested the CRBA for a printer that did not match the printer model identified and procured by the business unit;” 2)  the Crisis Task Force application, for which CST was tasked to enhance its Web-facing interaction. “The deployment of this application has been challenged by the lack of project ownership and decision controls, as well as by the incomplete requirements definition. The use of incorrect scripts that were provided by the CM group has further delayed the Crisis Task Force application’s deployment.”

 

If there’s somethin’ strange in your CCD, who ya gonna call? (Glitchbusters!)

The Consular Consolidated Database (CCD) is central to all consular operations. It is run by CST where according to the OIG, “the smooth functioning of every part of the office depends on its contractors.” And because it runs such an important element of U.S. national security systems, if all CST’s contractors, all 850 of them quit, this critical consular data delivery to the State Department and other Federal agencies would screech to a a halt.

To carry out its mandate, CST must provide uninterrupted support to 233 overseas posts, 21 passport agencies, 2 passport processing centers, and other domestic facilities, for a total of 30,000 end users across 16 Federal agencies and in nearly every country. CST faces 24/7/365 service requirements, as any disruption in automated support brings operations to an immediate halt, with very serious implications for travelers and the U.S. image.
[…]
CST is led by a director and is staffed by 68 full-time equivalent (FTE) employees (62 Civil Service and 6 Foreign Service). There are 12 positions (3 Foreign Service and 9 Civil Service) currently vacant. CA recently authorized CST 19 additional FTE positions. There are also more than 850 contractors operating under nearly 30 different contracts. In FY 2010, CST’s annual operating budget was approximately $266 million.

If CCD is compromised for a lengthy period such as the last couple of weeks, what is the back up plan to keep the operation going?  Obviously, none. It’s either down or running under limited or full capacity.  No one we know remember CCD problems persist this long.  Right now, we know from a reliable source that the system is not down, and some cases and going through but — what if the CCD is completely down for two weeks … four weeks … wouldn’t international travel come to a slow stop?

What if CCD goes down indefinitely whether by hardware or software glitch or through malicious penetration by foreign hackers, what happens then?

Currently, it appears nothing can be done but for folks to be patient and wait until the fixes are in.  We know they’re working hard at it but there’s got to be a better way.   Perhaps we can also agree that this has very serious national security implications on top of disgruntled travelers and a grave impact on the U.S. image overseas.

 

 Related items:

May 2011 |  Inspection of The Bureau of Consular Affairs, Office of Consular Systems and Technology (CST) Report Number ISP-I-11-51

-11/30/13   Audit of Department of State Information Security Program (FISMA) (AUD-IT-14-03)  [3610 Kb]  Posted January 29th, 2014

-01/13/14   Management Alert on OIG Findings of Significant, Recurring Weaknesses in Dept of State Info System Security Program (MA-A-0001)  [6298 Kb]  Posted on January 16, 2014

 

 Related articles