Inadequate Oversight of State Dept’s Consolidated Consular Database– Is This WikiLeaks Waiting to Happen, Again?

The State Department OIG has just posted online its Inspection of The Bureau of Consular Affairs, Office of Consular Systems and Technology (CST). (See Report Number ISP-I-11-51, May 2011; pdf file here)

The inspection took place in Washington, DC, between January 18 and March 4, 2011. The reports includes a whole lot of things but below is one of the most striking items:

Oversight of the Consolidated Consular Database

The CCD is the backbone of all consular applications and services and supports domestic and overseas passport and visa activities. The CCD is the repository for data from all the individual consular sections and passport office databases. It uses state-of-the-art technology with agile and robust design for scalability and availability. The CCD is accessed by 11,000 Department users, as well as 19,000 users in other agencies (for example, DHS, Federal Bureau of Investigation, and Social Security Administration.). The CCD is the personally identifiable data repository for American citizens services, passports, immigrant visas, and nonimmigrant visas. Because of the CCD’s importance to national security, ensuring its data integrity, availability, and confidentiality is vital.

CST does not have adequate oversight of CCD operations. [REDACTED]

Bold emphasis added.  Since the rest of that had been redacted, there is no way to tell just how inadequate is the oversight. If somebody downloads 500 gigabytes of personally identifiable information (PII) into a Lady Gaga CD, would anyone notice?

The OIG report provides a background of the Office of Consular Systems and Technology (CST) and why the inadequate oversight of this sensitive data is not good news:

CST succeeds in a very daunting mission. Every aspect of modern consular work depends on automated consular systems. CST systems in FY 2010 enabled CA to issue 13.8 million passports and 6.4 million visitor visas, respond to crises in Haiti and elsewhere, and protect the millions of Americans traveling abroad. The office maintains the CCD, a critical operational and national security database, which contains over 137 million American and foreign case records and over 130 million photographs and is growing at approximately 40,000 visa and passport cases every day. Serving 11,000 users in the Department and more than 19,000 users in other agencies, primarily the Department of Homeland Security (DHS) and various law enforcement elements, it is accessed more than 120 million times every month.

To carry out its mandate, CST must provide uninterrupted support to 233 overseas posts, 21 passport agencies, 2 passport processing centers, and other domestic facilities, for a total of 30,000 end users across 16 Federal agencies and in nearly every country. CST faces 24/7/365 service requirements, as any disruption in automated support brings operations to an immediate halt, with very serious implications for travelers and the U.S. image.

A snapshot of its staff:

CST is led by a director and is staffed by 68 full-time equivalent (FTE) employees (62 Civil Service and 6 Foreign Service). There are 12 positions (3 Foreign Service and 9 Civil Service) currently vacant. CA recently authorized CST 19 additional FTE positions. There are also more than 850 contractors operating under nearly 30 different contracts. In FY 2010, CST’s annual operating budget was approximately $266 million.

Other Key Findings:

  • The Office of Consular Systems and Technology (CST) succeeds in its mission of supporting consular operations around the world, enabling the Bureau of Consular Affairs (CA) in 2010 to issue 13.8 million passports and 6.4 million visitor visas, respond to crises in Haiti and elsewhere, and protect the millions of Americans traveling abroad. However, there are areas that require attention.
  • The director has a clear, well-articulated vision for his office as an exemplar of information management in the Federal Government. He deserves credit for his accomplishments and leadership role. His larger-than-life persona dominates the office. Although the director possesses strong technical skills and a fierce determination to achieve his goals, his managerial approach has created tension within CST and friction with other bureau and Department of State (Department) offices. Now that CA leadership has weighed in and the Department’s Civil Service ombudsman has begun a dialogue with CST, tension levels have eased; however, there is need for continued effort in and oversight of this matter.
  • The Consular Consolidated Database (CCD) is central to all consular operations and is an important element of U.S. national security systems. Although CST provides outstanding critical consular data delivery to the Department and other Federal agencies, [REDACTED]
  • Contract oversight is a major function of CST. The director takes great personal interest in this area, but contracting officer representatives (COR) vary in the effectiveness of their oversight. CST needs to ensure that all CORs are correctly trained and designated. CST also needs to carefully follow procedures for modifying existing contracts and task orders and to establish a procedure for periodic independent audits of CST contracts. To assist in this effort, CA has authorized 19 new positions for CST. The OIG team acknowledges the need for additional personnel but with the caveat that these increases be implemented within the context of a plan that reevaluates CST’s current organizational structure and is based on a strategic assessment of the office’s future needs.
  • Access controls for assigning and tracking user accounts in various critical systems in CST need to be strengthened.

Leadership and management issues and its impact on CST:

Possessing strong technical skills, a driving personality, and a fierce determination to achieve his goals, he deserves much of the credit for CST’s recent accomplishments. The Bureau Assistant Secretary and his supervisor, the principal deputy assistant secretary, support him and have signed on to his vision for CST and its IT strategy for supporting bureau stakeholders.

The director has instilled his sense of mission in many of his CST staff members, who generally, but not universally, respect his intellectual and technical skills. He takes pride in his commitment to broadening the professional skills of his technical staff. He dominates the office by his force of personality and rules with a blunt, no-nonsense, top-down managerial style that is punctuated by sporadic emotional public outbursts, often directed against individual staff members, as well as tirades occasioned by bureaucratic encounters with other CA and Department offices. At times, he has issued diktats to members of staff, prohibiting or restricting their contact with other entities in the Department (in one instance, the Visa Office came under his interdict). Whatever the efficacy of these prohibitions, they have constrained CST’s relationships within CA and have created an atmosphere in which some of CA’s other offices and some bureaus try to avoid personal encounters with the director. Unsurprisingly, communication and coordination with CST are often described as difficult and frustrating experiences. On the other hand, relations with other agencies, such as DHS, the Federal Bureau of Investigation, and the Social Security Administration, appear excellent. The director and the office are also well respected in the broader IT community.

Elsewhere in the report, the inspectors point out that the former deputy of CST, an FSO who apparently had left his position 18 months earlier had to be brought back from overseas to coordinate the preparation for the OIG inspection:

CST’s handling of the OIG inspection is illustrative of the problems that it faces in dealing with cross-cutting management issues. Before the inspection, CST brought back the former deputy director, now minister counselor at Embassy New Delhi, for three weeks to coordinate preparations for the inspection. During the inspection itself, the inspectors had to chase down documents and items relating to basic management issues. The OIG team explained to the director that the procedures related to the OIG compliance process would require the creation of an ad hoc staffing/procedural structure to manage the process, as the present organizational structure is inadequate for the task.

With the office facing a major expansion in personnel over the next 2 to 3 years, CST badly needs a structured management support unit in place now to plan for and handle these issues effectively and to provide the necessary oversight for and supervision of the contractors.

Morale, Emotional Outbursts, and Here the Ombudsman Steps In

Morale in CST can best be described as mixed. Some of the CST staff view the director’s management of the office as stimulating and challenging; others resent it. Even those who admire the director’s leadership and accomplishments most, view his emotional outbursts as embarrassing and inappropriate. Issues relating to the director’s management style, particularly his penchant for public emotional outbursts, came to a head in September and October 2010, when the Assistant Secretary and principal deputy assistant secretary reacted to reports of yet another angry eruption directed against individual employees at a CST meeting.

As a result, the Office of the Ombudsman established an ongoing dialogue with CST. A senior CA official also provided firm written guidance to the director on the need to curb his emotions and work on anger management. In the ensuing 4 months and through the OIG inspection, staff members confirmed that, although not perfect, the atmosphere had improved, with a perceptible easing of emotional tension within the office. Staff members expressed relief and hope that the improvement in the atmosphere would persist beyond the presence of the OIG inspection team.

The Leadership Tenet Not at Work — Communicate

“We help others understand the mission and their role in accomplishing it. We also listen actively to each others concerns and encourage innovation in consular work by listening intently for suggestions.”
-Consular Leadership Tenets

CST stands at the nexus of everything that CA does, and every part of the bureau praises CST’s technical expertise and accomplishments. At the same time, however, the interaction between CST and the business process stakeholders (the other directorates and offices within CA), is spotty, with frustration on the part of CA business units concerning a lack of a clear process for communicating and for establishing priorities for the maintenance, development, and support of automated systems. There is fault on both sides. The business process owners themselves need to take greater responsibility for establishing these priorities. At the same time, CST has sometimes made decisions for technical or budgetary reasons without coordinating with the operators in the field or the business process owners. One such decision seriously affected billing operations at the National Visa Center and required significant effort to redress.

Now, if you have ever used any of the consular systems, and pulled your hair in frustration at times at how nutty and unintuitive some of the functions within these systems are or why added keystrokes are there when they’re not needed, here is your official answer.  The second answer has to do with the number of FSOs working at CST — 6 out of 68! 

The OIG report includes 35 official recommendations, with recommendations #19-#25 redacted. It also contains 6 informal recommendations.

Iraq Transition: One of the Biggest Risks? Contractor-Led and Run Medical Supply Chain

In March, during the telconference with prospective bidders for the medical support services in Iraq, prospective bidders and DOD/State representatives had a Q&A. The transcript of that telcon was posted at FedBiz as part of the solicitation package.  The excerpts I selected below includes what the Army considers one of the biggest risks during the transition (medical supply chain), right of refusal by the contractor (DOD-left equipment), credentialing and vetting of personnel, and the challenges of the blood supply chain from the transhipment site in Qatar to various sites in Iraq.

As to how many DOD medical providers/medical support staff is currently in theater, apparently, according to the govt reps in this telcon, “There is not a way to answer that.”

INDUSTRY:  […] In the Army’s mind, what are the biggest risks that the contractor will face during transition?
GOVERNMENT:  Lieutenant Colonel […], CENTCOM.  I think one of the biggest risks are the transition to a contractor-led and run medical supply chain.  With that, I think they need to do a comprehensive study or analysis, if you will, of the transportation piece, and we will have to — and issues, and things like that that are involved with that.

INDUSTRY:  Question number six.  How many medical providers — physicians’ assistance, nurse practitioners, medical doctors, medics, and corpsmen — does the Department of Defense currently have at each site?
GOVERNMENT:  Yes.  Each of those sites, at this moment in time, serves a different purpose, and will not be analogous to our population at risk and the size of our missions. 
There is not a way to answer that.  There are some sites that have a troop medical clinic with one mid-level provider, and there are, of course, hospitals that, of course, have full staff.  So there is not really an answer that is suitable for this question.  It just can’t be compared to what we will have standing up.

INDUSTRY:  […] Question 11, does the awarded contractor have the right of refusal on all items left behind by the Department of Defense?  For example, medical equipment.

GOVERNMENT:  […] I think it would be the contractor’s call, whether to accept the equipment or not.  And if not, then the contractor would have to provide replacement equipment that we would have to agree to.  So
GOVERNMENT:  And this is [snip] in Baghdad.  I would agree.  There should be a compelling reason that the contractor would decline the use of equipment offered under this arrangement, and incur further expense to the U.S. Government.

INDUSTRY:  Question 16, will the transition period allow for proper credentialing, security vetting, redeployment training, and administrative processing?

GOVERNMENT:  Yes, I can.  The short answer is we are really uncertain at this point in time of the time frame that is going to be required for credentialing, security vetting, predeployment training, and the administrative processing.
The contractor, however, will not be penalized for delays due to the Department of State processes.

INDUSTRY:  Okay.  Question number 12.  What kind of support will DoD provide during the transition and full operating status of the following:  A, equipment, i.e. size, weight, and transportation issues; B, blood, critical resupply time, governing body for access, space to acquire blood supplies, transportation to Qatar; C, pharmaceuticals and vaccines, ministry of health regulations at each country; D, medical gases, hazmat issues, anesthesia and transportation?

GOVERNMENT:  This is [snip].  For A, USF-I will — we will go ahead and set up each of the locations’ equipment — medical supplies, minus pharmaceuticals.  And basically, we will set up each location ready for them to come in and go to work.  And that is portion A.

INDUSTRY:  Okay.  Major [….], on the blood?
GOVERNMENT:  All right, Major […] on blood.  I know blood will be supplied through the Armed Services Blood Program System, and this will be done via coordination through the blood program officer forward, and the blood transshipment center at Qatar.

Inventory levels will be communicated via an Excel spreadsheet, and this is going to be submitted daily to the blood transshipment center, just with general inventory numbers for the blood products, so that the BTC can generate the orders required to fill the inventory levels.

There should be a centralized person from the civilian locations that will consolidate the Department of State blood inventory information from the multiple facilities, and then this will be submitted in the report.  This individual will work directly with the blood program officer forward, and the transshipment center office or staff, for ordering the inventory to include the critical resupply need.
The blood transshipment center keeps the red cell products and the frozen products on hand for re-order and any re-supply that is needed.
As far as critical resupply times, this is going to vary.  From experience, this is usually dependent on the transportation available.  And for those, it’s my understanding that there will not be military transportation, that this will have to be a civilian-provided transport.
From our experience, you know, we can usually get blood products out to theater within a day.  If there is transport delays or storms or anything, it could get up to 48 to 72 hours.
Let’s see.  Once the actual transportation method has been chosen and coordinated, I recommend working directly with the transshipment center officer and staff for any final coordination details. 
Pick-up, if it should occur directly from the BTC location, that civilian transportation will need to gain access to the base.  I have communicated with our transshipment center officer, and she has the initial contact information for the POC at Qatar who would arrange any base access, be it via flight or ground transport to get there.  It’s just recommended that documentation for access to the base has to be started as soon as possible, due to the time of processing.  Over.

Active links added above. Almost all the moving parts above will now be taken over by different contractors – medical support, air transport, site access, security, etc. And that’s what worries me, frankly.

Read the MSSI transcript, edited 3-15-11.docx (35.38 Kb)

Also below is a diagram that represents how blood (red solid lines), pre-positioned frozen blood (red dotted lines), and reports (blue dotted lines) flow within the Armed Services system and demonstrates areas where efforts are coordinated (gray dashed lines).

Does the State Department or its contractors even have half a comparable system to this by end of 2011?

US Embassy Dublin: POTUS Visit May 23.2011 (Photos)

President Obama and First Lady Michelle Obama Visited Ireland on May 23, 2011

Created with Admarket’s flickrSLiDR.
Photos from US Embassy Dublin/Flickr