About the Customer

U.S. Immigration and Customs Enforcement (ICE) enforces federal laws governing border control, customs, trade, and immigration to promote homeland security and public safety. Created in 2003 through a merger of the former U.S. Customs Service and the Immigration and Naturalization Service, ICE has more than 20,000 employees in more than 400 offices in the United States and 46 foreign countries. The agency has an annual budget of approximately $6 billion.

ICE consists primarily of three operational directorates: Homeland Security Investigations (HSI), Enforcement and Removal Operations (ERO) and Office of the Principal Legal Advisor (OPLA). Management and Administration (M&A), a fourth directorate supports the three operational branches to advance the agency’s mission.

The Challenge

ICE’s Application Hosting Services (AHS) engineers routinely required tens of hours to manually deploy a single web application to a single target environment. Development teams often required multiple development environments to test concurrent code branches simultaneously. All applications were promoted through a minimum of three successive environments before reaching production. This complex and dynamic system relied solely on manual transition management (change, configuration, and release management) processes.

These manual processes were inefficient, error prone, and not easily and consistently repeatable. The lack of repeatability and consistency became increasingly obvious as defects compounded through the manual transition management processes. The resulting in poor quality control and inexplicable delays were a constant source of frustration for everyone involved.

From a governance perspective, the entire transition management process was an honor system. Since ICE could not build their code themselves, there was no way to verify that the executables delivered by developers were built from the source code the developers committed to source control. The infrequency of production releases left the source control repository with insufficient granularity for ICE to monitor or audit individual developer performance or development team progress.

Benefits

Development Consultants Incorporated was subcontracted by multiple prime vendors in support of the Engineering Support Service (ESS) contract for Immigration and Customs Enforcement (ICE). Our team was responsible for Information Technology professional services necessary to improve the efficiency and governance of custom software application development and deployments. The Application Hosting Services (AHS) team centrally managed all intranet and internet (E-government) applications and services applications for the entire ICE enterprise. Typical ICE applications demanded sophisticated requirements involving multiple government agencies, technology platforms, and network topologies.

Solutions

Re-Engineering

DCI developed a strategic business process re-engineering plan to mitigate governance weaknesses in the transition management processes. A prerequisite to our original objective of improving the deployment process was the establishment of a centrally managed enterprise source control as the sole authoritative system of record for source code throughout the ICE enterprise.

Intuitive Automation

All code changes originated in the Agile project management tool. Code changes triggered automated builds resulting in immutable tags in source control. Only tags were eligible for promoting code from environment to environment. Each successive promotion created a new immutable tag, so that the code could be unambiguously tracked from inception to production. Continuous integration automatically deployed code to the development environment(s). All automated builds produced a permanent and immutable online record sufficient for auditing every change that contributed to the build as well as a comprehensive log of the automated deployment process.

Automated deployments eliminated all manual AHS installation and configuration processes for applications under continuous integration. For the first time in ICE history, they had a complete record of every release, including each individual line of code that was changed, who changed it, when it was changed, and where and when it was tested.

Results

DCI developed ICE’s first continuous integration, continuous deployment, and Agile issue tracking systems in compliance with all NIST, FISMA, ICE, and DHS standards and policies. Requisite compliance documentation was developed, including a Disaster Recovery Plan, Contingency Plan and tests, and a System Security Plan. The system was comprised primarily of open-source products supplemented with custom code for Single Sign On (SSO) with ICE’s Active Directory (AD).

The system that DCI pioneered was certified by ICE and became the authoritative enterprise communication governance framework for ICE’s software change, release, and configuration management processes.