U.S. Customs and Border Protection Cargo Systems Program Directorate (CSPD) Agile Development, Operations, & Maintenance
The CBP CSPD Production application portfolio includes 105 applications which directly support the CBP mission of enforcing trade laws and securing the borders while facilitating legitimate trade. This means processing over 66,000 conveyances daily, collecting over $30B annually, and meeting the needs of diverse stakeholders including the CBP Offices of Field Operations and International Trade, the public trade community, Participating Government Agencies (PGAs), and foreign governments.
U.S. Antidumping and Countervailing Duties (AD/CVD) Portal
Modernization
The Department of Commerce (DoC) Office of Enforcement and Compliance collaborated with U.S. Customs and Border Protection (CBP) to initiate this project with the objective of streamlining the Antidumping and Countervailing Duties (AD/CVD) case messages and case reference data functions, resulting in a more user-friendly navigation for the trade community. This deployment of the enhanced AD/CVD module provides a more secure and transparent system and continues to contribute to the efforts by DoC and CBP to level the playing field for domestic industries injured by unfair trade practices.
Digital Services Application Development & Design Support for Department of Homeland Security (DHS)
The purpose of this effort is to utilize the U.S Digital Service Playbook along with emerging technologies to identify and solve core user needs through the development and release to end users of a Digital Minimum Viable Products (MVPs) in the form of enhancements to U.S. Customs and Border Protection (CBP) applications. To achieve this goal, our team establishes and executes a framework for user-centered design, usability research, and agile implementation.
U.S. Customs and Border Protection Cargo Systems Program Directorate (CSPD) Agile Development, Operations, & Maintenance
For Cargo Systems Program Division’s (CSPD) ACE program, we provide IT services to develop, migrate, and support cargo system applications, leveraging Agile development methodologies. The Cargo Production applications provide direct support to the CSPD mission of enforcing trade laws and securing the borders while facilitating legitimate trade. This means processing over 66,000 conveyances daily, collecting over $30B annually, and meeting the needs of diverse stakeholders including the CBP Offices of Field Operations and International Trade, the public trade community, Participating Government Agencies (PGAs), and foreign governments. The CSPD application portfolio includes 105 applications. Our 17 Agile teams leverage advanced DevSecOps techniques to average over 50 non-invasive deployments each month. This allows flexible mission responsiveness while ensuring the stability and quality of the critical production environment.
Operations and Maintenance, Performance Upgrades, and Enhancements
We manage our Agile teams and individual sprints to balance the capacity for O&M activities and new development initiatives as prioritized by Product Owners and program leadership. We monitor and respond to Remedy tickets submitted on the CSPD applications. Our DevOps team tunes our system alerts from AppDynamics, Oracle Enterprise Manager (OEM), and Splunk to make them meaningful, providing a high level of coverage across ACE applications. We enhance automated test coverage for all applications using a dedicated team that drives test automation across all our Agile teams through the development of new scripts and mentoring developers and testers. The result is a robust set of functional regression tests run daily, with results automatically emailed to our Agile teams.
One of the most important areas of development and maintenance are interfaces. We have established and maintain over 45interfaces between ACE applications and external systems, including ATS. With over 30 years of experience modernizing CSPD ACE and PSPD TECS systems, our team has unique subject matter expertise across a broad set of CBP functional processes. Our expertise is critical to effective integration and has been leveraged to establish the existing interfaces between ACE and TECS to TASPD to share the data used in downstream targeting.
Cloud Migration
We have migrated 47 applications to the CBP AWS Cloud East (CACE) environment and 16 applications to CSPD’s Docker container environment. An additional 55 migrations to these environments are planned. We accomplish these migrations at scale using Agile team and program support engagement, risk mitigation strategies, and proactive communication.
New Developers are assigned to our migration tiger team, where their first assignment is to support the migration of an application out of WebLogic, mentored by our Program Technical Lead. Once they have gained the needed experience working with the technologies and design patterns, they join an Agile team to apply this knowledge and accelerate migrations. We established a Pre-Production environment in CACE & Docker, certified to store Production data and scaled to handle representative Production volumes. We perform a two-part test in this environment before go-live. First, we complete performance testing of the migrated applications to verify performance under Production load. We then perform data comparison testing to confirm that a set of messages processed by both the existing Production application, and the migrated version of the application, return the same results.
Performance Innovation
To optimize the performance of our Agile teams at CSPD, we established a robust DevSecOps team that is focused on Production availability and monitoring, expanding the use of our CI/CD pipeline, and supporting release management. Organizing these resources in a separate team allows us to improve availability and efficiency and accommodate changing priorities. All Teams have fully migrated all ACE applications to automated CI/CD pipelines with code repositories in Stash and/or Jenkins.
A previous migration we completed was for the oldest set of ACE applications, built nearly 15 years ago, which still used Dimensions as their code repository and did not employ automation for deployments. Our team successfully migrated all the code from Dimensions to Stash, implemented automated build and deployments for all environments, and integrated the execution of automated unit tests into the build process.
Our DevOps team supports our release management process by creating Change Requests (CRs) for upcoming releases and managing the implementation of these CRs during our planned outage windows. This team has developed a custom utility to automate the creation of CRs by integrating JIRA and Remedy to automatically generate all necessary artifacts. This reduces CR creation time from 4 to 8 hours to minutes.
Project Management
Effective alignment to respond to evolving needs is the foundation of our ACE Project Management efforts. Our team has established a culture of collaboration, partnership, and investment with CBP leadership that is operationalized by several critical activities:
- We conduct daily stand ups with the Deputy Executive Director (DXD) and Director of Cargo Systems Development to review any outstanding delivery risks or issues and ensure effective real-time alignment of our team, CSPD, and stakeholders.
- We provide briefings to the CSPD Executive Director (XD) and his direct reports to provide an in-depth review of project status, upcoming delivery dates, improvement initiatives, and overall funding status.
- We conduct meetings between OIT & Office of Trade leadership to define and prioritize our program backlog, ensuring alignment on priorities.
- We ensure we have the right staff with the necessary skills at the right time. Through close alignment and planning with CSPD and program leaders, we anticipate potential staffing gaps in upcoming increments and staff in advance. Through our extensive experience at CBP we also understand what is necessary to ensure we provide clearable staff.
U.S. Antidumping and Countervailing Duties (AD/CVD) Portal Modernization
Department of Commerce (DoC) Office of Enforcement and Compliance collaborated with U.S. Customs and Border Protection (CBP) to initiate this project with the objective of streamlining the Antidumping and Countervailing Duties (AD/CVD) case messages and case reference data functions, resulting in a more user-friendly navigation for the trade community. This deployment of the enhanced AD/CVD module provides a more secure and transparent system and continues to contribute to the efforts by DoC and CBP to level the playing field for domestic industries injured by unfair trade practices.
The scope for this effort included the redesign and development of the user-interface, data structure, usability, and security of the existing ACE Antidumping/Countervailing Duty (AD/CVD) Portal. In addition to enhancing the AD/CVD portal, our team was responsible for user provisioning and training key stakeholders.
A challenge with this work was that the majority of our key stakeholders had not worked with Agile before. Therefore, it was critical that we started our efforts with an education on the benefits of the Agile process to ensure we received buy-in from the start. To most effectively train the group on key principles and benefits of Agile, we invited key stakeholders to a workshop where we customized the training to the DHS agile process and leveraged our team’s knowledge of AD/CVD to provide content-accurate examples. The feedback immediately following the training was excellent and this enabled the team to hit the ground running.
With the stakeholders aligned to the agile process, our team began to utilize the digital service playbook by understanding what the stakeholders needed and addressed the whole experience using UI/UX user-centric stories. Certain key features in the AD/CVD single window redesign include improved user-centric design with fewer clicks for ease of use, enhanced organization of key data points for transparency, automation of manual processes to increase collaboration between stakeholders, exportable data, advance search capability and quick search filters. A premier feature of this enhancement is the utilization of CBP Amazon Web Service Cloud Environment (CACE), which includes a variety of security benefits. A portion of this effort also included the conversion of data from IBM DB2 to PostgreSQL. We leveraged DevSecOps heavily to focus our continuous integration (CI) and continuous delivery (CD) to ensure active testing and verification of code through automation testing, code coverage, analytics, and blue/green deployments.
Through these efforts, our team was able to deliver a modernized AD/CVD application, that provided more accessibility, a stronger user experience, reliability, maintainability and significant cost savings to the government. To read more about this success, please see the following CBP Press Release: “U.S. Customs and Border Protection and Department of Commerce Enhance ACE”.
Digital Services Application Development & Design Support for Department of Homeland Security (DHS)
The purpose of this effort is to utilize the U.S Digital Service Playbook along with emerging technologies to identify and solve core user needs through the development and release to end users of a Digital Minimum Viable Products (MVPs) in the form of enhancements to U.S. Customs and Border Protection (CBP) applications. To achieve this goal, our team establishes and executes a framework for user-centered design, usability research, and agile implementation.
Key program goals for this project are to:
- Leverage the CBP Amazon Web Services Cloud Environment (CACE) as much as possible in the design and delivery of functionality
- Migrate to a common processing environment for functionality in current CSPD systems (ACE and ACS) as well supporting newly developed functionality using a common processing platform for all Cargo systems.
- Support a component and service-based architecture operating in a distributed environment, leading to more efficient maintenance, extension, updates and upgrades to Cargo systems.
- Use proven open-source technologies to the greatest extent possible to reduce acquisition and Operations and Maintenance (O&M) costs.
Our team collaborates with DHS to determine how problems will be solved through digitalization, utilizing user-centered design techniques and product management methods to create an initial product vision, product roadmap, and scope of the MVP.
We begin each portion of scope by exploring and pinpointing the needs of the people who will use the service, and the ways the service will fit into their lives. We continually test the products with end users to ensure delivery is focused on meeting their needs. Key to our success has been the upfront time spent collaborating with current and prospective users of the service. We leverage that feedback using a range of qualitative and quantitative research methods to determine people’s goals, needs, and behaviors.
Our team then develops mock-ups or prototypes of solutions to share with prospective users. The feedback then fuels the backlog of prioritized user stories developed during the agile development process. While the iterative development process is underway, there is a continuous cycle of testing and demonstrating the progress to stakeholders so that the feedback loop can continuously improve the output.
Agile ceremonies leveraged by our team include iteration retrospectives, release planning, backlog grooming, and other common activities associated with iterative design and agile methodologies. These agile best practices lead to the release of high-quality MVPs to the end users.