Background:

Working in collaboration with NESDIS, ACIO-S, OCIO/SDD, and AWS; we successfully researched, developed, tested, and assisted in the evaluation of an emerging technology Cloud pilot.

Solution:

GAMA-1 effectively exceeded the pilot objectives including securely ingesting external and inspect satellite data, while providing both data integrity and quality checks from six external data providers to include foreign national (EUMESAT, UCAR, Indian Space Research Organization (ISRO)) and commercial weather data (GeoOptics, Spire). As a result, NESDIS’ Secure Ingest Gateway Project (SIGP) is a highly scalable, portable, stable, and secure cloud-native network environment with ~4000 CPU cores, unlimited object storage, and 10Gbps direction connection from AWS to the NOAA N-WAVE to AWS. SIGP’s “Leaner, Meaner, and Faster” engine consists of native-cloud services and an HPC framework that allows for parallel processing, significantly improving processing performance and scalability.

Realizing the intended Business Technology IT investment value, to date, the SIGP has ingested over 40 data sources from satellites and ground radar sources from NASA, University of Wisconsin-Madison (UW-M), UCAR, EUMETSAT, NOAA Big Data Projects, ISRO, and Commercial data providers: Spire and, GeoOptics, etc. As well as, delivering data in real-time to the NESDIS STAR Program Office for scientific evaluation purposes. Achieving cost avoidance of approximately $1.2M, because no on-premise infrastructure capital expenses were required to build SIGP in the cloud. 

SIGP/OSIS met the NESDIS enterprise service implementation requirements to ingest external data, perform data quality checking, data integrity auditing, and make data available to internal users for 15 days via internal distribution systems. While the pilot leverages an AWS cloud-native solution, the HPC framework provides for mobility that can also be applied to an on-premise solution and/or other CSP as it allows for the architecture, application processes, and technologies to ensure external data is securely ingested, processed, and made available for distribution to authorized users.

Outcomes:

As a fully cloud (Amazon AWS) hosted enclave, OSIS is the portal to source foreign satellite and commercial weather data for downstream consumers such as National Weather Service, DoD agencies for operational weather, environmental prediction, research, it is highly available through redundant infrastructure design, has a velocity with on-demand cloud computing, and veracity with data inspection pipelines, employs encryption in-transit and at-rest to ingest variety and volume of data with the auto-scaling capability to achieve optimal cost model (value).

Upon successful transition to operations (planned for 3/25/2020), OSIS will serve as the primary data ingestion service for NESDIS, and provide a foundational framework for following cloud implementations such as product generation, product distribution. Additionally, OSIS sets an architecture reference model for security integration with NOAA enterprise security services. Upon readiness of NOAA OCIO cloud security instrumentation, OSIS will integrate with NOAA OCIO Cloud Access Security Broker (CASB).

The SIGP provides NESDIS with the tools to support a secure, consolidated, and standard approach to data ingest across the enterprise that is cohesive, scalable, secure, and cost-effective for ingestion of external data to NOAA internal customers. Based on the success of the SIGP, the NESDIS Operational Secure Ingest Service (OSIS) operationalizes and integrates into the NESDIS Cloud architectural framework and leveraged throughout the enterprise. OSIS awarded to GAMA-1 Aug 2019. 

On March 17, 2020, NESDIS Deputy Administrator, Mark Paese, along with NOAA CIO, Zach Goldstein, approved Operational Secure Ingest Service (OSIS), to transition to operations. A significant milestone for NESDIS and NOAA to launch a large scale cloud-based operation to support its ongoing mission of weather forecasting and climate research.

Article of Interest for this Success Story