-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.Rmd
96 lines (65 loc) · 10.9 KB
/
index.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
---
title: "DEQ Water Quality Automated Assessment User Guide"
author: "DEQ Automated Assessment Team"
date: "Last Updated: `r Sys.Date()`"
site: bookdown::bookdown_site
# output: bookdown::gitbook
documentclass: book
bibliography: [book.bib, packages.bib]
biblio-style: apalike
link-citations: yes
colorlinks: yes
github-repo: openscapes/series
description: "A companion document describing the WQA automated assessment methodology."
split_by: "section"
---
# Introduction {#Introduction}
This manual serves as a companion document to DEQ's Water Quality Assessment (WQA) automated assessment methodology for completion of the biennial [305b/303d Integrated Report](https://www.deq.virginia.gov/water/water-quality/assessments/integrated-report). Virginia is among a handful of state agencies that have organized concerted efforts to systematically automate and expedite the water quality assessment process. These entities have all taken different approaches to best meet their state-specific analysis and reporting needs. Through the [Tools for Automated Data Assessment (TADA) working group](https://github.com/USEPA/TADA), EPA is organizing an effort to share code and create national tools to assist partners with assessment analyses. Considering the technological advances in water monitoring that have increased the quantity and types of data requiring consideration during an assessment period, as well as the static or decreasing staff time allocated to the assessment process, automation appears the only solution to keep up with Clean Water Act requirements and keep the public informed of the status of waterbody health and restoration timelines for those waterbodies not meeting water quality standards. By embracing automated assessment procedures, entities aim to provide constituents with higher quality, standardized, and transparent water quality assessment results while adhering to federally mandated deadlines.
Virginia has taken a hybrid approach to the process of automating the water quality assessments. DEQ relies upon an automated process to manipulate hundreds of thousands of water quality data collected throughout the state during a given assessment period (six year window). These scripts evaluate each record for exceedances of appropriate Water Quality Standards. Regional assessment staff may use interactive web-based analytical tools provide more context to tabular results of exceedance analyses to assist with Quality Assurance/Quality Control (QA/QC) prior to submitting data to EPA's [Assessment, Total Maximum Daily Load (TMDL) Tracking and Implementation System (ATTAINS)](https://www.epa.gov/waterdata/attains){target="_blank"} system through DEQ's internal [Comprehensive Environmental Data System (CEDS)](https://ceds.deq.virginia.gov/ui#){target="_blank"}. This approach maximizes the benefits of each partner in the assessment process, allowing computers to excel at systematic and expedited data manipulation and analysis at scale and allowing humans to digest multiple lines of evidence to identify any potential data collection or analysis errors or areas of environmental concern.
## How to Use Document {#HowToUseDocument}
The following report is an agency effort to facilitate the further adoption of automated assessment methods statewide. The intention of this project is to provide a non-programmer audience with accessible and understandable narratives describing the automated water quality assessment process. This document is not a comprehensive [WQA Guidance Manual](https://www.deq.virginia.gov/water/water-quality/assessments/wqa-guidance-manual){target="_blank"}, nor is it an introduction to using the R programming language. Here, we document the overall automated assessment process, explaining reasons certain decisions were made, how to unpack analyses, and where to seek further assistance through additional resources or interactive data exploration through web-based application interfaces.
### Programming Expectations {#ProgrammingExpectations}
The Water Quality Assessment process is complicated. Learning a programming language is complicated. Combine these topics, the subject of this book, and we arrive at complicated^2^.
The narrative script descriptions are written for understanding at an elementary programming level; however, the individual scripts utilize advanced programming techniques for parsimony, efficiency, and ease of long term maintenance. While it is not necessary to understand every element of code provided to run the automated assessment methods, certain programming themes listed below can be helpful:
* [Tidyverse](https://www.tidyverse.org/){target="_blank"}
* [Rmarkdown](https://rmarkdown.rstudio.com/){target="_blank"}
* [Custom functions](https://datacarpentry.org/semester-biology/lessons/R-functions/){target="_blank"}
* [Optimizing list objects with purrr](https://jennybc.github.io/purrr-tutorial/){target="_blank"}
* [Pinned data](https://cran.r-project.org/web/packages/pins/vignettes/pins.html){target="_blank"}
The internally-published [DEQ R Methods Encyclopedia](https://rconnect.deq.virginia.gov/MethodsEncyclopedia/){target="_blank"} is a good resource when getting started with R. These articles are useful to familiarize yourself:
* [Getting Started With R](https://rconnect.deq.virginia.gov/MethodsEncyclopedia/gettingStartedWithR.html#gettingStartedWithR){target="_blank"}
* [How to Learn (or Continue Learning) R?](https://rconnect.deq.virginia.gov/MethodsEncyclopedia/gettingStartedWithR.html#how-to-learn-or-continue-learning-r){target="_blank"}
* [Connecting to R Connect (for Pinned Data)](https://rconnect.deq.virginia.gov/MethodsEncyclopedia/connectToConnectPins.html#connectToConnectPins){target="_blank"}
The [WQA Technical Support Team](#wqa-technical-support-team) is available for assistance should you encounter any questions.
## Project History{#ProjectHistory}
The agency began efforts toward automating components of the IR with a dissolved metals assessment written in SAS. These tabular results were provided to regional assessment staff along with raw data queries encompassing data stored in CEDS for each IR data window.
The [Freshwater Probabilistic Monitoring](https://www.deq.virginia.gov/water/water-quality/monitoring/probabilistic-monitoring){target="_blank"} program began automating analyses and a final report for inclusion in the 2016 IR using the open source [R programming language](https://cran.r-project.org/){target="_blank"}. These practices have been further refined each IR cycle, improving report quality and graphics as well as significantly reducing the amount of time required to generate a report after data collection. These methods were identified as a possible solution to the ongoing challenges in the WQA program to complete vast amounts of data analysis on increasingly short timelines with limited staff.
An effort to systematically analyze all water quality data stored in CEDS for assessments began in 2017. As a pilot project, lakes and reservoirs in the Blue Ridge Regional Office (BRRO) assessment region were selected for first waterbody type to undergo automation and receive an interactive web-based tool to assist regional assessment staff for the 2018 IR. These initial automated assessment efforts were completed using the open source [R programming language](https://cran.r-project.org/){target="_blank"} and interactive applications were built using the [Shiny](https://shiny.rstudio.com/){target="_blank"} package. Rivers and streams followed suit for the 2020 IR, joining lacustrine waterbodies with automated assessment methods and interactive assessment tools. The 2020 IR automated methods were scaled from just the BRRO region to statewide applicability. A pilot project to incorporate citizen monitoring data requiring assessment was undertaken in the BRRO assessment region for the 2020 IR. This pilot proved effective in standardizing and increasing efficiency of incorporating this disparate data and was officially adopted as a process for future IR windows.
After thorough QA from regional assessment staff across the state, the riverine and lacustrine automated assessment tools were rebooted for the 2022 IR with increased functionality and the ability to assess more parameters, including those not stored in CEDS. However, due to delays organizing citizen monitoring data statewide, automated results for these data were not provided for the 2022 IR. A new database schema for archiving station-specific water quality standards and assessment unit information was implemented. This system benefited the assessment process as well as numerous DEQ water programs that previously did not have access to WQS information at that spatial scale. Appendices and fact sheets for the IR were generated using R and [Rmarkdown](https://rmarkdown.rstudio.com/){target="_blank"} for the first time during the 2022 IR.
The 2024 IR further refines improvements to the riverine and lacustrine automated assessment methods and interactive tools. By partnering with the [Chesapeake Monitoring Cooperative (CMC)](https://www.allianceforthebay.org/project/chesapeake-monitoring-cooperative/){target="_blank"}, DEQ leveraged an existing public-facing citizen monitoring data portal to expand utility outside the Chesapeake Bay watershed and incorporate all of Virginia's citizen monitoring data. These data are now automatically cleaned and stored by the CMC and can more easily be incorporated in the automated assessment process with web scraping techniques.
To date, no estuarine-specific assessment methods have been completed, but the WQA program is investigating utility and potential adoption among regions with estuarine waters.
## WQA Technical Support Team
Should you encounter any techical issues running the included scripts or have questions about the automated assessment methods described, please reach out to these technical staff for assistance:
* Emma Jones ([email protected])
* Joe Famularo ([email protected])
* Kristen Bretz ([email protected])
## Acknowledgements {#Acknowledgements}
Many Water Quality Assessment (WQA) and Water Quality Standards (WQA) staff have contributed to this effort:
* Emma Jones ([email protected])
* Joe Famularo ([email protected])
* Reid Downer ([email protected])
* Kristen Bretz ([email protected])
* Jason Hill ([email protected])
* Sandy Mueller ([email protected])
* Cleo Baker ([email protected])
* Amanda Shaver ([email protected])
* Tish Robertson ([email protected])
* Paula Main ([email protected])
* Mary Dail ([email protected])
* Martha Chapman ([email protected])
* Sara Jordan ([email protected])
* Kristie Britt ([email protected])
* Jennifer Palmore ([email protected])
* Kelley West ([email protected])
* Rebecca Shoemaker ([email protected])
Please direct any project questions to Emma Jones ([email protected]).