-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathaccelerated-evaluation.html
339 lines (297 loc) · 27 KB
/
accelerated-evaluation.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
<!DOCTYPE HTML>
<!--
Editorial by Ding Zhao
Free for personal and commercial use under the CCA 3.0 license (html5up.net/license)
-->
<html>
<head>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-85426991-3"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag() {
dataLayer.push(arguments);
}
gtag('js', new Date());
gtag('config', 'UA-85426991-3');
</script>
<title>Ding Zhao's Website</title>
<meta name="author" content="Ding Zhao">
<meta name="keywords" content="ding,zhao,ding zhao,zhaoding,cmu,carnegie,mellon,umich,michigan,umtri,transportation,automated,automated vehicle,av,intelligent,accelerated evaluation,connected,safety,DSRC">
<meta name="description" content="Ding Zhao umich Accelerated Evaluation">
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" />
<!--[if lte IE 8]><script src="assets/js/ie/html5shiv.js"></script><![endif]-->
<link rel="stylesheet" href="assets/css/main.css" />
<!--[if lte IE 9]><link rel="stylesheet" href="assets/css/ie9.css" /><![endif]-->
<!--[if lte IE 8]><link rel="stylesheet" href="assets/css/ie8.css" /><![endif]-->
<link rel="icon" href="assets/icon/cmu.ico" />
</head>
<body>
<!-- Wrapper -->
<div id="wrapper">
<!-- Main -->
<div id="main">
<div class="inner">
<!-- Header -->
<header id="header">
<a href="index.html" class="logo"><strong>Highlighed
Research</strong></a>
<ul class="icons">
<li><a href="https://www.linkedin.com/in/ding-zhao-01130730/" class="icon fa-linkedin"><span class="label">LinkedIn</span></a></li>
<li><a href="https://scholar.google.com/citations?user=uirPPuYAAAAJ&hl=en" class="icon fa-graduation-cap"><span class="label">Scholar</span></a></li>
<li><a href="https://github.com/zhao-ding" class="icon fa-github"><span class="label">Github</span></a></li>
<li><a href="https://www.strava.com/athletes/7560016" class="icon fa-heartbeat"><span class="label">Run</span></a></li>
</ul>
</header>
<!-- Content -->
<section>
<header class="main">
<h1>Rigorous Evaluation of Intelligent Robots </h1>
</header>
<span class="image main"><img src="research/AccEval/AccEval.png" alt="" /></span>
<p>How to explore limitation and evaluate potential risks of an intelligent physical system has been a growing issue in the fields of autonomous vehicles, drones, and smart home robotics. A new approach, called Accelerated Evaluation, was proposed to
help solve this problem using synthesized approaches rooted in advanced statistics, modeling, optimization, control, and big data analysis. The basic idea is to model the operating environment and uncertainty using stochastic models. A learning-based
sampling scheme then can be derived based on prior knowledge and performance in preliminary tests. By adaptively adjusting the sampling strategy, the cost for evaluation can be significantly reduced. I proposed this concept while working towards
my Ph.D. I am still actively improving the methodology and extending its usage to broader conditions. <a href="research/AccEval/accel-evaluation.pdf" target="_blank"><i class="fad fa-presentation"></i></a></p>
<hr class="major" />
<h2 id="cutin">Accelerated Evaluation based on Importance Sampling</h2>
<p><span class="image right"><img src="research/AccEval/LaneChangeEvent.png" alt="" /></span>
<b><u>Ding Zhao</u>, Henry Lam, Huei Peng, David J. LeBlanc, Shan Bao, Kazutoshi Nobukawa, Christopher S. Pan, '' Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques ,''
<i>IEEE Transactions on Intelligent Transportation Systems</i>, vol. 18, no. 3, pp. 595-607, 2017.</b> |
<a href="https://arxiv.org/abs/1605.04965" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=uirPPuYAAAAJ&citation_for_view=uirPPuYAAAAJ:8k81kl-MbHgC"><i class="icon fa-graduation-cap"></i></a>
<br />
<b><u>Ding Zhao</u>, Huei Peng, Henry Lam, Shan Bao, Kazutoshi Nobukawa, David J. LeBlanc, Christopher S. Pan, ''Accelerated evaluation of automated vehicles in lane change scenarios, 2015,'' in <i>Proceedings of the ASME 2015 Dynamic Systems and Control Conference (<b>DSCC</b>)</i>,
Columbus, U.S.A., Oct 28-30, 2015.</b> |
<a href="http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/v001t17a002-dscc2015-9718.pdf" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=uirPPuYAAAAJ&citation_for_view=uirPPuYAAAAJ:UeHWp8X0CEIC"><i class="icon fa-graduation-cap"></i></a>
<br /><br />
<i>Abstract</i> — Automated vehicles (AVs) must be evaluated thoroughly before their release and deployment. A widely-used evaluation approach is the Naturalistic-Field Operational Test (N-FOT), which tests prototype vehicles directly on the
public roads. Due to the low exposure to safety-critical scenarios, N-FOTs are time-consuming and expensive to conduct. In this paper, we propose an accelerated evaluation approach for AVs. The results can be used to generate motions of the primary
other vehicles to accelerate the verification of AVs in simulations and controlled experiments. Frontal collision due to unsafe cut-ins is the target crash type of this paper. Human-controlled vehicles making unsafe lane changes are modeled as the
primary disturbance to AVs based on data collected by the University of Michigan Safety Pilot Model Deployment Program. The cut-in scenarios are generated based on skewed statistics of collected human driver behaviors, which generate risky testing
scenarios while preserving the statistical information so that the safety benefits of AVs in non-accelerated cases can be accurately estimated. The Cross Entropy method is used to recursively search for the optimal skewing parameters. The frequencies
of occurrence of conflicts, crashes and injuries are estimated for a modeled automated vehicle, and the achieved accelerated rate is around 2,000 to 20,000. In other words, in the accelerated simulations, driving for 1,000 miles will expose the
AV with challenging scenarios that will take about 2 to 20 million miles of real-world driving to encounter. This technique thus has the potential to reduce greatly the development and validation time for AVs.</p>
<br />
<!-- <div class="row">
<div class="embed-responsive embed-responsive-16by9">
<iframe class="embed-responsive-item" width="560" height="315" src="https://www.youtube.com/embed/RKxRgnWFN7Y" frameborder="0" allowfullscreen></iframe>
</div>
<div class="embed-responsive embed-responsive-16by9">
<iframe class="embed-responsive-item" width="560" height="315" src="https://www.youtube.com/embed/mN2dApQtCj8" frameborder="0" allowfullscreen></iframe>
</div>
</div>
-->
<hr class="major" />
<h2 id="carfollow">Accelerated Evaluation in a Dynamic Environment</h2>
<p><span class="image right"><img src="research/AccEval/CarfollowingAcc.png" alt="" /></span>
<b><u>Ding Zhao</u>, Xianan Huang, Huei Peng, Henry Lam, David J. LeBlanc, ''Accelerated Evaluation of Automated Vehicles in Car-Following Maneuvers,''
<i>IEEE Transactions on Intelligent Transportation Systems</i>, 2017.</b> |
<a href="https://arxiv.org/abs/1607.02687" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=uirPPuYAAAAJ&citation_for_view=uirPPuYAAAAJ:LkGwnXOMwfcC"><i class="icon fa-graduation-cap"></i></a>
<br />
<b><u>Ding Zhao</u>, Huei Peng, Shan Bao, Kazutoshi Nobukawa, David J. LeBlanc, and Christopher S. Pan, ''Accelerated evaluation of automated vehicles using extracted naturalistic driving data,'' in <i>24th International Symposium on Dynamics of Vehicles on Roads and Tracks (<b>IAVSD</b>)</i>,
Graz, Austria, August 17-21, 2015.</b> |
<a href="https://books.google.com/books?hl=en&lr=&id=8cb1CwAAQBAJ&oi=fnd&pg=PA287&ots=AwhEN0Ua59&sig=edUO86p8H9oU3B-2qh6GJLbtMoE#v=onepage&q&f=false" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=uirPPuYAAAAJ&citation_for_view=uirPPuYAAAAJ:qjMakFHDy7sC"><i class="icon fa-graduation-cap"></i></a>
<br /><br />
<i>Abstract</i> — The safety of Automated Vehicles (AVs) must be assured before their release and deployment. The current approach to evaluation relies primarily on (i) testing AVs on public roads or (ii) track testing with scenarios defined
in a test matrix. These two methods have completely opposing drawbacks: the former, while offering realistic scenarios, takes too much time to execute; the latter, though it can be completed in a short amount of time, has no clear correlation to
safety benefits in the real world. To avoid the aforementioned problems, we propose Accelerated Evaluation, focusing on the car-following scenario. The stochastic human-controlled vehicle (HV) motions are modeled based on 1.3 million miles of naturalistic
driving data collected by the University of Michigan Safety Pilot Model Deployment Program. The statistics of the HV behaviors are then modified to generate more intense interactions between HVs and AVs to accelerate the evaluation procedure. The
Importance Sampling theory was used to ensure that the safety benefits of AVs are accurately assessed under accelerated tests. Crash, injury and conflict rates for a simulated AV are simulated to demonstrate the proposed approach. Results show that
test duration is reduced by a factor of 300 to 100,000 compared with the non-accelerated (naturalistic) evaluation. In other words, the proposed techniques have great potential for accelerating the AV evaluation process.</p>
<hr class="major" />
<h2>Accelerated Evaluation using Piecewise Mixture to Emphasize Tail statistics</h2>
<p><span class="image right"><img src="research/AccEval/PiecewiseMixtureDist.png" alt="" /></span>
<b>Zhiyuan Huang, Henry Lam, David J. LeBlanc, <u>Ding Zhao</u>, ''Accelerated Evaluation of Automated Vehicles using Piecewise Mixture Models,''
<i>IEEE Transactions on Intelligent Transportation Systems</i>, 2017.</b> |
<a href="https://arxiv.org/abs/1701.08915" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=uirPPuYAAAAJ&citation_for_view=uirPPuYAAAAJ:kNdYIx-mwKoC"><i class="icon fa-graduation-cap"></i></a>
<br />
<b> Zhiyuan Huang, <u>Ding Zhao</u>, Henry Lam, David J. LeBlanc, Huei Peng, ''Evaluation of Automated Vehicles in the Frontal Cut-in Scenario - an Enhanced Approach using Piecewise Mixture Model,''
<i>IEEE International Conference on Robotics and Automation (<b>ICRA</b>)</i>, Singapore, May 29-June 3,2017.</b> |
<a href="https://arxiv.org/abs/1610.09450" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/scholar?hl=en&q=Evaluation+of+Automated+Vehicles+in+the+Frontal+Cut-in+Scenario-an+Enhanced+Approach+using+Piecewise+Mixture+Models&btnG=&as_sdt=1%2C23&as_sdtp="><i class="icon fa-graduation-cap"></i></a>
<br /><br />
<i>Abstract</i> — The process to certify highly Automated Vehicles has not yet been defined by any country in the world. Currently, companies test Automated Vehicles on public roads, which is time-consuming and inefficient. We proposed the
Accelerated Evaluation concept, which uses a modified statistics of the surrounding vehicles and the Importance Sampling theory to reduce the evaluation time by several orders of magnitude, while ensuring the evaluation results are statistically
accurate. In this paper, we further improve the accelerated evaluation concept by using Piecewise Mixture Distribution models, instead of Single Parametric Distribution models. We developed and applied this idea to forward collision control system
reacting to vehicles making cut-in lane changes. The behavior of the cut-in vehicles was modeled based on more than 403,581 lane changes collected by the University of Michigan Safety Pilot Model Deployment Program. Simulation results confirm that
the accuracy and efficiency of the Piecewise Mixture Distribution method outperformed single parametric distribution methods in accuracy and efficiency, and accelerated the evaluation process by almost four orders of magnitude.</p>
<hr class="major" />
<h2>Generalized Accelerated Evaluation Approach with Mixture Models and Kernel Methods</h2>
<p><span class="image right"><img src="research/AccEval/KernelMethod.png" alt="" /></span>
<b>Zhiyuan Huang, Henry Lam, <u>Ding Zhao</u>, ''An Accelerated Testing Approach for Automated Vehicles with Background Traffic Described by Joint Distributions, '' <i>Proceedings of the IEEE 20th International Intelligent Transportation Systems Conference (<b>ITSC</b>)</i>,
Yokohama, Japan, October 16-19, 2017.</b> |
<a href="https://arxiv.org/abs/1707.04896" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/citations?view_op=view_citation&continue=/scholar%3Fhl%3Den%26as_sdt%3D0,23%26scilib%3D1&citilm=1&citation_for_view=uirPPuYAAAAJ:HIFyuExEbWQC&hl=en&oi=p"><i class="icon fa-graduation-cap"></i></a>
<br />
<b> Zhiyuan Huang, Yaohui Guo, Mansur Arief, Henry Lam, <u>Ding Zhao</u>, ''A Versatile Approach for the Evaluation and Testing of Automated Vehicles based on Kernel Methods,'' <i>American Control Conference (<b>ACC</b>)</i>, Milwaukee, June 27-29, 2018.</b> |
<a href="https://arxiv.org/abs/1710.00283" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/scholar?hl=en&as_sdt=0%2C23&q=A+Versatile+Approach+to+Evaluating+and+Testing+Automated+Vehicles+based+on+Kernel+Methods&btnG="><i class="icon fa-graduation-cap"></i></a>
<!-- <a href="https://github.com/cav-localization/network-propagation"><i class="fa fa-github"></i></a> -->
<br /> <br />
<i>Abstract</i> — This line of research proposes a new framework based on Gaussian Mixture Model for evaluating risks of Automated Vehicles in a naturalistic driving environment. We extended the previous studies on the Accelerated Evaluation for Automated Vehicles. The proposed procedure includes Gaussian Mixture Model fitting and Importance Sampling distribution construction for Gaussian Mixture Model. We define and exploit the monotonic property of the rare event set in constructing Importance Sampling distribution. A tractable kernel formulation is then incorporated to amplify the approximation power of the models. We present simulation results to confirm that the procedure is valid and statistically efficient.</p>
<hr class="major" />
<h2 id="learning-based">Accelerated Evaluation with Statistical Learning Models</h2>
<p><span class="image right"><img src="research/AccEval/LearningBased.png" alt="" /></span>
<b>Zhiyuan Huang, Henry Lam, <u>Ding Zhao</u>, ''Rare-event Simulation without Structural Information: A
Learning-based Approach,'' <i>Proceedings of the 2018 Winter Simulation Conference</i>, Gothenberg, Sweden, December 9-12, 2018.</b> |
<a href="http://www.columbia.edu/~khl2114/files/con357s3-file1.pdf" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/scholar?q=RARE-EVENT+SIMULATION+WITHOUT+STRUCTURAL+INFORMATION%3A+A+LEARNING-BASED+APPROACH" target="_blank"><i class="icon fa-graduation-cap"></i></a>
<br />
<b>Zhiyuan Huang, Henry Lam, <u>Ding Zhao</u>, ''Designing Importance Samplers to Simulate Machine Learning Predictors via Optimization,'' <i>Proceedings of the 2018 Winter Simulation Conference</i>, Gothenberg, Sweden, December 9-12, 2018.</b> |
<a href="http://www.columbia.edu/~khl2114/files/con356s3-file1.pdf" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/scholar?q=DESIGNING+IMPORTANCE+SAMPLERS+TO+SIMULATE+MACHINE+LEARNING+PREDICTORS+VIA+OPTIMIZATION" target="_blank"><i class="icon fa-graduation-cap"></i></a>
<br /><br />
<i>Abstract</i> — Importance sampling has been extensively studied as a variance reduction tool in rare-event simulation. The design and efficiency of this method often relies on structural knowledge about the target problem in hand. In this line of research, we consider a simple rare-event setting, driven by Gaussian input variates, where there is no direct information about the system of interest, or where the system of interest can only be evaluated through a “black-box” model. This setting mimics, on a basic level, situations where the system of interest could be too complex to analyze as in the conventional importance sampling literature. We investigate an approach based on two-stage sampling, where the first stage learns the rare-event set of interest, and the second stage uses a change of measure that exploits the first-stage finding. We present some guarantees and discuss comparisons of this scheme with more naive approaches. In addition, We study the problem of designing good importance sampling (IS) schemes to simulate the probability that a sophisticated predictor, built for instance from an off-the-shelf machine learning toolbox, gives a prediction that exceeds a large threshold. This problem is motivated as a step towards building good learning algorithms that takes into account the extremal risks of the prediction. We provide a framework to design IS for two common machine learning models, namely random forest and a basic neural network. Our approach utilizes some available mathematical programming formulations to optimize over these models and a simple “cutting plane” idea to look for dominating points under Gaussian input distributions.</p>
<hr class="major" />
<h2>Synthesis of Multiple Autonomous Vehicles Tests</h2>
<p><span class="image right"><img src="research/AccEval/MFM.png" alt="" /></span>
<b> Zhiyuan Huang, Henry Lam, <u>Ding Zhao</u>,
''Towards Affordable On-track Testing for Autonomous Vehicle - A Kriging-based Statistical Approach, ''
<i>Proceedings of the IEEE 20th International Intelligent Transportation Systems Conference (<b>ITSC</b>)</i>, Yokohama, Japan, October 16-19, 2017.</b> |
<a href="https://arxiv.org/abs/1707.04897" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/citations?view_op=view_citation&hl=en&user=uirPPuYAAAAJ&citation_for_view=uirPPuYAAAAJ:4vMrXwiscB8C"><i class="icon fa-graduation-cap"></i></a>
<br /><br />
<b>Zhiyuan Huang, Mansur Arief, Henry Lam, <u>Ding Zhao</u>, ''Synthesis of Different Autonomous Vehicles Test Approaches,'' <i>Proceedings of the IEEE 20th International Intelligent Transportation Systems Conference (<b>ITSC</b>)</i>, Hawaii, USA, November 4-7, 2018.</b> |
<a href="https://arxiv.org/abs/1809.02911" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="https://scholar.google.com/scholar?q=Synthesis+of+Different+Autonomous+Vehicles+Test+Approaches" target="_blank"><i class="icon fa-graduation-cap"></i></a>
<br /><br />
<i>Abstract</i> — The first paper discusses the use of Kriging model in Automated Vehicle evaluation. We explore how a Kriging model can help reduce the number of experiments or simulations in the Accelerated Evaluation procedure. We also propose
an adaptive sampling scheme for selecting samples to construct the Kriging model. Application examples in the lane change scenario are presented to illustrate the proposed methods.
<br />
The second paper extends the discussion to address the problem of how to best integrate the information from different autonomous vehicle tests to further improve testing efficiencies. The underlying proposal is the use of multi-fidelity models that can take account the issue of varying 'informativeness' from different test approaches, historical data, and so on. It is believed that by strategically synthesize the information from different test approaches, both efficiency and informativeness of autonomous vehicle testing can be improved. </p>
<hr class="major" />
<h2>Accelerated Evaluation Framework for Highly Automated Vehicles</h2>
<p><span class="image right"><img src="research/AccEval/Dissertation.png" alt="" /></span>
<b>Zhao, Ding. "Accelerated Evaluation of Automated Vehicles." <i>PhD dissertation, The University of Michigan</i>, 2016.</b> |
<a href="https://deepblue.lib.umich.edu/handle/2027.42/120657" target="_blank"><i class="fa fa-link" aria-hidden="true"></i></a>
<a href="publication/Zhao_2016_Accelerated Evaluation of Automated Vehicles.pdf" target="_blank"><i class="icon fa-file-pdf-o"></i></a>
<a href="publication/Zhao_2016_Accelerated Evaluation of Automated Vehicles.bib"><i class="icon fa-graduation-cap"></i></a>
<br /><br />
<i>Abstract</i> — Automated Vehicles (AVs), which monitor the driving environment and conduct some or all of the driving tasks, must be evaluated thoroughly before their release and deployment. The challenges of AV evaluation stem from two
facts. i) Crashes are exceedingly rare events. In the U.S., one needs to drive on average 530 thousand miles to experience a police-reported crash and nearly 100 million miles for a fatal crash. The low exposure to safety-critical scenarios makes
the Naturalistic-Field Operational Tests (NFOT) very time-consuming and expensive to conduct, in which prototype AVs are driven by volunteers or test engineers on public roads. ii) AVs can “cheat” to pass predefined tests. Traditionally, vehicle
test protocols and test conditions are pre-defined and fixed. This is not a problem when the vehicle is “dumb”, but becomes a problem when the vehicle is intelligent and can be customized to excel in the predefined tests, and performance in other
test conditions receives less attention. An evaluation approach that represents the real world but not as time-consuming as the N-FOT is needed to address the problems mentioned above.<br /> In this research, we propose an “Accelerated Evaluation”
concept to accelerate the evaluations of AV by several orders of magnitude. The interactions between the AV and the surrounding Human-controlled Vehicles (HVs) are modeled based on the naturalistic driving data collected by the University of Michigan
Transportation Research Institute in the Safety Pilot Model Deployment Program and the Integrated Vehicle-Based Safety Systems Program. Probabilities of conflict, crash, and severe injury are used as the main metrics to assess the safety of AV designs.
In general, Accelerated Evaluation consists of six steps. <a href="https://deepblue.lib.umich.edu/handle/2027.42/120657" target="_blank">... </a></p>
</section>
</div>
</div>
<!-- Sidebar -->
<div id="sidebar">
<div class="inner">
<!-- Search -->
<section id="search" class="alt">
<form method="post" action="#">
<input type="text" name="query" id="query" placeholder="Search" />
</form>
</section>
<!-- Menu -->
<nav id="menu">
<header class="major">
<h2>Menu</h2>
</header>
<ul>
<li><a href="index.html">Homepage</a></li>
<li><a href="research.html">Research</a></li>
<li><a href="publications.html">Publication</a></li>
<!--<li>-->
<!--<span class="opener">Submenu</span>-->
<!--<ul>-->
<!--<li><a href="#">Lorem Dolor</a></li>-->
<!--<li><a href="#">Ipsum Adipiscing</a></li>-->
<!--<li><a href="#">Tempus Magna</a></li>-->
<!--<li><a href="#">Feugiat Veroeros</a></li>-->
<!--</ul>-->
<!--</li>-->
<!--<li><a href="#">Teaching</a></li>-->
<li><a href="talk.html">Talk</a></li>
<li><a href="student.html">Student</a></li>
<li><a href="calendar.html">Calendar</a></li>
<!--<li>-->
<!--<span class="opener">Another Submenu</span>-->
<!--<ul>-->
<!--<li><a href="research.html">Lorem Dolor</a></li>-->
<!--<li><a href="#">Ipsum Adipiscing</a></li>-->
<!--<li><a href="#">Tempus Magna</a></li>-->
<!--<li><a href="#">Feugiat Veroeros</a></li>-->
<!--</ul>-->
<!--</li>-->
<!--<li><a href="#">Maximus Erat</a></li>-->
<!--<li><a href="#">Sapien Mauris</a></li>-->
<!--<li><a href="#">Amet Lacinia</a></li>-->
</ul>
</nav>
<!-- Section -->
<!--<section>-->
<!--<header class="major">-->
<!--<h2>Ante interdum</h2>-->
<!--</header>-->
<!--<div class="mini-posts">-->
<!--<article>-->
<!--<a href="#" class="image"><img src="images/pic07.jpg" alt="" /></a>-->
<!--<p>Aenean ornare velit lacus, ac varius enim lorem ullamcorper dolore aliquam.</p>-->
<!--</article>-->
<!--<article>-->
<!--<a href="#" class="image"><img src="images/pic08.jpg" alt="" /></a>-->
<!--<p>Aenean ornare velit lacus, ac varius enim lorem ullamcorper dolore aliquam.</p>-->
<!--</article>-->
<!--<article>-->
<!--<a href="#" class="image"><img src="images/pic09.jpg" alt="" /></a>-->
<!--<p>Aenean ornare velit lacus, ac varius enim lorem ullamcorper dolore aliquam.</p>-->
<!--</article>-->
<!--</div>-->
<!--<ul class="actions">-->
<!--<li><a href="#" class="button">More</a></li>-->
<!--</ul>-->
<!--</section>-->
<!-- Section -->
<section>
<header class="major">
<h2>Get in touch</h2>
</header>
<p>The best way to reach me, generally, is via email. During the day, I am usually in my office.</p>
<ul class="contact">
<li class="fa-envelope-o">[email protected]</li>
<li class="fa-phone">412-268-3348</li>
<li class="fa-map-marker">5000 Forbes Avenue<br /> Scaife Hall 315<br /> Pittsburgh, PA 15213</li>
</ul>
</section>
<!-- Footer -->
<footer id="footer">
<p class="copyright">© Ding Zhao. All rights reserved. Design: <a href="http://html5up.net">HTML5 UP</a>.</p>
</footer>
</div>
</div>
</div>
<!-- Scripts -->
<script src="assets/js/jquery.min.js"></script>
<script src="assets/js/skel.min.js"></script>
<script src="assets/js/util.js"></script>
<!--[if lte IE 8]><script src="assets/js/ie/respond.min.js"></script><![endif]-->
<script src="assets/js/main.js"></script>
<script>
(function(i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function() {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date();
a = s.createElement(o),
m = s.getElementsByTagName(o)[0];
a.async = 1;
a.src = g;
m.parentNode.insertBefore(a, m)
})(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');
ga('create', 'UA-85426991-2', 'auto');
ga('send', 'pageview');
</script>
</body>
</html>