AIAA 2005-685

43rd AIAA Aerospace Sciences Meeting and Exhibit

10 - 13 January 2005, Reno, Nevada

Honey, I Shrunk the Grids!

A New Approach to CFD Verificiation Studies
Shelley Hebert , E. A. Luke
Computational Simulation and Design Center, Mississippi State, MS 39762, USA
There has been recent interest in the use of the method of manufactured solutions as a technique for verification that computational codes solve the specified partial differential equations. Such verification techniques
provide a mechanism to verify that a particular code is solves the specified equations to a specific order of accuracy. The technique requires solving for the manufactured solution on a sequence of progressively finer grids
and measuring the exact error (difference from the exact solution) to establish that this error decreases consistently with the expected order of accuracy. There are some shortcomings of the approach in that it can become
expensive to show that an asymptotic result has been found, particularly for 3D problems where each level of
refinement is 8 times larger. In addition, it is difficult, and in some cases impossible, to apply the technique
to generalized grids where self-similar refinements may not be available. We present an alternative method
to achieve this goal that samples the solution space at different random locations using different scalings of
the same grid (shrinking grids). A statistical sample is used to estimate the error at that level. This approach
has advantages in terms of both cost savings and the evaluation of generalized solvers and meshes that are not
possible with grid refinement methodologies.

I. Introduction
Software verification, in the context of numerical software for computational mechanics, consists of testing numerical solver software to confirm that it indeed solves the set of partial differential equations that it was intended to
solve. Many methods15 have been advanced for the verification of solvers such as those used in computational fluid
dynamics (CFD). Typically these approaches involve grid convergence studies where the software is tested to determine how error is reduced as the grid (mesh) is refined on problems with known analytic solutions. This approach can
be used to confirm that the numerical solver does indeed converge to the specified equations, but in addition that the
numerical solver converges with the specified order of accuracy.
The difficult part of grid convergence studies is knowing the exact error defined by the difference between the
approximate discrete solution and the actual continuum solution. This is not the same as estimating the error (verification of calculations2 ) or determining how well the solution compares with experimental results (validation), but
determining the actual exact error. Here, the Method of Manufactured Solutions (MMS) 2, 6 has become an important
tool because it allows one to know the exact difference between a numerical solution and its analytic counterpart.
In most real flow problems, there is no known analytic solution. If there were, a numerical solution wouldnt be
required. As a result, determining the error E = qreal qnumeric is difficult. The MMS allows one to specify the analytic
solution before a code is run. The solution is filtered through the governing equations which are usually of the form
Dq = 0 where D is the differential operator and q is the solution variable. When the solution is passed through the
equations, a leftover source term f is obtained. The manufactured solution q mms is the real analytic solution to Dq = f .
The source term is added to the code and when run, qmms should be approximately recovered. The difference is the
error.
Convergence of the numerical method is demonstrated by showing diminishing error as a function of grid spacing
(usually defined in terms of some h or x). The usual approach to controlling h is through grid refinement whereby the
density of points in the mesh is increased. For structured grids, there is a straightforward approach to grid refinement.
However, for unstructured grids the task can become more difficult. For example, the refined grid should maintain
similarity to the original grid, otherwise bias is introduced in the sampling for error. In addition, for three dimensional
problems, refined grids become large quickly. However, it is desirable that the code is verified using grids that are

Graduate Research Assistant, Engineering Research Center, Mississippi State University.

Professor, Department of Computer Science and Engineering, Mississippi State University, AIAA Member.

Assistant

1 of 12
American Institute of Aeronautics and Astronautics

Copyright 2005 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved.

representative of what will be applied in practice as this will provide an estimate of actual error observed in practice.
These problems can be made even more severe when considering generalized grids (grids consisting of arbitrary
polyhedral elements) such as illustrated in figure 1. For generalized grids, there may not be a suitable self-similar
approach to grid refinement. We desire an approach that will facilitate evaluating a particular grid and its suitability
for solving a given problem by a given solver. The traditional verification approach falls short of this goal.

Figure 1. Example of the symmetry plane of a typical adapted generalized grid

The motivation for this work is to demonstrate a new methodology for performing a grid convergence study using
shrinking grids instead of refined grids. That is, we will take the same grid but scale it to smaller and smaller dimensions instead of refining the grid. However, in order to make this approach feasible, we have to estimate error using
a statistical technique whereby we translate the grid to different regions of a periodic solution space to determine a
mean error at the given grid scaling. This approach is potentially more economical than traditional grid convergence
studies where grid doubling is used since, for 3-D problems, each refinement increases the number of grid points by
a factor of 8. Additionally, it may be difficult to determine if the grids are in the asymptotic range for 3-D problems
because of the rapid increase in the number of grid points. Further this method should be applicable to structured and
unstructured grids alike, even unstructured grids for which there is no self-similar method of refinement that preserve
grid quality.
In this work we apply the verification technique to the CHEM solver, 7, 8 a chemically reacting compressible flow
solver that is used for modeling combustion and hypersonic problems. The CHEM solver uses a second-order finitevolume algorithm for both steady-state and time accurate solutions to multi-component reacting flows on generalized
three-dimensional meshes.

II. Grid Convergence Studies and Theoretical Order of Accuracy

Most grid convergence studies involve several, successively finer grids where some measure of discretization h (or
x, y, z) is reduced by adding more and more points. If the code is found to be, say, second-order accurate, then E
will reduce as h2 ; if h is reduced by half, then E should be reduced by a factor of four.
If E1 and E2 are some measure of error on two grids, with x1 = rx2 , then the order of accuracy is
p =
=

log EE21

(1)

x
log rx

log EE21

logr

2 of 12
American Institute of Aeronautics and Astronautics

(2)

For say, grid doubling, r = 12 . In three dimensions, this means 8 the number of grid points in the finer grid. For
unstructured grids, r can be determined using the ratio of the number of grid points or cells in the two grids:
r
N2
(3)
r= d
N1

where N1 and N2 are the number of cells in the fine and coarse grids, respectively, and d is the number of dimensions.
So for a 2D refinement, r = 2 means 4 the number of grid points and 8 in 3D as mentioned above.
Now typically, the finer grids are produced by using more points in the same space. For a 1-D grid, the coarsest
grid may divide the interval [0, 1] into 4 intervals using 5 points at 0, 0.2, . . . , 1.0. The next finer grid may have 9
intervals with 10 points spaced at 0, 0.1,. . . , 1.0. Then further divide that into 19 intervals with 20 points at 0, 0.05,
0.10, 0.15,. . . , 1.0. In this study, instead of doubling the number of grid points to halve h, we simply halve h. Instead
of dividing the interval [0, 1] with five points, 10 points, 20 points,. . . we divide [0, 1] with 10 points, [0, .5] with 10
points, [0, 0.25] with 10 points,. . . . Using this grid scaling, we still get a sequence of x values of 0.2, 0.1, 0.05, . . . .
Note that this eliminates portions of the solution space with each shrinking. The second grid does not include error
in the interval [0.5, 1], the third does not include error in the interval [0.25, 1]. If the manufactured solution were, say,
ecx , then there would naturally be more error at larger values of x, simply because the function values are larger there
(for positive c).
Since this method eliminates portions of the solution space at each step, portions that may have high error (e.g.,
high gradient regions), we may see unrepresentative errors in the smaller grids. To correct this problem, we sample
solution space at N random points and compute an average error that is representative of error over the entire domain.
To make sure we sample the same space for all grid scales, we choose a periodic function for the manufactured
solution, and then we translate the grid to random points within the periodic interval. Due to the selection of the
periodic function, we dont have to concern ourselves with the location of the outer boundary of the grid since if the
grid extends past a periodic boundary we can consider it as if the grid wrapped around automatically to the same
sampling domain.
For example, if we wish to estimate what the `2 norm of error would be for a grid that spanned the entire periodic
space with a grid spacing similar to our sample grid, we could compute the average error as in
kerrork1 =

1
N

Ei

(4)

i=1

where the mean error would be computed from the results of random samples taken from within the periodic space.
This technique has several advantages. First is that in the 3-D case, refining the grid means doubling the number of
points in each of three directions, so each finer grid is eight times larger than the previous grid. The downside is that
while the grids are not any larger, each one is run N times at different locations in the solution space. This is easily
parallelizable, though, and may even be run on a workstation, while the larger, refined grids may require a cluster or
other large system to run.
Further, the technique is applicable to grids for which there is no self-similar way to refine them such as mixed
element unstructured or generalized grids. The finer grids are simply scaled copies of the coarser grids. Traditional
grid convergence studies on unstructured grids must use refinement metrics like the ratio of the number of points in
each grid since there may not be anything like x to halve. The problem is that the grids may no longer be similar
unless special care was taken to generate the finer grids.
Unstructured grids often have bad cells. In some cases, they are unavoidable. In a traditional grid convergence
study, these cells may be refined away or, more likely, in the finer grid they will encompass a smaller proportional
volume or proportion of the total cells. As a result of the increase in grid quality, the accuracy may go up more than the
true order would indicate. One could consider such effects under the umbrella of asymptotic range, however they
make the traditional grid refinement approach for unstructured grids more difficult, particularly in three dimensions.

III. Manufactured Solution

The equations solved were the viscous steady-state Navier-Stokes equations for NS species:
div(n u)
= 0 for n = 1, . . . , NS

div( uu + I p) = div
div(( e0 + p)u)
= div( u)
+ div(k gradT )
3 of 12
American Institute of Aeronautics and Astronautics

where

= (div u)
I + (grad u + (grad u)
T)
= 32 .

(5)

and
For the inviscid cases, = = k = 0. n is the density of species n and =
The solution used was for three species:


1 
x 
y
1 (x, y, z) = 0 1 +
2 + cos
2 + cos
(2 + cosz)
10
2
3



x
z
1 
2 + cos
(2 + cosy) 2 + cos
2 (x, y, z) = 0 1 +
10
3
2



1
z
y 
2 + cos
3 (x, y, z) = 0 1 + (2 + cos(x + 1)) 2 + cos
10
2
3
(x, y, z) = 1 (x, y, z) + 2 (x, y, z) + 3 (x, y, z)


1 
x 
y
u(x, y, z) = u0 1 +
2 + cos
2 + sin
(2 + sinz)
10
2
3


x 
y
1 
2 + sin
2 + cos
(2 + sinz)
v(x, y, z) = u0 1 +
10
2
3


1 
x 
y
w(x, y, z) = u0 1 +
2 + sin
2 + sin
(2 + cosz)
10
2
3


x 
y
1 
2 + sin
2 + sin
(2 + sinz)
T (x, y, z) = T0 1 +
10
2
3

NS
n=1 n .

where 0 , u0 , T0 are base values, we used 0 = 2, u0 = 150, and T0 = 300; the solution is simply some perturbation
about the base value. Temperature T was related to pressure p through the equation of state p = R s T and total energy
e0 consisted of two parts, internal energy ei = ns Rs T and kinetic energy:
1
1
1
e 0 = u 2 + v 2 + w2 + e i .
(6)
2
2
2
The source terms were added to the code as an arbitrary distributed source. Dirichlet Boundary conditions were
imposed, so the value of the manufactured solution was used at the boundary points. The initial conditions were set
to the solution values to speed convergence. Error was evaluated and quantified using the ` 1 norm. Both a traditional
grid convergence study and the shrinking grid study were performed. Additionally, the traditional grid convergence
study included the `2 norm.
Several different grids, both structured and unstructured were tested. For structured grid tests we used a simple
open cube with the same number of points on each side and a curvilinear donut grid made by revolving a rectangular
section around the y-axis. For the unstructured grids we consider an unstructured hexahedral element grid that was
created by refining, through centroid splitting, an isotropic tetrahedral mesh on a sphere. In refinements of this hexahedral grid used in the traditional study, the resulting hexahedra were each further split into 8 similar hexahedra using
centroid splitting. Finally, for the grid shrinking study, a mixed element unstructured grid is used that contains prisms,
pyramids, and tetrahedrons.

IV. Traditional Method

A. Cube Grid
Five grids from 5 nodes on a side to 80 nodes on a side were used, with side length of 0.125. The results are given in
table 1.
The order of accuracy is at least 2 in both cases and seems to be converging to 2. This agrees with the theoretical
order of the algorithms in the CHEM code.
B. Donut Grid
Here, four grids were used, with 10, 20, 40, and 80 points in each of the radial, , and z directions. The inner radius
of the donut was 0.25, the outer radius 0.5. The results are given in table 2. Again, the order of accuracy is at least 2
and appears to be converging to 2.
4 of 12
American Institute of Aeronautics and Astronautics

Table 1. Error for traditional study using cube grid

Grid
`1 -norm
5
10
20
40
80
`2 -norm
5
10
20
40
80

Order

Order

Order

1.4869e-04
2.6484e-05
5.3476e-06
1.1813e-06
2.7695e-07

2.49
2.31
2.18
2.09

1.8721e-01
3.2410e-02
6.0924e-03
1.2773e-03
2.8924e-04

2.53
2.41
2.25
2.14

3.1804e+02
6.3243e+01
1.3577e+01
3.1206e+00
7.4737e-01

2.33
2.22
2.12
2.06

1.695e-04
3.008e-05
6.312e-06
1.434e-06
3.411e-07

2.49
2.25
2.14
2.07

2.4174e-01
3.9019e-02
7.0700e-03
1.4284e-03
3.1385e-04

2.63
2.46
2.31
2.19

4.0529e+02
7.2825e+01
1.4734e+01
3.2698e+00
7.6752e-01

2.48
2.31
2.17
2.09

Table 2. Error for traditional study using donut grid

Grid
`1 -norm
10
20
40
80
`2 -norm
10
20
40
80

Order

Order

Order

5.0466e-04
9.5104e-05
2.2738e-05
5.8364e-06

2.41
2.06
1.96

7.2312e-01
1.3844e-01
3.2940e-02
7.9578e-03

2.38
2.07
2.05

8.9774e+02
1.8462e+02
3.8727e+01
8.8911e+00

2.28
2.25
2.12

6.8762e-04
1.3400e-04
3.2036e-05
8.1071e-06

2.36
2.06
1.98

8.9984e-01
1.9250e-01
4.5713e-02
1.1240e-02

2.22
2.07
2.02

1.1748e+03
2.2793e+02
4.8178e+01
1.1207e+01

2.37
2.24
2.10

C. Unstructured Sphere
The grid is a sphere composed of hexahedral elements, both on the boundary and in the interior. Refinement was done
using the refinement code from Thompson, et al.9 The ratio of the number of cells in each grid was used to calculate
the refinement. The number
qof cells in each grid is given in table 3. The actual refinement ratio used in calculating the

order of accuracy was r =

Ci+1
Ci

where Ci is the number of cells in grid i. This is consistent with the refinement ratios

used in other grid convergence studies6, 10 where grid doubling in two dimensions uses a refinement ratio of 2 even
though the finer grids have 4 times the number of cells (both in the structured and unstructured cases).
Table 3. Number of cells in unstructured sphere grid
Grid
1
2
3
4

Number of cells
1272
10176
81408
651264

Increase

Refinement

8
8
8

2
2
2

Using this measure of grid refinement, the order of accuracy is given in table 4. Of note is that the order of accuracy
is not as high as for the two regular structured grids. The order of accuracy may be converging to 1 instead of 2. We
shall see this behavior again with the scaled grid study and the unstructured grids.

V. Grid Scaling Study

The manufactured solutions above are periodic with smallest common period 6 . The region {(x, y, z) | x, y, z
[0, 6 ]} covers at least one period in any direction for each of the solution functions. A traditional grid convergence
study, to get the most complete estimation of error in that space, would need a grid that covers that region, e.g., a cube
5 of 12
American Institute of Aeronautics and Astronautics

Table 4. Order of accuracy for traditional study using unstructured sphere grid
Grid
`1 -norm
1
2
3
4
`2 -norm
1
2
3
4

Order

Order

Order

8.9772e-04
3.1438e-04
1.0759e-04
4.5054e-05

1.51
1.55
1.26

1.9164e+00
6.3034e-01
2.2235e-01
7.2158e-02

1.60
1.50
1.62

2.9322e+03
8.3730e+02
2.6404e+02
8.1905e+01

1.81
1.66
1.69

1.1932e-03
4.1334e-04
1.4140e-04
6.0662e-05

1.53
1.55
1.22

2.5567e+00
8.3703e-01
2.9198e-01
9.5836e-02

1.61
1.52
1.61

3.9699e+03
1.1193e+03
3.5219e+02
1.1017e+02

1.83
1.67
1.68

6 on a side or a sphere that would circumscribe that region. A grid that covered a 6 cube would need a lot of points
to keep x small.
Instead of covering the space completely to get measure of the error, the grid scaling method would sample the
space, uniformly and randomly, to estimate the mean error. For grids with side lengths of 2 i for i = 0, . . . , NG where
NG is the number of grid levels, the space is sampled with N samples. We used 20 samples per grid level for this
study.
To get the error on each grid, we compute the `1 -norm of the volume-weighted error:
E=

volcell

|q qnumeric| Voltotal

(7)

cells

This gives the error on each grid. To get the mean error for each sized grid, treat the samples as pieces of a single
disconnected grid:
1 N
E = Ei
(8)
N i=1
Note that the total volume in disconnected grid i is 8 the volume in disconnected grid i + 1 while using the
traditional approach, the total grid volume is the same for all grids. In the traditional case, the total volume is the same
but the individual cell volumes reduce by a factor of 8 with each refinement. However, there are 8 as many cells to
add up. We can show that the order of accuracy should be the same for this case as well.
Let E1 , E2 be the mean errors on consecutive grid levels computed as above. Then for i = 1, 2,
!

 voli,cell
1 N
Ei =
qexact qnumeric, j  Voli,total
N j=1
cells
Then since vol1,cell = 8vol2,cell and Vol1,total = 8Vol2,total ,




1 N
qexact qnumeric,1, j  vol1,cell

cells
N j=1
Vol1,total
E1

=

 vol2,cell 
1 N
E2


N j=1 cells qexact qnumeric,2, j Vol2,total


 8vol2,cell 
1 N


N j=1 cells qexact qnumeric,1, j 8Vol2,total


=


1 N
qexact qnumeric,2, j  vol2,cell

cells
j=1
N
Vol2,total


 vol2,cell 
1 N


N j=1 cells qexact qnumeric,1, j Vol2,total

=

 vol2,cell 
1 N


N j=1 cells qexact qnumeric,2, j Vol2,total

Even though the volume of the next smaller grid size is one eighth of the current grid, since the cell volumes are
normalized by the total grid volume the errors are still comparable. If E 1 is four times larger than E2 , there is still a
second-order reduction in error.
Four grids were used for the grid scaling case: a 5 5 5 cube, a 10 30 10 donut (30 points in the angular
direction), the coarsest hexahedral sphere, and an unstructured cube with a viscous wall.
6 of 12
American Institute of Aeronautics and Astronautics

First consider the simple cube grid. The results of the scaling study are given in table 5. The order of accuracy
is roughly 2 for each grid. Taking the average over all of the grids, the order of accuracy is very close to 2. Since
the orders of accuracy are not exactly 2 (or typically even within one decimal place), we used a statistical hypothesis
test to determine whether we could confidently say that the code was second order. At a confidence level of 95%, a
two-sided hypothesis test says that there is insufficient evidence to say that the code is not second order.
Table 5. Error and order of accuracy for shrinking cube grid
Grid scale (2i )
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Average
Std Dev

1
0.0032973
0.00094625
0.00025568
5.0772e-05
1.4553e-05
3.7699e-06
8.5082e-07
2.5312e-07
4.9377e-08
1.3457e-08
4.1450e-09
9.0498e-10
1.7086e-10
5.0058e-11
1.6707e-11
3.3992e-12
1.0754e-12
2.8978e-13
7.6114e-14
2.1401e-14

Order
1.80
1.89
2.33
1.80
1.95
2.15
1.75
2.36
1.88
1.70
2.20
2.41
1.77
1.58
2.30
1.66
1.89
1.93
1.83
1.96
0.256

u
2.5356
0.61893
0.15816
0.038067
0.011860
0.0030943
0.00057723
0.00015468
3.2955e-05
9.8604e-06
1.9955e-06
5.4768e-07
1.4664e-07
2.6859e-08
7.7727e-09
2.3071e-09
7.4009e-10
1.7928e-10
4.3916e-11
1.4356e-11

Order
2.03
1.97
2.05
1.68
1.94
2.42
1.90
2.23
1.74
2.30
1.87
1.90
2.45
1.79
1.75
1.64
2.05
2.03
1.61
1.97
0.248

e
5373.7
1274.8
325.00
88.552
21.626
7.5131
1.1850
0.28869
0.098963
0.020790
0.0052503
0.0011441
0.00030098
6.3677e-05
1.9738e-05
6.3754e-06
1.9918e-06
5.2557e-07
1.1218e-07
3.5692e-08

Order
2.08
1.97
1.88
2.03
1.53
2.66
2.04
1.54
2.25
1.99
2.20
1.93
2.24
1.69
1.63
1.68
1.92
2.23
1.65
1.95
0.292

Next, consider the donut grid. The code was second order for the first few levels but the order dropped off radically
at scales of 29 . This is shown in table 6. Using the first eight levels of refinement (the first seven orders), we get
that the average orders of the variables 1 , u, and e are 1.98, 1.91, and 1.94 respectively with standard deviations
of 0.160, 0.210, and 0.290. Statistically, the code is second order for those levels. Over all 20 levels, the order is, of
course, much less.
Next is the unstructured sphere grid. The error and orders of accuracy are given in table 7. Statistically, in both
cases the code is first order.
And finally, the viscous cube grid. Like the unstructured sphere grid, statistically the code is first order accurate.

VI.

Statistical Issues

Since the order of accuracy is only a statistical estimate, an obvious question is how many samples are required
to get sufficient accuracy in the estimate. 20 samples and 20 levels of scaling seem like they might be a lot. Thats
400 samples, so as an alternative, we used four levels of scaling with 100 samples per level. This should give a better
estimate of the mean error and hence less variation in the computed orders of accuracy. The results of this study are
given in table 9.
Since there is still a good bit of variation in the orders, it is reasonable to say that there is simply a lot of variation
in the order of accuracy. This is one reason why several grids are used rather than only two. However, the accuracy of
the mean error estimate is much better, as can be seen in table 10. This is only natural since were covering more and
more of the period space with more samples.
Note that since the standard deviation for the samples is on the same order as the sample mean, computing the
logarithm of the samples helps normalize (in the statistical sense) the sample values, giving a more intuitive estimation
of the mean. It is reasonable to consider the the logarithms because we use the logarithm to compute the order of
accuracy anyway. The only real difference is that the log of the mean of the samples is not quite the same as the mean
of the logarithms of the samples (that is the log of the geometric mean). Nevertheless, the order of accuracy is similar,
7 of 12
American Institute of Aeronautics and Astronautics

Table 6. Error and order of accuracy for shrinking donut grid, viscous three-species equations
Grid scale (2i )
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Average
Std Dev

1
0.00092906
0.00025021
7.0378e-05
1.4245e-05
3.3566e-06
8.1131e-07
2.1576e-07
5.8004e-08
1.6152e-08
7.4618e-09
5.0962e-09
5.6391e-09
4.2723e-09
4.0262e-09
3.0744e-09
2.0478e-09
1.4306e-09
6.9494e-10
3.3390e-10
1.8518e-10

Order
1.89
1.83
2.30
2.09
2.05
1.91
1.90
1.84
1.11
0.55
-0.15
0.40
0.09
0.39
0.59
0.52
1.04
1.06
0.85
1.17
0.774

u
0.90929
0.26231
0.061176
0.015953
0.0039279
0.00085652
0.00025436
6.7538e-05
2.3272e-05
1.1636e-05
6.4396e-06
6.1246e-06
5.6158e-06
3.9344e-06
2.9257e-06
2.1859e-06
1.5526e-06
7.7430e-07
3.3711e-07
2.0580e-07

Order
1.79
2.10
1.94
2.02
2.20
1.75
1.91
1.54
1.00
0.85
0.07
0.13
0.51
0.43
0.42
0.49
1.00
1.20
0.71
1.16
0.719

e
1500.7
470.68
121.79
26.416
6.2023
1.1904
0.42650
0.10810
0.030947
0.013468
0.0056586
0.0058544
0.0055648
0.0045796
0.0039098
0.0032234
0.0026313
0.0013550
0.00060357
0.00035714

Order
1.67
1.95
2.20
2.09
2.38
1.48
1.98
1.80
1.20
1.25
-0.05
0.07
0.28
0.23
0.28
0.29
0.96
1.17
0.76
1.16
0.801

Table 7. Error and order of accuracy for shrinking unstructured sphere grid, viscous three-species equations
Grid scale (2i )
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Average
Std Dev

1
0.002948
0.001030
0.0005055
0.0002350
0.0001203
5.3891e-05
2.7614e-05
1.4507e-05
8.3958e-06
3.8270e-06
2.3344e-06
1.1472e-06
6.4658e-07
4.4873e-07
3.2288e-07
1.7235e-07
1.0872e-07
5.8567e-08
3.0156e-08
1.5864e-08

Order
1.52
1.03
1.10
0.97
1.16
0.96
0.93
0.79
1.13
0.71
1.02
0.83
0.53
0.47
0.91
0.66
0.89
0.96
0.93
0.92
0.237

u
4.0349
1.6262
0.78574
0.36752
0.14574
0.093578
0.039780
0.020615
0.010083
0.0044509
0.0026608
0.0011453
0.00060160
0.00031177
0.00020144
0.00013590
8.7916e-05
4.4408e-05
2.5868e-05
1.4954e-05

Order
1.31
1.05
1.10
1.33
0.64
1.23
0.95
1.03
1.18
0.74
1.22
0.93
0.95
0.63
0.57
0.63
0.99
0.78
0.79
0.95
0.243

8 of 12
American Institute of Aeronautics and Astronautics

e
5648.0
2564.4
1113.4
507.64
231.98
126.42
54.057
27.283
14.362
6.5967
3.7736
1.6762
0.94912
0.55588
0.41270
0.27806
0.19774
0.11780
0.054850
0.035559

Order
1.14
1.20
1.13
1.13
0.88
1.23
0.99
0.93
1.12
0.81
1.17
0.82
0.77
0.43
0.57
0.49
0.75
1.10
0.63
0.91
0.253

Table 8. Error and order of accuracy for shrinking viscous cube grid, viscous three-species equations
Grid scale (2i )
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Average
Std Dev

1
0.0038265
0.0013263
0.00067458
0.00029598
0.00011960
7.4921e-05
3.4758e-05
1.8243e-05
9.0945e-06
4.5284e-06
2.6348e-06
1.2166e-06
6.2881e-07
3.8697e-07
2.4438e-07
1.2443e-07
7.1490e-08
3.8098e-08
1.8700e-08
9.5546e-09

Order
1.53
0.98
1.19
1.31
0.67
1.11
0.93
1.00
1.01
0.78
1.11
0.95
0.70
0.66
0.97
0.80
0.91
1.03
0.97
0.98
0.217

u
4.0337
2.0319
0.94773
0.45545
0.19946
0.13542
0.053742
0.034337
0.012651
0.0060302
0.0034046
0.0013753
0.00065299
0.00029058
0.00016605
9.0962e-05
5.5471e-05
2.8196e-05
1.4472e-05
8.2439e-06

Order
0.99
1.10
1.06
1.19
0.56
1.33
0.65
1.44
1.07
0.82
1.31
1.07
1.17
0.81
0.87
0.71
0.98
0.96
0.81
0.99
0.237

Order

e
5244.8
2594.8
1021.8
523.56
226.53
156.93
73.528
41.789
15.864
7.1118
4.0607
1.7155
0.88888
0.47668
0.32205
0.18880
0.12464
0.073804
0.031496
0.020367

1.02
1.34
0.96
1.21
0.53
1.09
0.82
1.40
1.16
0.81
1.24
0.95
0.90
0.57
0.77
0.60
0.76
1.23
0.63
0.95
0.269

Table 9. Error, order of accuracy, and statistics for many-sample study

Grid scale (2i )
3
4
5
6
Average

1
5.6882e-05
1.3972e-05
3.7492e-06
9.3746e-07

Order
2.03
1.90
2.00
1.97

u
0.047193
0.0099279
0.0024241
0.00062889

Order
2.25
2.03
1.95
2.08

9 of 12
American Institute of Aeronautics and Astronautics

e
98.465
21.022
5.4875
1.3146

Order
2.23
1.94
2.06
2.06

approximately 1.9-2.0. The means in table 10 are different than those in tables 9 and 5; table 10 gives the mean of the
2-norms, the other tables have the means as computed using the formula given in section II. This only goes to show
that different legitimate ways of computing the mean error give the same results.
Table 10. Statistics for large and small samples sizes at grid scale 23
Number of Samples

1 , grid scale 23
20
100
1 , Grid Scale 24
20
100

Mean

Std Dev

Std Error ( N )

Log Mean

Log Std Dev

Log Std Error

0.00005077
0.00005688

0.00004366
0.00003846

9.763e-6
3.846e-6

-10.14
-9.974

0.6869
0.6376

0.15360
0.06376

0.00001455
0.00001397

1.232e-5
8.8953e-6

2.755e-6
8.8953e-7

-11.45
-11.36

0.8141
0.6186

0.18204
0.06186

Note, however, that for the simple cube grid, the 80 80 80 grid is 4000 times larger than the 5 5 5 grid, if
it were run on the same system as the smaller grid. In our case, reasonable turnaround time dictated that we run the
larger cases (40 40 40 and larger) on a cluster. The time required to process the 400 samples used in this study is
still less than the time required for the 80 80 80 grid on a parallel system.
Another question that bears investigating is whether 20 levels of scaling are really justified. Could skipping levels give comparable results and similar confidence in those results? Say, instead of scaling the grid by a factor of
1
20 , . . . , 219 , use scales of 20 , 24 , 28 ,. . . and using a refinement ratio of 16
instead of 21 . The properties of the logarithm say that this should work, and indeed it does, as shown in table 11. The error data comes from table 5. The order
of accuracy is still, of course, 2, but the standard deviation has gone down by a factor of 2-4. The downside to this
approach is that it only has a quarter of the sample size, which is statistically less desirable. Nevertheless, the smaller
sample size is somwhat offset by the smaller standard deviation. Additionally, this is more consistent with traditional
studies which use a half-dozen or fewer grids.
Table 11. Errors and orders of accuracy found by skipping levels
Grid scale (2i )
0
4
8
12
16
Average
Std Dev

1
0.0032973
1.4553e-05
4.9377e-08
1.7086e-10
1.0754e-12

Order
1.96
2.05
2.04
1.83
1.97
0.101

u
2.5356
0.011860
3.2955e-05
1.4664e-07
7.4009e-10

Order
1.93
2.12
1.95
1.91
1.98
0.0964

e
5373.7
21.626
0.098963
0.00030098
1.9918e-06

Order
1.99
1.94
2.09
1.81
1.96
0.116

VII. Pros and Cons

This approach has several advantages and disadvantages compared to traditional approaches.
Con: Over-counting the boundary. If the samples are considered to be parts of a single, disconnected grid,
then the number of boundary points will be a much higher proportion of the total number of points compared
to the traditional approach. In the traditional approach, the number of boundary points for, say, the cube grid,
increases quadratically while the total number of points in the grid increases cubically. In this approach, the
number of boundary points increases linearly with the number of samples, just like the total number of grid
points. Since the boundary values are prescribed, we would expect there to be significantly less error at points
near the boundary and so the total error could be artificially lowered. Since this affects each level equally,
however, it may not affect the outcome.
Con: For the purpose of collecting statistics, it is assumed that error can be treated as a random independent
variable. However, this may not be in reality true (for example as error may be transported with the solution to
influence error in other parts of the mesh). However, based on empirical evidence, it does not appear that this
effect is significant.
Pro: This method seems to be somewhat conservative in that the order demonstrated by the traditional studies is
10 of 12
American Institute of Aeronautics and Astronautics

higher than the order demonstrated by the scaled grid studies. The traditional study, in the case of the unstructured sphere grid, was inconclusive. It certainly did not appear to be second order like the other two grids, and
in the density case, looked like it would converge to first order. Nevertheless, with that data, it does not appear
to be second order. The scaled grid study confirms that; the code is more likely to be first order than second in
this case.
Pro: We believe that this method does the right thing for unstructured grids. Since there is no mesh length
like x for unstructured grids, refinement of unstructured grids is taken from the ratio of the number of cells in
the grid. But how good a measure of refinement this is that? Even if the grid could be refined self-similarly
(which is not always the case), one could imagine the case where half of the grid is refined, say, twice, while
the other half is not refined at all. Simply using the ratio of the number of cells does not give a good indication
of how much finer the second grid is than the first. Here, there is a clear refinement ratio since there is only one
grid used in the study and it is simply scaled in three dimensions by the refinement ratio.
Pro: This method is easily parallelizable, even if the code to be verified is not. Each level of scaling could
be run simultaneously, even each sample at each level. Of course, theres no reason why the grids used in a
traditional convergence study could not be run simultaneously, but there is significantly more opportunity for
parallelization using this method.
Pro: The grid used does not have to be that big. The grids used here were quite small compared to the largest
of the grids used in the traditional grid convergence study. The 5 5 5 cube grid runs in a matter of seconds
on a workstation (indeed, about half of the time required was startup time), so even with 400 samples, a single
workstation is more than adequate to do the scaling grid study. For the 80 80 80 cube grid, a cluster was
required to get results at all, much less in a reasonable amount of time. In the time required on the cluster for
the largest grid, thousands of samples of the smallest grid could be run.
Pro: Real grids can be used. It may not be feasible to refine real production grids due to their size (coarsening
is an option, to an extent), and as a result, many traditional grid convergence studies use contrived grids or real
grids with very simple geometries (e.g., nozzle or a compression ramp) rather than a grid for an engineering
application.
We found two items of interest during this study. First, our code had limit-cycle problems for grids composed of
only tetrahedral elements. A cube and sphere composed of tetrahedral elements were also studied and in certain cases,
limit-cycles would prevent the code from converging sufficiently to a solution to perform the grid convergence study.
As a result, a new method for computing gradients was implemented which seems to have solved the problem. While
the order of of accuracy did not change as a result, the code is now more robust.
Second, we found that our code does not have the same order of accuracy for structured and unstructured grids,
even though it is a solver for generalized grids. We created a set of 10 10 10 cube grids where the interior points
were randomly offset by some times x
4 . As the magnitude of the offset increased, the order dropped from second to
first. For a 5% maximum offset ( [0, 0.05]), the order of accuracy was approximately 1.5 and for a 10% maximum
offset the order of accuracy was approximately 1.3. Therefore, we suspect that unrealistically ideal mesh quality may
be required to achieve second order accuracy on unstructured meshes for the algorithms used in the CHEM solver.

VIII. Conclusion
We have developed a new methodology that can determine the consistency and order of accuracy of a solver.
The technique combines the method of manufactured solutions with a statistical approach that estimates how error is
reduced as mesh spacing is reduced. Since this technique uses the same mesh in all samples, it simplifies significantly
the process of refining unstructured grids. In some cases, it makes it possible to evaluate a solver on grids for which
no appropriate refinement method exists. In addition, we find that the methodology is more economical than the
traditional approach to mesh refinement, particularly for three-dimensional problem domains due to the excessive cost
of grid refinement. In addition, we suggest that the methodology can be used to validate grid generation techniques
as well, as the methodology can evaluate the impact of various grid configurations on solution accuracy without the
requirement of a refinement or coarsening strategy.

11 of 12
American Institute of Aeronautics and Astronautics

References
1 Stern, F., Wilson, R. V., Coleman, H. W., and Paterson, E. G., Verification and Validation of CFD SimulationsPart 1: Methodologies and
Procedures, Journal of Fluids Engineering, , No. 4, 2001, pp. 737958.
2 Roache, P. J., Verification and Validation in Computational Science and Engineering, Hermosa Publishers, Albequerque, New Mexico, 1998.
3 Oberkampf, W. L. and Blottner, F. G., Issues in Computational Fluid Dynamics Code Verification and Validation, AIAA Journal, Vol. 36,
No. 5, May 1998, pp. 687695.
4 Rizzi, A. and Vos, J., Toward Establishing Credibility in Computational Fluid Dynamics Simulations, AIAA Journal, Vol. 36, No. 5, May
1998, pp. 668675.
5 Marvin, J. G., Perspective on Computational Fluid Dynamics Validation, AIAA Journal, Vol. 33, No. 10, October 1995, pp. 17781787.
6 Salari, K. and Knupp, P., Code Verification by the Method of Manufactured Solutions, Tech. Rep. SAND2000-1444, Sandia National
Laboratories, 2000.
7 Luke, E. A., Tong, X.-L., Wu, J., and Cinnella, P., CHEM 2: A Finite-Rate Viscous Chemistry Solver The User Guide, Tech. Rep.
MSSU-COE-ERC-04-07, Mississippi State University, 2004.
8 Luke, E. A., Tong, X., Wu, J., Tang, L., and Cinnella, P., A Step Towards Shape-Shifting Algorithms: Reacting Flow Simulations Using
Generalized Grids, Proceedings of the 39th AIAA Aerospace Sciences Meeting and Exhibit, AIAA, January 2001, AIAA-2001-0897.
9 Senguttuvan, V., Chalasani, S., Luke, E., and Thompson, D., Adaptive Mesh Refinement Using General Elements, 43rd AIAA Aerospace
Sciences Meeting and Exhibit, 2005.
10 Burg, C. O. E. and Murali, V. K., Efficient Code Verification Using the Residual Formulation of the Method of Manufactured Solutions,
34th AIAA Fluid Dynamics Conference and Exhibit, 2004.

12 of 12
American Institute of Aeronautics and Astronautics